First of all, a huge thank you to the competition hosts and Kaggle for organizing this competition.
\nI can't wait to read all the other write-ups in detail.
My approach is a weighted ensemble of different BERT backbones and topic-dependent postprocessing. To enable training Shortformer-like models for this competition, I use a sliding window approach. Although I only achieved 55th place, I hope to still share and discuss some new ideas with you.
\nSliding Window Training / Inference
\nBy default, Shortformer-like models can’t process long documents. To overcome this issue, I split long essays into chunks with a left and right context. Then I only make predictions for the central part of the chunk, but not for the context window. ![\"flert-paper.png\"](\"https://i.postimg.cc/6qPBBzcq/flert-paper.png\")
This approach was inspired by this paper and improved my single models by 0.005+ compared to applying normal stride.
Training Setup and Hyperparamters
\nFor training, I use learning rate of 1e-5, batch size of 4 (with gradient accumulation), linear/cosine lr schedule with warmup of 10%, BIO-label scheme and 8-bit Adam.
Furthermore, I tried to apply multi-sample dropout, data augmentation with [MASK], gradually (un)freezing layers, (grouped) discriminative learning rate, BIEO-label scheme, and within-task-pretraining. However, all these techniques did not lead to significant improvements.
\nError Analysis
\nImproving deep learning systems is a highly iterative process. It is important to perform error analysis to identify areas to improve frequently. I wrote a function that takes the raw submission.csv, the post-processed submission.csv, and the ground truth train.csv and displays them side-by-side while ranking them from worst-to-best/best-to-worst overall or discourse-dependent performance. This visualization helped me quite a lot throughout this competition. ![\"error-analysis.png\"](\"https://i.postimg.cc/g0Pw8kWs/error-analysis.png\")
Models
\nDebertaV3-Large
\n5 folds, 512 max length, 128+128 context window, public/private LB 0.698/0.709, CV 0.697
Funnel-Large
\n5 folds, 512 max length, 128+128 context window, public/private LB 0.692/0.704, CV 0.92
Roberta-Large
\n5 folds, 512 max length, 128+128 context window, public/private LB 0.684/0.695, CV 0.682
Longformer-Large
\n5 folds, 1536 max length, no context window, public/private LB 0.688/0.699, CV 0.686
Ensamble
\nI built a weighted ensemble of DebertaV3 (0.4788), Funnel Transformer (0.3192), Longformer (0.152), and Roberta (0.05) for submission by applying the ensemble pseudo code from this discussion.
Using weighted ensemble improved private/public LB 0.702/0.714 compared to public/private LB 0.700/0.712 using average ensemble.
\nTopic-dependent Postprocessing
\nIn the early days of the competition, I discovered 15 essay topics in the dataset. Later on this was also revealed in a discussion post and notebook by Chris.
Using BERTopic, I quickly came up with a simple model that allowed me to predict the topic for a specific essay with over 99.5% accuracy. With this model, I can apply topic-dependent postprocessing, i.e. tuning individual length_threshold and proba_threshold dictionaries for each topic, rather than having two global dictionaries. However, I think my training/tuning setup was suboptimal for this postprocessing since I only consider the discourse, but not the topic in StratifiedKFold. With the right CV strategy, the score probably could have been further improved.
Even with my CV strategy, topic-dependent postprocessing still boosted my score slightly (public/private LB 0.706/0.716).
\nI also tried to add a prompt at the start of the essay, e.g. ![\"prompt.png\"](\"https://i.postimg.cc/C567xn9m/prompt.png\")
Unfortunately, this did not lead to any significant improvements. That is because topic information is already present in the individual word(piece) embeddings based on topic-specific words that appear in an essay.
Additional Trick: Replace \\n by <newline> token
\nThe available public notebooks showed that \\n is an important feature. However, \\n is handled differently by different tokenizers (Some split ‘\\n’ into ‘\\’ and ’n’, some don’t split ‘\\n’, some map it to <UNK>, etc.). That is why I mapped all \\n to a <newline> token and added this token as a new special token to the tokenizer during initialization.
Hardware Resources
\nI have access to Kaggle + Colab.
Final toughts
\nIn conclusion, FeedbackPrize was a fun competition.
However, I am still skeptical about using Token Classification models for this competition. That is because the naive decoding procedure (using softmax) feels similar to applying a Greedy Search to Neural Machine Translation, which may lead to suboptimal output sequences, even if the models were trained correctly. As an example, it was common for my prototype models to make predictions like this:
\n![\"suboptimal-output.png\"](\"https://i.postimg.cc/Y2yzggyx/suboptimal-output.png\")
\nI was experimenting with a Conditional Random Field decoder because we can add constraints to such a model, that can prevent these results to occur. But I was unable to beat postprocessing using length_threshold and proba_threshold dictionaries.
More research has to be done in this direction😊
", "comments": [ { "id": 1730620, "author_name": "Pingzhi Li", "author_url": "", "post_date": "2022-03-21T13:06:24.800000", "content": "How do you efficiently find optimal thresholds, given there are more than 7*15=105 hyper-parameters.
\nI used some auto-tuning tools baed on random searching, however, it takes tooooo much time🥲
I checked different hyperparameter settings in parallel using joblib library.
\nI think the code took around 9 hours to find the optimal setting.😊
import itertools\nfrom joblib import Parallel, delayed \n\ndef apply_topic_length_threshold(df):\n df = df.copy()\n df['len'] = df['predictionstring'].apply(lambda x: len(x.split()))\n for TOPIC in range(0, 15):\n for key, value in length_topic_thresh[TOPIC].items():\n index = df.loc[(df['class']==key)&(df['topic']==TOPIC)].query(f'len<{value}').index\n df.drop(index, inplace = True)\n return df\n\ndef apply_topic_proba_threshold(df):\n df = df.copy()\n for TOPIC in range(0, 15):\n for key, value in proba_topic_thresh[TOPIC].items():\n index = df.loc[(df['class']==key)&(df['topic']==TOPIC)].query(f'pred_proba<{value}').index\n df.drop(index, inplace = True)\n return df\n\ndef check_hyperparamters(oof, gt_dataframe, TOPIC, LENGTH_THRESH, PROBA_THRESH):\n\n # change threshold\n length_topic_thresh[TOPIC][OPTIMIZE] = LENGTH_THRESH\n proba_topic_thresh[TOPIC][OPTIMIZE] = PROBA_THRESH\n\n # apply postprocessing\n oof_pp = oof.copy()\n oof_pp = apply_topic_length_threshold(oof_pp)\n oof_pp = apply_topic_proba_threshold(oof_pp)\n\n # compute cv for postprocessed oof\n metrics = {}\n f1_scores = []\n classes = [OPTIMIZE]\n for c in classes:\n pred_df = oof_pp.loc[oof_pp['class']==c].copy()\n gt_df = gt_dataframe.loc[gt_dataframe['discourse_type']==c].copy()\n f1 = score_feedback_comp(pred_df, gt_df)\n f1_scores.append(f1)\n metrics[f\"PP-F1-{c}\"] = f1\n metrics[f\"PP-F1-Overall\"] = np.mean(f1_scores)\n\n for key, val in metrics.items():\n if f'PP-F1-{OPTIMIZE}' in key:\n return val\n\nlength_topic_thresh = {topic: defaultdict(int) for topic in range(0, 15)}\nproba_topic_thresh = {topic: defaultdict(float) for topic in range(0, 15)}\n\n# topic-dependent parameter search\n# define search space\nOPTIMIZE_LIST = ['Lead', 'Position', 'Counterclaim', 'Rebuttal', 'Claim', 'Evidence', 'Concluding Statement']\nLENGTH_LIST = [[i for i in range(1, 30)] for _ in OPTIMIZE_LIST]\nPROBA_LIST = [[round(i,2) for i in np.arange(0.2, 1, 0.05)] for _ in OPTIMIZE_LIST]\n\nfor OPTIMIZE, length, proba in zip(OPTIMIZE_LIST, LENGTH_LIST, PROBA_LIST):\n\n for TOPIC in range(0, 15):\n\n # filter data\n ID2topic_topic = {key: val for key, val in ID2topic.items() if val == TOPIC}\n gt_dataframe_topic = gt_dataframe[gt_dataframe['id'].isin(ID2topic_topic)]\n oof_topic = oof[oof['id'].isin(ID2topic_topic)]\n\n # find best parameters\n COMBINATIONS = list(itertools.product(length, proba))\n results = Parallel(n_jobs=4, backend=\"multiprocessing\")(\ndelayed(check_hyperparamters)(oof_topic, gt_dataframe_topic, TOPIC, LENGTH_THRESH, PROBA_THRESH) for LENGTH_THRESH, PROBA_THRESH in COMBINATIONS\n)\n\n best_score = 0\n best_setting = None\n for result, (LENGTH_THRESH, PROBA_THRESH) in zip(results, list(itertools.product(length, proba))):\n if result > best_score:\n best_score = result\n best_setting = (LENGTH_THRESH, PROBA_THRESH)\n print(TOPIC, best_score, best_setting)\n length_topic_thresh[TOPIC][OPTIMIZE] = best_setting[0]\n proba_topic_thresh[TOPIC][OPTIMIZE] = best_setting[1]\nthx, and congrats!
", "votes": 0, "replies": [] } ] }, { "id": 1725585, "author_name": "Shreyans Dhankhar", "author_url": "", "post_date": "2022-03-17T08:20:54.793000", "content": "Great Approach !! Can you share your code as well, if possible.
", "votes": 1, "replies": [] }, { "id": 1724314, "author_name": "TianyuZhu", "author_url": "", "post_date": "2022-03-16T07:08:54.890000", "content": "I also came up with the idea of adding context from right hand side (to make it bidirectional context). 😄
\nHowever, that was in the late stage of competition, and I didn't have enough time to re-train all my models, so I just gave up.
\nHappy to see that you gained decent improvement adopting bidirectional context.
\nAnyway, congrats on solo finishing with a silver medal🎉
Thank you @zhutianyucs. The same goes for you. Congrats on an impressive solo performance!!
\nI still remember our discussion on generating reproducible performance using Longformer. It seems like everything worked out well in the end.😊
Doesn't the model lose the information from the previous chunk? For example, assume there is a claim and counterclaim in the previous chunk and a rebuttal in the next. If the model sees just the rebuttal chunk, its probably not very different from a 'claim'. So what is this 'chunked approach' actually learning?
", "votes": 1, "replies": [ { "id": 1724365, "author_name": "Gregor Lied", "author_url": "", "post_date": "2022-03-16T08:09:04.117000", "content": "Thank you for your question! Let me briefly explain.😊
\nIn general you are correct, this is the main downside we have to keep in mind while dealing with shortformer models. While dealing with a long document (e.g. 1536 tokens) and having a shortformer of maximal sequence length 512, the Lead tokens won't be able to attend to e.g. Concluding Statement tokens. This was the motivation why Longformer / Big Bird got proposed in order to tackle this problem on a global level.
\nBut Longformer / Big Bird are hard to train and it turns out that for this competition local information captured by shortformer models might be actually enough to predict the discourse types correctly, This is why most of my shortformer models outperformed Longformer and Big Bird.
\nThe problem you mentioned is rather a local one and in order to prevent it from occurring I applied a context window approach rather than using \"normal\" stride, as \"normal\" stride only takes the left context into account but not the right context. This also explains the observed boost in performance while using the context window approach, compared to \"normal\" stride.
", "votes": 3, "replies": [] }, { "id": 1724391, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-16T08:18:26.817000", "content": "thank you, congrats !!
", "votes": 1, "replies": [] } ] }, { "id": 1724290, "author_name": "YHPark_physicsedu", "author_url": "", "post_date": "2022-03-16T06:43:29.217000", "content": "Thank you for your sharing! If you don't mind, can you share us some of the code for visualizing the result?
", "votes": 1, "replies": [ { "id": 1724385, "author_name": "Gregor Lied", "author_url": "", "post_date": "2022-03-16T08:15:36.770000", "content": "You can have a look at the this notebook by @kaggleqrdl and modify it according to your workflow. I mainly added another column to the side-by-side plot.
", "votes": 0, "replies": [] }, { "id": 1727421, "author_name": "YHPark_physicsedu", "author_url": "", "post_date": "2022-03-18T00:57:54.453000", "content": "Once again, thank you so much for your kind sharing!
", "votes": 0, "replies": [] } ] }, { "id": 1724644, "author_name": "Gaju Ahmed", "author_url": "", "post_date": "2022-03-16T12:14:37.613000", "content": "Congratulation and thanks for sharing
", "votes": 0, "replies": [] } ], "comment_count": 14, "messages": [ { "id": 1724279, "postDate": "2022-03-16T06:19:34.303Z", "content": "First of all, a huge thank you to the competition hosts and Kaggle for organizing this competition.
\nI can't wait to read all the other write-ups in detail.
My approach is a weighted ensemble of different BERT backbones and topic-dependent postprocessing. To enable training Shortformer-like models for this competition, I use a sliding window approach. Although I only achieved 55th place, I hope to still share and discuss some new ideas with you.
\nSliding Window Training / Inference
\nBy default, Shortformer-like models can’t process long documents. To overcome this issue, I split long essays into chunks with a left and right context. Then I only make predictions for the central part of the chunk, but not for the context window. This approach was inspired by this paper and improved my single models by 0.005+ compared to applying normal stride.
Training Setup and Hyperparamters
\nFor training, I use learning rate of 1e-5, batch size of 4 (with gradient accumulation), linear/cosine lr schedule with warmup of 10%, BIO-label scheme and 8-bit Adam.
Furthermore, I tried to apply multi-sample dropout, data augmentation with [MASK], gradually (un)freezing layers, (grouped) discriminative learning rate, BIEO-label scheme, and within-task-pretraining. However, all these techniques did not lead to significant improvements.
\nError Analysis
\nImproving deep learning systems is a highly iterative process. It is important to perform error analysis to identify areas to improve frequently. I wrote a function that takes the raw submission.csv, the post-processed submission.csv, and the ground truth train.csv and displays them side-by-side while ranking them from worst-to-best/best-to-worst overall or discourse-dependent performance. This visualization helped me quite a lot throughout this competition.
Models
\nDebertaV3-Large
\n5 folds, 512 max length, 128+128 context window, public/private LB 0.698/0.709, CV 0.697
Funnel-Large
\n5 folds, 512 max length, 128+128 context window, public/private LB 0.692/0.704, CV 0.92
Roberta-Large
\n5 folds, 512 max length, 128+128 context window, public/private LB 0.684/0.695, CV 0.682
Longformer-Large
\n5 folds, 1536 max length, no context window, public/private LB 0.688/0.699, CV 0.686
Ensamble
\nI built a weighted ensemble of DebertaV3 (0.4788), Funnel Transformer (0.3192), Longformer (0.152), and Roberta (0.05) for submission by applying the ensemble pseudo code from this discussion.
Using weighted ensemble improved private/public LB 0.702/0.714 compared to public/private LB 0.700/0.712 using average ensemble.
\nTopic-dependent Postprocessing
\nIn the early days of the competition, I discovered 15 essay topics in the dataset. Later on this was also revealed in a discussion post and notebook by Chris.
Using BERTopic, I quickly came up with a simple model that allowed me to predict the topic for a specific essay with over 99.5% accuracy. With this model, I can apply topic-dependent postprocessing, i.e. tuning individual length_threshold and proba_threshold dictionaries for each topic, rather than having two global dictionaries. However, I think my training/tuning setup was suboptimal for this postprocessing since I only consider the discourse, but not the topic in StratifiedKFold. With the right CV strategy, the score probably could have been further improved.
Even with my CV strategy, topic-dependent postprocessing still boosted my score slightly (public/private LB 0.706/0.716).
\nI also tried to add a prompt at the start of the essay, e.g. Unfortunately, this did not lead to any significant improvements. That is because topic information is already present in the individual word(piece) embeddings based on topic-specific words that appear in an essay.
Additional Trick: Replace \\n by <newline> token
\nThe available public notebooks showed that \\n is an important feature. However, \\n is handled differently by different tokenizers (Some split ‘\\n’ into ‘\\’ and ’n’, some don’t split ‘\\n’, some map it to <UNK>, etc.). That is why I mapped all \\n to a <newline> token and added this token as a new special token to the tokenizer during initialization.
Hardware Resources
\nI have access to Kaggle + Colab.
Final toughts
\nIn conclusion, FeedbackPrize was a fun competition.
However, I am still skeptical about using Token Classification models for this competition. That is because the naive decoding procedure (using softmax) feels similar to applying a Greedy Search to Neural Machine Translation, which may lead to suboptimal output sequences, even if the models were trained correctly. As an example, it was common for my prototype models to make predictions like this:
\n
\nI was experimenting with a Conditional Random Field decoder because we can add constraints to such a model, that can prevent these results to occur. But I was unable to beat postprocessing using length_threshold and proba_threshold dictionaries.
More research has to be done in this direction😊
", "rawMarkdown": "## My approach – Shortformer + Sliding Window Training / Inference + Topic-dependent Postprocessing\nFirst of all, a huge thank you to the competition hosts and Kaggle for organizing this competition.\nI can't wait to read all the other write-ups in detail.\n\nMy approach is a weighted ensemble of different BERT backbones and topic-dependent postprocessing. To enable training Shortformer-like models for this competition, I use a sliding window approach. Although I only achieved 55th place, I hope to still share and discuss some new ideas with you.\n\n**Sliding Window Training / Inference**\nBy default, Shortformer-like models can’t process long documents. To overcome this issue, I split long essays into chunks with a left and right context. Then I only make predictions for the central part of the chunk, but not for the context window. [](https://postimg.cc/V0XxRq3P) This approach was inspired by this [paper](https://arxiv.org/pdf/2011.06993.pdf) and improved my single models by 0.005+ compared to applying normal stride.\n\n**Training Setup and Hyperparamters**\nFor training, I use learning rate of 1e-5, batch size of 4 (with gradient accumulation), linear/cosine lr schedule with warmup of 10%, BIO-label scheme and 8-bit Adam.\n\nFurthermore, I tried to apply multi-sample dropout, data augmentation with [MASK], gradually (un)freezing layers, (grouped) discriminative learning rate, BIEO-label scheme, and within-task-pretraining. However, all these techniques did not lead to significant improvements.\n\n**Error Analysis**\nImproving deep learning systems is a highly iterative process. It is important to perform error analysis to identify areas to improve frequently. I wrote a function that takes the raw submission.csv, the post-processed submission.csv, and the ground truth train.csv and displays them side-by-side while ranking them from worst-to-best/best-to-worst overall or discourse-dependent performance. This visualization helped me quite a lot throughout this competition. [](https://postimg.cc/BjMQG3tP)\n\n**Models**\nDebertaV3-Large\n5 folds, 512 max length, 128+128 context window, public/private LB 0.698/0.709, CV 0.697\n\nFunnel-Large\n5 folds, 512 max length, 128+128 context window, public/private LB 0.692/0.704, CV 0.92\n\nRoberta-Large\n5 folds, 512 max length, 128+128 context window, public/private LB 0.684/0.695, CV 0.682\n\nLongformer-Large\n5 folds, 1536 max length, no context window, public/private LB 0.688/0.699, CV 0.686\n\n**Ensamble**\nI built a weighted ensemble of DebertaV3 (0.4788), Funnel Transformer (0.3192), Longformer (0.152), and Roberta (0.05) for submission by applying the ensemble pseudo code from this [discussion](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175344).\n\nUsing weighted ensemble improved private/public LB 0.702/0.714 compared to public/private LB 0.700/0.712 using average ensemble.\n\n**Topic-dependent Postprocessing**\nIn the early days of the competition, I discovered 15 essay topics in the dataset. Later on this was also revealed in a [discussion post](https://www.kaggle.com/c/feedback-prize-2021/discussion/301481) and [notebook](https://www.kaggle.com/cdeotte/rapids-umap-tfidf-kmeans-discovers-15-topics) by Chris.\n\nUsing BERTopic, I quickly came up with a simple model that allowed me to predict the topic for a specific essay with over 99.5% accuracy. With this model, I can apply topic-dependent postprocessing, i.e. tuning individual `length_threshold` and `proba_threshold` dictionaries for each topic, rather than having two global dictionaries. However, I think my training/tuning setup was suboptimal for this postprocessing since I only consider the discourse, but not the topic in `StratifiedKFold`. With the right CV strategy, the score probably could have been further improved.\n\nEven with my CV strategy, topic-dependent postprocessing still boosted my score slightly (public/private LB 0.706/0.716).\n\nI also tried to add a prompt at the start of the essay, e.g. [](https://postimg.cc/K3BB9jGM) Unfortunately, this did not lead to any significant improvements. That is because topic information is already present in the individual word(piece) embeddings based on topic-specific words that appear in an essay.\n\n**Additional Trick: Replace \\n by `How do you efficiently find optimal thresholds, given there are more than 7*15=105 hyper-parameters.
\nI used some auto-tuning tools baed on random searching, however, it takes tooooo much time🥲
I checked different hyperparameter settings in parallel using joblib library.
\nI think the code took around 9 hours to find the optimal setting.😊
import itertools\nfrom joblib import Parallel, delayed \n\ndef apply_topic_length_threshold(df):\n df = df.copy()\n df['len'] = df['predictionstring'].apply(lambda x: len(x.split()))\n for TOPIC in range(0, 15):\n for key, value in length_topic_thresh[TOPIC].items():\n index = df.loc[(df['class']==key)&(df['topic']==TOPIC)].query(f'len<{value}').index\n df.drop(index, inplace = True)\n return df\n\ndef apply_topic_proba_threshold(df):\n df = df.copy()\n for TOPIC in range(0, 15):\n for key, value in proba_topic_thresh[TOPIC].items():\n index = df.loc[(df['class']==key)&(df['topic']==TOPIC)].query(f'pred_proba<{value}').index\n df.drop(index, inplace = True)\n return df\n\ndef check_hyperparamters(oof, gt_dataframe, TOPIC, LENGTH_THRESH, PROBA_THRESH):\n\n # change threshold\n length_topic_thresh[TOPIC][OPTIMIZE] = LENGTH_THRESH\n proba_topic_thresh[TOPIC][OPTIMIZE] = PROBA_THRESH\n\n # apply postprocessing\n oof_pp = oof.copy()\n oof_pp = apply_topic_length_threshold(oof_pp)\n oof_pp = apply_topic_proba_threshold(oof_pp)\n\n # compute cv for postprocessed oof\n metrics = {}\n f1_scores = []\n classes = [OPTIMIZE]\n for c in classes:\n pred_df = oof_pp.loc[oof_pp['class']==c].copy()\n gt_df = gt_dataframe.loc[gt_dataframe['discourse_type']==c].copy()\n f1 = score_feedback_comp(pred_df, gt_df)\n f1_scores.append(f1)\n metrics[f\"PP-F1-{c}\"] = f1\n metrics[f\"PP-F1-Overall\"] = np.mean(f1_scores)\n\n for key, val in metrics.items():\n if f'PP-F1-{OPTIMIZE}' in key:\n return val\n\nlength_topic_thresh = {topic: defaultdict(int) for topic in range(0, 15)}\nproba_topic_thresh = {topic: defaultdict(float) for topic in range(0, 15)}\n\n# topic-dependent parameter search\n# define search space\nOPTIMIZE_LIST = ['Lead', 'Position', 'Counterclaim', 'Rebuttal', 'Claim', 'Evidence', 'Concluding Statement']\nLENGTH_LIST = [[i for i in range(1, 30)] for _ in OPTIMIZE_LIST]\nPROBA_LIST = [[round(i,2) for i in np.arange(0.2, 1, 0.05)] for _ in OPTIMIZE_LIST]\n\nfor OPTIMIZE, length, proba in zip(OPTIMIZE_LIST, LENGTH_LIST, PROBA_LIST):\n\n for TOPIC in range(0, 15):\n\n # filter data\n ID2topic_topic = {key: val for key, val in ID2topic.items() if val == TOPIC}\n gt_dataframe_topic = gt_dataframe[gt_dataframe['id'].isin(ID2topic_topic)]\n oof_topic = oof[oof['id'].isin(ID2topic_topic)]\n\n # find best parameters\n COMBINATIONS = list(itertools.product(length, proba))\n results = Parallel(n_jobs=4, backend=\"multiprocessing\")(\ndelayed(check_hyperparamters)(oof_topic, gt_dataframe_topic, TOPIC, LENGTH_THRESH, PROBA_THRESH) for LENGTH_THRESH, PROBA_THRESH in COMBINATIONS\n)\n\n best_score = 0\n best_setting = None\n for result, (LENGTH_THRESH, PROBA_THRESH) in zip(results, list(itertools.product(length, proba))):\n if result > best_score:\n best_score = result\n best_setting = (LENGTH_THRESH, PROBA_THRESH)\n print(TOPIC, best_score, best_setting)\n length_topic_thresh[TOPIC][OPTIMIZE] = best_setting[0]\n proba_topic_thresh[TOPIC][OPTIMIZE] = best_setting[1]\nthx, and congrats!
", "rawMarkdown": "thx, and congrats!" } ] }, { "id": 1725585, "postDate": "2022-03-17T08:20:54.793Z", "content": "Great Approach !! Can you share your code as well, if possible.
", "rawMarkdown": "Great Approach !! Can you share your code as well, if possible.", "votes": 1 }, { "id": 1724314, "postDate": "2022-03-16T07:08:54.890Z", "content": "I also came up with the idea of adding context from right hand side (to make it bidirectional context). 😄
\nHowever, that was in the late stage of competition, and I didn't have enough time to re-train all my models, so I just gave up.
\nHappy to see that you gained decent improvement adopting bidirectional context.
\nAnyway, congrats on solo finishing with a silver medal🎉
Thank you @zhutianyucs. The same goes for you. Congrats on an impressive solo performance!!
\nI still remember our discussion on generating reproducible performance using Longformer. It seems like everything worked out well in the end.😊
Doesn't the model lose the information from the previous chunk? For example, assume there is a claim and counterclaim in the previous chunk and a rebuttal in the next. If the model sees just the rebuttal chunk, its probably not very different from a 'claim'. So what is this 'chunked approach' actually learning?
", "rawMarkdown": "**Doesn't the model lose the information from the previous chunk**? For example, assume there is a claim and counterclaim in the previous chunk and a rebuttal in the next. If the model sees just the rebuttal chunk, its probably not very different from a 'claim'. **So what is this 'chunked approach' actually learning?**", "votes": 1, "replies": [ { "id": 1724365, "postDate": "2022-03-16T08:09:04.117Z", "content": "Thank you for your question! Let me briefly explain.😊
\nIn general you are correct, this is the main downside we have to keep in mind while dealing with shortformer models. While dealing with a long document (e.g. 1536 tokens) and having a shortformer of maximal sequence length 512, the Lead tokens won't be able to attend to e.g. Concluding Statement tokens. This was the motivation why Longformer / Big Bird got proposed in order to tackle this problem on a global level.
\nBut Longformer / Big Bird are hard to train and it turns out that for this competition local information captured by shortformer models might be actually enough to predict the discourse types correctly, This is why most of my shortformer models outperformed Longformer and Big Bird.
\nThe problem you mentioned is rather a local one and in order to prevent it from occurring I applied a context window approach rather than using \"normal\" stride, as \"normal\" stride only takes the left context into account but not the right context. This also explains the observed boost in performance while using the context window approach, compared to \"normal\" stride.
", "rawMarkdown": "Thank you for your question! Let me briefly explain.😊\n\nIn general you are correct, this is the main downside we have to keep in mind while dealing with shortformer models. While dealing with a long document (e.g. 1536 tokens) and having a shortformer of maximal sequence length 512, the Lead tokens won't be able to attend to e.g. Concluding Statement tokens. This was the motivation why Longformer / Big Bird got proposed in order to tackle this problem on a global level. \n\nBut Longformer / Big Bird are hard to train and it turns out that for this competition local information captured by shortformer models might be actually enough to predict the discourse types correctly, This is why most of my shortformer models outperformed Longformer and Big Bird.\n\nThe problem you mentioned is rather a local one and in order to prevent it from occurring I applied a context window approach rather than using \"normal\" stride, as \"normal\" stride only takes the left context into account but not the right context. This also explains the observed boost in performance while using the context window approach, compared to \"normal\" stride.", "votes": 3 }, { "id": 1724391, "postDate": "2022-03-16T08:18:26.817Z", "content": "thank you, congrats !!
", "rawMarkdown": "thank you, congrats !!", "votes": 1 } ] }, { "id": 1724290, "postDate": "2022-03-16T06:43:29.217Z", "content": "Thank you for your sharing! If you don't mind, can you share us some of the code for visualizing the result?
", "rawMarkdown": "Thank you for your sharing! If you don't mind, can you share us some of the code for visualizing the result?", "votes": 1, "replies": [ { "id": 1724385, "postDate": "2022-03-16T08:15:36.770Z", "content": "You can have a look at the this notebook by @kaggleqrdl and modify it according to your workflow. I mainly added another column to the side-by-side plot.
", "rawMarkdown": "You can have a look at the this [notebook](https://www.kaggle.com/kaggleqrdl/oof-vrs-valid-eda) by @kaggleqrdl and modify it according to your workflow. I mainly added another column to the side-by-side plot." }, { "id": 1727421, "postDate": "2022-03-18T00:57:54.453Z", "content": "Once again, thank you so much for your kind sharing!
", "rawMarkdown": "Once again, thank you so much for your kind sharing!" } ] }, { "id": 1724644, "postDate": "2022-03-16T12:14:37.613Z", "content": "Congratulation and thanks for sharing
", "rawMarkdown": "Congratulation and thanks for sharing " } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:35:21Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 7902824, "solution_url": "/competitions/feedback-prize-2021/writeups/inductive-bias-55th-place-solution-shortformer-sli" }