![\"overview\"](\"https://raw.githubusercontent.com/Shujun-He/Feedback_Prize_2022_3rd_place_solution/main/graphics/write_up_graphic.png\")
First of all, thanks to the organizers for hosting an interesting and very competitive competition and congrats to all the winners. Next, thanks to my amazing teammates @aerdem4 @thedrcat Although we shook down by one place, it was still a good team effort and I'm happy about the results. This was my first NLP comp although it does fall in a familiar category (i.e. seq2seq predictions) for me.
\nOur inference notebook is public: https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322/notebook and training code is public as well: https://github.com/Shujun-He/3rd_Solution_Feedback_Prize_Evaluating_Student_Writing.
\nOur solution is a combination of token classification models using transformers and a stacking framework that classifies spans for 7 discourse types separately. Credit to @cdeotte and @chasembowers for sharing their amazing notebooks (https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633 and https://www.kaggle.com/chasembowers/sequence-postprocessing-v2-67-lb ).
\n
Our best ensemble consists of 6 folds of Longformer and 6 folds of sliding windows Deberta-xl models (weighted by discourse type based on cv). Due to the sequential nature of the problem, we add a 2-layer GRU on top of the transformer hidden states before outputting predictions for each token. We find it beneficial to increase max_len during training/inference to 2048, beyond which we actually saw performance degradation. In our case, deberta-l outperforms longformer large by quite a bit, and deberta-xl is even better, although deberta-xl is more difficult to train.
\nBelow I will discuss some specificities of training these models.
\nTraining longformers is relatively simple because longformers can take sequences longer than 512 without any issues. Our pipeline is similar to what's publicly available, mostly from @cdeotte
\nTraining deberta is a little trickier than longformers since we can't directly input a sequence longer than 512. Instead, we use a sliding window method when the sequence length is longer than 512. First, the first 512 positions are inputted to the deberta encoder. When we input the next segment into the deberta, we actually only increment the end position by 384, input position [512-64:512-64+512], and only concat the hidden states of the middle segment [64:448] to the hidden states of the first 512 positions, which avoids edge effects. We do this until we reach the end of the sequence and if the last segment is equal to smaller than 64, we simply take the last positions of the previous segment. After we run and concat all the segments, we run the sequence through a 2-layer GRU layer. Since the GRU processes the concatted segments sequentially, it essentially reconnects all the segments.
\nB,L=input_ids.shape\nif L<=self.window_size:\n x=self.backbone(input_ids=input_ids,attention_mask=attention_mask,return_dict=False)[0]\nelse:\n segments=(L-self.window_size)//self.inner_len\n if (L-self.window_size)%self.inner_len>self.edge_len:\n segments+=1\n elif segments==0:\n segments+=1\n x=self.backbone(input_ids=input_ids[:,:self.window_size],attention_mask=attention_mask[:,:self.window_size],return_dict=False)[0]\n for i in range(1,segments+1):\n start=self.window_size-self.edge_len+(i-1)*self.inner_len\n end=self.window_size-self.edge_len+(i-1)*self.inner_len+self.window_size\n end=min(end,L)\n x_next=input_ids[:,start:end]\n mask_next=attention_mask[:,start:end]\n x_next=self.backbone(input_ids=x_next,attention_mask=mask_next,return_dict=False)[0]\n if i==segments:\n x_next=x_next[:,self.edge_len:]\n else:\n x_next=x_next[:,self.edge_len:self.edge_len+self.inner_len]\n x=torch.cat([x,x_next],1)\nWe used 2 forms of augmentations:
\nif np.random.uniform()<0.5:\n cut=0.25\n perm=torch.randperm(ids.shape[0]).cuda()\n rand_len=int(ids.shape[1]*cut)\n start=np.random.randint(ids.shape[1]-int(ids.shape[1]*cut))\n ids[:,start:start+rand_len]=ids[perm,start:start+rand_len]\n mask[:,start:start+rand_len]=mask[perm,start:start+rand_len]\n labels[:,start:start+rand_len]=labels[perm,start:start+rand_len]\nSince we use augmentations, we train for 7 epochs at learning rates of [2.5e-5, 2.5e-5, 2.5e-5, 2.5e-6, 2.5e-6, 2.5e-6, 2.5e-7] for longformer and deberta-l; for deberta-xl, we use the same schedule with 1/5 of the learnig rate. For longformer, we use a batch size/max_length of 4/2048, for deberta-large 3/2048, and for debrta xl 2/1536 (mostly due to memory constraints).
\nDuring training, instead of padding every sequence to a fixed max length, we pad sequences to the max length in its batch, which speeds up training considerably. During inference, we also sort all texts by their lengths before batching, thereby minimizing the amount of padding in each batch, which speeds up inference quite a bit.
\nOur stacking framework is the same as the one posted by chase bowers; we just made a lot of improvements to it. In short (for those who aren't aware of the notebook), for each discourse type, using out of fold predictions, we generate features for candidate spans based on a begin token probability threshold and a max span length (99.5 percentile for each discourse type), and train gradient boosting models that operate on these features and classify spans. Therefore, we have 7 binary classification models, one for each discourse type. During inference, we sort all candidate spans by predicted probability and take those as predictions until a low threshold, while removing spans that intersect more than 0.15/0.2 with existing predicted spans.
\nBelow I will discuss the improvements.
\nWe set up cross-validation which ended up being mostly consistent with public lb. As a result, most of our improvements locally observed translated to public lb improvements as well. In some cases, however, improvements in noisy classes (e.g. Rebuttal), we saw some inconsistencies, but that was to be expected.
\nFirst we fixed a minor error with calculations of probability that a word corresponds to either a 'B'-egin or 'I'-nside token for a class in the original stacking notebook, where B and I tokens happening for the same class are considered independent events. From:
\nprob_or = lambda word_preds: (1-(1-word_preds[:,disc_begin]) * (1-word_preds[:,disc_inside]))\nto
\nprob_or = lambda word_preds: word_preds[:,disc_begin] + word_preds[:,disc_inside]\nWe added more probabilities at the edges of the span as well as the probability of another B token for any class following the span. Additionally, we added the location of the max/min probabilities for the class in the span. Further, we added something we call instability, which is the average squared difference in prob_or from position to position:
\ns = prob_or(text_preds[pred_start:pred_end])\ninstability = 0\nif len(s) > 1:\n instability = (np.diff(s)**2).mean()\nAside from span specific features, we added global features of the average prob_or of every discourse type and the positions with the max B token probs of every disourse type.
\nLast but not least, not all features are used for every discourse type, and instead we tune our feature selection based on CV. We have used around 25 features on average.
\nFor some discourse types, we reduced the min begin token probability threshold so we have more candidate spans.
\nIn our best ensemble, we have both an lgbm and an xgb model, each with 5 folds trained on oofs. The predictions from both lgbm and xgb are weighted equally.lgbm is trained on dart mode. Since it is not possible to do early stopping on dart mode, first xgb is trained and its optimal number of trees x 1.4 is used for number of trees. We accelerated our gradient boosting models with RAPIDS ForestInference and got room for an extra deberta fold.
\nAll candidate spans are sorted by predicted probability and taken as predictions until a low threshold. The original notebook did not allow any intersection with existing predicted spans during decoding, but we were able to improve our score by allowing a certain level of intersection (normalized by span length) instead, such as 0.2.
", "comments": [ { "id": 1725153, "author_name": "Marรญlia Prata", "author_url": "", "post_date": "2022-03-16T21:17:42.283000", "content": "Congrats Shujun, Ahmet and Darek (big fan here).
", "votes": 1, "replies": [] }, { "id": 1731315, "author_name": "Junbo Zhang", "author_url": "", "post_date": "2022-03-22T08:28:44.283000", "content": "Congratulations!
", "votes": 0, "replies": [] }, { "id": 1730751, "author_name": "Vadim Irtlach", "author_url": "", "post_date": "2022-03-21T15:26:30.823000", "content": "The amazing idea with CutMix! Did you tune the p and cut parameters?
Thanks. We didn't tune the p and cut parameters extensively. We tried decreasing p when training deberta-xl mostly due to its tendency to diverge but cv was worse. Earlier I tried incresing p or cut, but results became worse in both cases.
", "votes": 1, "replies": [] }, { "id": 1730989, "author_name": "Vadim Irtlach", "author_url": "", "post_date": "2022-03-21T22:08:20.540000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 1725592, "author_name": "Ansh Chourasia", "author_url": "", "post_date": "2022-03-17T08:34:33.663000", "content": "Congratulations and thank you for sharing this.
", "votes": 0, "replies": [] }, { "id": 1724971, "author_name": "Yasir Abdukadir", "author_url": "", "post_date": "2022-03-16T17:15:46.857000", "content": "well designed models
", "votes": 0, "replies": [] }, { "id": 1724887, "author_name": "Ishan Mehta115", "author_url": "", "post_date": "2022-03-16T15:47:16.117000", "content": "Congratulations and thank you for sharing this.
", "votes": 0, "replies": [] }, { "id": 1724563, "author_name": "SchenbergZ", "author_url": "", "post_date": "2022-03-16T10:29:13.760000", "content": "Congrats.
\nBut I fell a bit confusing reading your feature selection part. Maybe you can elaborate a bit like what does prob_or mean (the origin notebook is inaccessible), and why you do fixed a minor error with calculations,added more probabilities at the edges of the span as well as the probability of another B token for any class following the span and added the location of max/min probabilities
Thanks. The link should work now. The link included an extra parentheses before so it didn't work. The minor error is with the calculations of prob_or. I think you can find more details in the notebook. By \"more probabilities at the edges of the span\" I just mean probabilities at start_position - 2, end_position + 1, and so fourth. The probability of another B token for any class is essentially the probability of another span of any class starting right after the current one. Hope I cleared it yp
", "votes": 1, "replies": [] }, { "id": 1725628, "author_name": "SchenbergZ", "author_url": "", "post_date": "2022-03-17T09:11:19.963000", "content": "Thank you very much for your reply๐. One more question is: when you plug in new ideas, would you redo hyper-parameter search again or just keep those hyper parameters same as before?
", "votes": 0, "replies": [] }, { "id": 1727639, "author_name": "Shujun", "author_url": "", "post_date": "2022-03-18T06:21:57.510000", "content": "It depends. When we add features, we keep everything the same, but sometimes it makes sense to change hyperparameters. For instance, when we added augmentations, it made sense to increase the amount of epochs from 5 to 7 since training with heavier augmentations usually require more epochs to converge
", "votes": 1, "replies": [] } ] }, { "id": 1724525, "author_name": "neoistheone", "author_url": "", "post_date": "2022-03-16T09:48:44.430000", "content": "Nice write up. Data Pipeline is awesome.
", "votes": 0, "replies": [] }, { "id": 1724480, "author_name": "CPMP", "author_url": "", "post_date": "2022-03-16T09:20:03.690000", "content": "Congrats on the result. You designed a very interesting pipeline. I'm sure part of it will be reused by others.
", "votes": 0, "replies": [ { "id": 1725203, "author_name": "Shujun", "author_url": "", "post_date": "2022-03-16T22:50:03.127000", "content": "Thank you. The main ideas were publicly available, so we kind of just utilized the power of the kaggle community and added a lot of improvements.
", "votes": 1, "replies": [] } ] }, { "id": 1724309, "author_name": "Gregor Lied", "author_url": "", "post_date": "2022-03-16T07:03:21.893000", "content": "Thank you for your detailed write-up, and congratulations on an impressive 3rd place to you and your team!๐
\nCan you share how much masked augmentation improved your score? I also tried it, yet I was not observing significant improvements.
Thank you! For us, just adding masked augmentation only led to a very small amount of boost, less than 0.001, and the combination of masked aug and cutmix works the best. I don't know the exact amount of boost but I'd say around 0.001 to 0.002
", "votes": 3, "replies": [] } ] }, { "id": 1727290, "author_name": "", "author_url": "", "post_date": "2022-03-17T20:49:25.663000", "content": "Thank you very musch for sharing!๐
", "votes": 1, "replies": [] }, { "id": 1731009, "author_name": "Sofia Sidorov", "author_url": "", "post_date": "2022-03-21T22:56:09.677000", "content": "Thank you!!!
", "votes": 0, "replies": [] }, { "id": 1727589, "author_name": "Sri Krishna Chaitanya Kuracha", "author_url": "", "post_date": "2022-03-18T05:19:57.680000", "content": "Thank you for sharing the solution
", "votes": 0, "replies": [] }, { "id": 1724470, "author_name": "Kita Renji", "author_url": "", "post_date": "2022-03-16T09:11:46.590000", "content": "Thank you. Really appreciate your work
", "votes": 0, "replies": [] } ], "comment_count": 21, "messages": [ { "id": 1724301, "postDate": "2022-03-16T06:54:51.777Z", "content": "First of all, thanks to the organizers for hosting an interesting and very competitive competition and congrats to all the winners. Next, thanks to my amazing teammates @aerdem4 @thedrcat Although we shook down by one place, it was still a good team effort and I'm happy about the results. This was my first NLP comp although it does fall in a familiar category (i.e. seq2seq predictions) for me.
\nOur inference notebook is public: https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322/notebook and training code is public as well: https://github.com/Shujun-He/3rd_Solution_Feedback_Prize_Evaluating_Student_Writing.
\nOur solution is a combination of token classification models using transformers and a stacking framework that classifies spans for 7 discourse types separately. Credit to @cdeotte and @chasembowers for sharing their amazing notebooks (https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633 and https://www.kaggle.com/chasembowers/sequence-postprocessing-v2-67-lb ).
\n
Our best ensemble consists of 6 folds of Longformer and 6 folds of sliding windows Deberta-xl models (weighted by discourse type based on cv). Due to the sequential nature of the problem, we add a 2-layer GRU on top of the transformer hidden states before outputting predictions for each token. We find it beneficial to increase max_len during training/inference to 2048, beyond which we actually saw performance degradation. In our case, deberta-l outperforms longformer large by quite a bit, and deberta-xl is even better, although deberta-xl is more difficult to train.
\nBelow I will discuss some specificities of training these models.
\nTraining longformers is relatively simple because longformers can take sequences longer than 512 without any issues. Our pipeline is similar to what's publicly available, mostly from @cdeotte
\nTraining deberta is a little trickier than longformers since we can't directly input a sequence longer than 512. Instead, we use a sliding window method when the sequence length is longer than 512. First, the first 512 positions are inputted to the deberta encoder. When we input the next segment into the deberta, we actually only increment the end position by 384, input position [512-64:512-64+512], and only concat the hidden states of the middle segment [64:448] to the hidden states of the first 512 positions, which avoids edge effects. We do this until we reach the end of the sequence and if the last segment is equal to smaller than 64, we simply take the last positions of the previous segment. After we run and concat all the segments, we run the sequence through a 2-layer GRU layer. Since the GRU processes the concatted segments sequentially, it essentially reconnects all the segments.
\nB,L=input_ids.shape\nif L<=self.window_size:\n x=self.backbone(input_ids=input_ids,attention_mask=attention_mask,return_dict=False)[0]\nelse:\n segments=(L-self.window_size)//self.inner_len\n if (L-self.window_size)%self.inner_len>self.edge_len:\n segments+=1\n elif segments==0:\n segments+=1\n x=self.backbone(input_ids=input_ids[:,:self.window_size],attention_mask=attention_mask[:,:self.window_size],return_dict=False)[0]\n for i in range(1,segments+1):\n start=self.window_size-self.edge_len+(i-1)*self.inner_len\n end=self.window_size-self.edge_len+(i-1)*self.inner_len+self.window_size\n end=min(end,L)\n x_next=input_ids[:,start:end]\n mask_next=attention_mask[:,start:end]\n x_next=self.backbone(input_ids=x_next,attention_mask=mask_next,return_dict=False)[0]\n if i==segments:\n x_next=x_next[:,self.edge_len:]\n else:\n x_next=x_next[:,self.edge_len:self.edge_len+self.inner_len]\n x=torch.cat([x,x_next],1)\nWe used 2 forms of augmentations:
\nif np.random.uniform()<0.5:\n cut=0.25\n perm=torch.randperm(ids.shape[0]).cuda()\n rand_len=int(ids.shape[1]*cut)\n start=np.random.randint(ids.shape[1]-int(ids.shape[1]*cut))\n ids[:,start:start+rand_len]=ids[perm,start:start+rand_len]\n mask[:,start:start+rand_len]=mask[perm,start:start+rand_len]\n labels[:,start:start+rand_len]=labels[perm,start:start+rand_len]\nSince we use augmentations, we train for 7 epochs at learning rates of [2.5e-5, 2.5e-5, 2.5e-5, 2.5e-6, 2.5e-6, 2.5e-6, 2.5e-7] for longformer and deberta-l; for deberta-xl, we use the same schedule with 1/5 of the learnig rate. For longformer, we use a batch size/max_length of 4/2048, for deberta-large 3/2048, and for debrta xl 2/1536 (mostly due to memory constraints).
\nDuring training, instead of padding every sequence to a fixed max length, we pad sequences to the max length in its batch, which speeds up training considerably. During inference, we also sort all texts by their lengths before batching, thereby minimizing the amount of padding in each batch, which speeds up inference quite a bit.
\nOur stacking framework is the same as the one posted by chase bowers; we just made a lot of improvements to it. In short (for those who aren't aware of the notebook), for each discourse type, using out of fold predictions, we generate features for candidate spans based on a begin token probability threshold and a max span length (99.5 percentile for each discourse type), and train gradient boosting models that operate on these features and classify spans. Therefore, we have 7 binary classification models, one for each discourse type. During inference, we sort all candidate spans by predicted probability and take those as predictions until a low threshold, while removing spans that intersect more than 0.15/0.2 with existing predicted spans.
\nBelow I will discuss the improvements.
\nWe set up cross-validation which ended up being mostly consistent with public lb. As a result, most of our improvements locally observed translated to public lb improvements as well. In some cases, however, improvements in noisy classes (e.g. Rebuttal), we saw some inconsistencies, but that was to be expected.
\nFirst we fixed a minor error with calculations of probability that a word corresponds to either a 'B'-egin or 'I'-nside token for a class in the original stacking notebook, where B and I tokens happening for the same class are considered independent events. From:
\nprob_or = lambda word_preds: (1-(1-word_preds[:,disc_begin]) * (1-word_preds[:,disc_inside]))\nto
\nprob_or = lambda word_preds: word_preds[:,disc_begin] + word_preds[:,disc_inside]\nWe added more probabilities at the edges of the span as well as the probability of another B token for any class following the span. Additionally, we added the location of the max/min probabilities for the class in the span. Further, we added something we call instability, which is the average squared difference in prob_or from position to position:
\ns = prob_or(text_preds[pred_start:pred_end])\ninstability = 0\nif len(s) > 1:\n instability = (np.diff(s)**2).mean()\nAside from span specific features, we added global features of the average prob_or of every discourse type and the positions with the max B token probs of every disourse type.
\nLast but not least, not all features are used for every discourse type, and instead we tune our feature selection based on CV. We have used around 25 features on average.
\nFor some discourse types, we reduced the min begin token probability threshold so we have more candidate spans.
\nIn our best ensemble, we have both an lgbm and an xgb model, each with 5 folds trained on oofs. The predictions from both lgbm and xgb are weighted equally.lgbm is trained on dart mode. Since it is not possible to do early stopping on dart mode, first xgb is trained and its optimal number of trees x 1.4 is used for number of trees. We accelerated our gradient boosting models with RAPIDS ForestInference and got room for an extra deberta fold.
\nAll candidate spans are sorted by predicted probability and taken as predictions until a low threshold. The original notebook did not allow any intersection with existing predicted spans during decoding, but we were able to improve our score by allowing a certain level of intersection (normalized by span length) instead, such as 0.2.
", "rawMarkdown": "First of all, thanks to the organizers for hosting an interesting and very competitive competition and congrats to all the winners. Next, thanks to my amazing teammates @aerdem4 @thedrcat Although we shook down by one place, it was still a good team effort and I'm happy about the results. This was my first NLP comp although it does fall in a familiar category (i.e. seq2seq predictions) for me.\n\nOur inference notebook is public: https://www.kaggle.com/code/aerdem4/xgb-lgb-feedback-prize-cv-0-7322/notebook and training code is public as well: https://github.com/Shujun-He/3rd_Solution_Feedback_Prize_Evaluating_Student_Writing.\n\n# Overview\nOur solution is a combination of token classification models using transformers and a stacking framework that classifies spans for 7 discourse types separately. Credit to @cdeotte and @chasembowers for sharing their amazing notebooks (https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633 and https://www.kaggle.com/chasembowers/sequence-postprocessing-v2-67-lb ).\n\n\n\n# Deep learning models\n\nOur best ensemble consists of 6 folds of Longformer and 6 folds of sliding windows Deberta-xl models (weighted by discourse type based on cv). Due to the sequential nature of the problem, we add a 2-layer GRU on top of the transformer hidden states before outputting predictions for each token. We find it beneficial to increase max_len during training/inference to 2048, beyond which we actually saw performance degradation. In our case, deberta-l outperforms longformer large by quite a bit, and deberta-xl is even better, although deberta-xl is more difficult to train.\n\nBelow I will discuss some specificities of training these models.\n\n### Longformer\nTraining longformers is relatively simple because longformers can take sequences longer than 512 without any issues. Our pipeline is similar to what's publicly available, mostly from @cdeotte \n\n### Sliding window (SW) Deberta-xl\nTraining deberta is a little trickier than longformers since we can't directly input a sequence longer than 512. Instead, we use a sliding window method when the sequence length is longer than 512. First, the first 512 positions are inputted to the deberta encoder. When we input the next segment into the deberta, we actually only increment the end position by 384, input position [512-64:512-64+512], and only concat the hidden states of the middle segment [64:448] to the hidden states of the first 512 positions, which avoids edge effects. We do this until we reach the end of the sequence and if the last segment is equal to smaller than 64, we simply take the last positions of the previous segment. After we run and concat all the segments, we run the sequence through a 2-layer GRU layer. Since the GRU processes the concatted segments sequentially, it essentially reconnects all the segments.\n\n```\nB,L=input_ids.shape\nif L<=self.window_size:\n x=self.backbone(input_ids=input_ids,attention_mask=attention_mask,return_dict=False)[0]\nelse:\n segments=(L-self.window_size)//self.inner_len\n if (L-self.window_size)%self.inner_len>self.edge_len:\n segments+=1\n elif segments==0:\n segments+=1\n x=self.backbone(input_ids=input_ids[:,:self.window_size],attention_mask=attention_mask[:,:self.window_size],return_dict=False)[0]\n for i in range(1,segments+1):\n start=self.window_size-self.edge_len+(i-1)*self.inner_len\n end=self.window_size-self.edge_len+(i-1)*self.inner_len+self.window_size\n end=min(end,L)\n x_next=input_ids[:,start:end]\n mask_next=attention_mask[:,start:end]\n x_next=self.backbone(input_ids=x_next,attention_mask=mask_next,return_dict=False)[0]\n if i==segments:\n x_next=x_next[:,self.edge_len:]\n else:\n x_next=x_next[:,self.edge_len:self.edge_len+self.inner_len]\n x=torch.cat([x,x_next],1)\n```\n\n### Augmentation\nWe used 2 forms of augmentations:\n1. Masked aug, where we mask 15% of the tokens during training.\n2. Cutmix, similar to how cutmix works for images, we cut a portion of one sequence and paste it (and its labels) into another sequence in the same batch. Implementation is quite simple:\n```\nif np.random.uniform()<0.5:\n cut=0.25\n perm=torch.randperm(ids.shape[0]).cuda()\n rand_len=int(ids.shape[1]*cut)\n start=np.random.randint(ids.shape[1]-int(ids.shape[1]*cut))\n ids[:,start:start+rand_len]=ids[perm,start:start+rand_len]\n mask[:,start:start+rand_len]=mask[perm,start:start+rand_len]\n labels[:,start:start+rand_len]=labels[perm,start:start+rand_len]\n```\n\n### Hyperparameters\n\nSince we use augmentations, we train for 7 epochs at learning rates of [2.5e-5, 2.5e-5, 2.5e-5, 2.5e-6, 2.5e-6, 2.5e-6, 2.5e-7] for longformer and deberta-l; for deberta-xl, we use the same schedule with 1/5 of the learnig rate. For longformer, we use a batch size/max_length of 4/2048, for deberta-large 3/2048, and for debrta xl 2/1536 (mostly due to memory constraints).\n\n### Training/inference tricks\nDuring training, instead of padding every sequence to a fixed max length, we pad sequences to the max length in its batch, which speeds up training considerably. During inference, we also sort all texts by their lengths before batching, thereby minimizing the amount of padding in each batch, which speeds up inference quite a bit.\n\n# Stacking framework\n\nOur stacking framework is the same as the one posted by chase bowers; we just made a lot of improvements to it. In short (for those who aren't aware of the notebook), for each discourse type, using out of fold predictions, we generate features for candidate spans based on a begin token probability threshold and a max span length (99.5 percentile for each discourse type), and train gradient boosting models that operate on these features and classify spans. Therefore, we have 7 binary classification models, one for each discourse type. During inference, we sort all candidate spans by predicted probability and take those as predictions until a low threshold, while removing spans that intersect more than 0.15/0.2 with existing predicted spans. \n\nBelow I will discuss the improvements.\n\n### CV setup\n\nWe set up cross-validation which ended up being mostly consistent with public lb. As a result, most of our improvements locally observed translated to public lb improvements as well. In some cases, however, improvements in noisy classes (e.g. Rebuttal), we saw some inconsistencies, but that was to be expected.\n\n### Features\n\nFirst we fixed a minor error with calculations of probability that a word corresponds to either a 'B'-egin or 'I'-nside token for a class in the original stacking notebook, where B and I tokens happening for the same class are considered independent events. From:\n```\nprob_or = lambda word_preds: (1-(1-word_preds[:,disc_begin]) * (1-word_preds[:,disc_inside]))\n```\nto\n```\nprob_or = lambda word_preds: word_preds[:,disc_begin] + word_preds[:,disc_inside]\n```\n\nWe added more probabilities at the edges of the span as well as the probability of another B token for any class following the span. Additionally, we added the location of the max/min probabilities for the class in the span. Further, we added something we call instability, which is the average squared difference in prob_or from position to position:\n```\ns = prob_or(text_preds[pred_start:pred_end])\ninstability = 0\nif len(s) > 1:\n instability = (np.diff(s)**2).mean()\n```\nAside from span specific features, we added global features of the average prob_or of every discourse type and the positions with the max B token probs of every disourse type.\n\nLast but not least, not all features are used for every discourse type, and instead we tune our feature selection based on CV. We have used around 25 features on average.\n\n### Increasing the amount of candidate spans\n\nFor some discourse types, we reduced the min begin token probability threshold so we have more candidate spans.\n\n### Gradient boosting models\n\nIn our best ensemble, we have both an lgbm and an xgb model, each with 5 folds trained on oofs. The predictions from both lgbm and xgb are weighted equally.lgbm is trained on dart mode. Since it is not possible to do early stopping on dart mode, first xgb is trained and its optimal number of trees x 1.4 is used for number of trees. We accelerated our gradient boosting models with RAPIDS ForestInference and got room for an extra deberta fold.\n\n### Decoding\n\nAll candidate spans are sorted by predicted probability and taken as predictions until a low threshold. The original notebook did not allow any intersection with existing predicted spans during decoding, but we were able to improve our score by allowing a certain level of intersection (normalized by span length) instead, such as 0.2.\n", "votes": 72 }, { "id": 1725153, "postDate": "2022-03-16T21:17:42.283Z", "content": "Congrats Shujun, Ahmet and Darek (big fan here).
", "rawMarkdown": "Congrats Shujun, Ahmet and Darek (big fan here).", "votes": 1 }, { "id": 1731315, "postDate": "2022-03-22T08:28:44.283Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 1730751, "postDate": "2022-03-21T15:26:30.823Z", "content": "The amazing idea with CutMix! Did you tune the p and cut parameters?
Thanks. We didn't tune the p and cut parameters extensively. We tried decreasing p when training deberta-xl mostly due to its tendency to diverge but cv was worse. Earlier I tried incresing p or cut, but results became worse in both cases.
", "rawMarkdown": "Thanks. We didn't tune the p and cut parameters extensively. We tried decreasing p when training deberta-xl mostly due to its tendency to diverge but cv was worse. Earlier I tried incresing p or cut, but results became worse in both cases.", "votes": 1 }, { "id": 1730989, "postDate": "2022-03-21T22:08:20.540Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 1725592, "postDate": "2022-03-17T08:34:33.663Z", "content": "Congratulations and thank you for sharing this.
", "rawMarkdown": "Congratulations and thank you for sharing this." }, { "id": 1724971, "postDate": "2022-03-16T17:15:46.857Z", "content": "well designed models
", "rawMarkdown": "well designed models\n" }, { "id": 1724887, "postDate": "2022-03-16T15:47:16.117Z", "content": "Congratulations and thank you for sharing this.
", "rawMarkdown": "Congratulations and thank you for sharing this." }, { "id": 1724563, "postDate": "2022-03-16T10:29:13.760Z", "content": "Congrats.
\nBut I fell a bit confusing reading your feature selection part. Maybe you can elaborate a bit like what does prob_or mean (the origin notebook is inaccessible), and why you do fixed a minor error with calculations,added more probabilities at the edges of the span as well as the probability of another B token for any class following the span and added the location of max/min probabilities
Thanks. The link should work now. The link included an extra parentheses before so it didn't work. The minor error is with the calculations of prob_or. I think you can find more details in the notebook. By \"more probabilities at the edges of the span\" I just mean probabilities at start_position - 2, end_position + 1, and so fourth. The probability of another B token for any class is essentially the probability of another span of any class starting right after the current one. Hope I cleared it yp
", "rawMarkdown": "Thanks. The link should work now. The link included an extra parentheses before so it didn't work. The minor error is with the calculations of prob_or. I think you can find more details in the notebook. By \"more probabilities at the edges of the span\" I just mean probabilities at start_position - 2, end_position + 1, and so fourth. The probability of another B token for any class is essentially the probability of another span of any class starting right after the current one. Hope I cleared it yp", "votes": 1 }, { "id": 1725628, "postDate": "2022-03-17T09:11:19.963Z", "content": "Thank you very much for your reply๐. One more question is: when you plug in new ideas, would you redo hyper-parameter search again or just keep those hyper parameters same as before?
", "rawMarkdown": "Thank you very much for your reply๐. One more question is: when you plug in new ideas, would you redo hyper-parameter search again or just keep those hyper parameters same as before?" }, { "id": 1727639, "postDate": "2022-03-18T06:21:57.510Z", "content": "It depends. When we add features, we keep everything the same, but sometimes it makes sense to change hyperparameters. For instance, when we added augmentations, it made sense to increase the amount of epochs from 5 to 7 since training with heavier augmentations usually require more epochs to converge
", "rawMarkdown": "It depends. When we add features, we keep everything the same, but sometimes it makes sense to change hyperparameters. For instance, when we added augmentations, it made sense to increase the amount of epochs from 5 to 7 since training with heavier augmentations usually require more epochs to converge", "votes": 1 } ] }, { "id": 1724525, "postDate": "2022-03-16T09:48:44.430Z", "content": "Nice write up. Data Pipeline is awesome.
", "rawMarkdown": "Nice write up. Data Pipeline is awesome." }, { "id": 1724480, "postDate": "2022-03-16T09:20:03.690Z", "content": "Congrats on the result. You designed a very interesting pipeline. I'm sure part of it will be reused by others.
", "rawMarkdown": "Congrats on the result. You designed a very interesting pipeline. I'm sure part of it will be reused by others.", "replies": [ { "id": 1725203, "postDate": "2022-03-16T22:50:03.127Z", "content": "Thank you. The main ideas were publicly available, so we kind of just utilized the power of the kaggle community and added a lot of improvements.
", "rawMarkdown": "Thank you. The main ideas were publicly available, so we kind of just utilized the power of the kaggle community and added a lot of improvements.", "votes": 1 } ] }, { "id": 1724309, "postDate": "2022-03-16T07:03:21.893Z", "content": "Thank you for your detailed write-up, and congratulations on an impressive 3rd place to you and your team!๐
\nCan you share how much masked augmentation improved your score? I also tried it, yet I was not observing significant improvements.
Thank you! For us, just adding masked augmentation only led to a very small amount of boost, less than 0.001, and the combination of masked aug and cutmix works the best. I don't know the exact amount of boost but I'd say around 0.001 to 0.002
", "rawMarkdown": "Thank you! For us, just adding masked augmentation only led to a very small amount of boost, less than 0.001, and the combination of masked aug and cutmix works the best. I don't know the exact amount of boost but I'd say around 0.001 to 0.002", "votes": 3 } ] }, { "id": 1727290, "postDate": "2022-03-17T20:49:25.663Z", "content": "Thank you very musch for sharing!๐
", "rawMarkdown": "Thank you very musch for sharing!๐", "votes": 1 }, { "id": 1731009, "postDate": "2022-03-21T22:56:09.677Z", "content": "Thank you!!!
", "rawMarkdown": "Thank you!!!" }, { "id": 1727589, "postDate": "2022-03-18T05:19:57.680Z", "content": "Thank you for sharing the solution
", "rawMarkdown": "Thank you for sharing the solution" }, { "id": 1724470, "postDate": "2022-03-16T09:11:46.590Z", "content": "Thank you. Really appreciate your work
", "rawMarkdown": "Thank you. Really appreciate your work" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:38:13Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 7887016, "solution_url": "/competitions/feedback-prize-2021/writeups/bestoverfitting-3rd-place-solution-w-code-and-note" }