Our team ensembled 26 models in the following table. (including knowledge distilled and pseudo labeled ones) Ridge regression was used to integrate the prediction results of each model. Post-processing was applied to adjust the output values of the final ensemble's predictions. For the single model, the following techniques contributed to score improvement.
\n| \n | Model | \nCV / LB | \nDescription | \n
|---|---|---|---|
| 1 | \nDeberta v3 base | \n0.448 | \n512 Max length , Pseudo labels inferred from the tensorflow Deberta v3 base (LLRD technique) , and those plus competition data trained on the deberta v3 base (freezing+meanpooling) | \n
| 2 | \nDeberta v3 large | \n0.448 | \nSame as above , deberta v3 large trained on meanpooling + freezing | \n
| 3 | \nDeberta v3 small | \n0.447 | \nKnowledge distilled model | \n
| 4 | \nDeberta v2 xlarge | \n0.448 / 0.44 | \nLSTM + mean pooling +freezing (Same pseudo label from tensorflow) | \n
| 5 | \nDeberta v3 large | \n0.4495 / 0.43 | \nMax len = 470, trained with decreasing max len - 768/512/470/470 | \n
| 6 | \nDeberta v3 base | \n0.4515 / 0.43 | \nMax len =470, Mean pooling, AWP, val_steps = 250, reinit layers | \n
| 7 | \nDeberta xlarge | \n0.4534 / 0.43 | \nCLS token, val steps =20 | \n
| 8 | \ndeberta v3 large | \n0.4495 / 0.43 | \nCLS Token, AWP, val steps = 250, max len 768 | \n
| 9 | \nElectra large | \n0.4545 | \nMeanpool, LSTM | \n
| 10 | \ndeepset/deberta-v3-bbase-squad2 | \n0.4522 | \n\n |
| 11 | \nLuke Large | \n0.4551 | \nMax len 512, Mean pooling, bidirectional-LSTM | \n
| 12 | \nDeberta v3 large | \n0.4467 | \n\n |
| 13 | \nDeberta v3 base | \n0.4471 | \nAutoModelForTokenClassification, last 4 layer concatenate, mean pooling,max_len=768 | \n
| 14 | \nDeberta v3 large | \n0.4469 | \nPL Model, 2 seed averaged, CLS Token | \n
| 16 | \nCocolm large | \n0.4568 | \nNo [PARAGRAPH], CLS Token, AWP | \n
| 17 | \nGPT2 Medium | \n0.4648 | \nmax_len=1024, Mean pooling, SWA | \n
| 18 | \nLongformer Large | \n0.4592 | \nmax_len=1536, Mean pooling | \n
| 19 | \nBart Large Squad | \n0.4649 | \nmax_len=1024, Mean pooling | \n
| 20 | \nDistill Bart CNN | \n0.4607 | \nmax_len=1024, Mean pooling | \n
| 21 | \nOPT-350M | \n0.4639 | \nmax_len=1536, Mean pooling | \n
| 22 | \nBart Base | \n0.4788 | \nmax_len=1024, Mean pooling, SWA | \n
| 23 | \nT5 Large | \n0.4717 | \nmax_len=1024, Mean pooling | \n
| 24 | \nDeberta v3 base | \n0.4490 | \nmax_len=512 | \n
| 25 | \nDeberta v3 base | \n0.4484 | \nmax_len =384 | \n
| 26 | \nDeberta v3 large | \n0.4498 | \nmax_len=1429, Mean pooling, Pseudo Labeling | \n
Preprocessing: - replacing “\\n\\n” with [PARAGRAPH] There are about 17.5K instances of \\n\\n in the dataset. Deberta v3 large tokenizer simply ignores these. So we decided to create one additional special token by replacing \\n\\n with [PARAGRAPH]. Also resized embeddings to take care of this one extra token. Just this change moved deberta v3 large from 0.44 to 0.43.
Knowledge Distillation: We had one deberta v3 x small model with knowledge distillation.
\nWe decided to distill knowledge i.e. (get OOF data) from an ensemble of 9 models and utilize these to train the model. This is only utilizing current competition data not previous feedback data.
Pseudo labels (PL)
\nThis was challenging to determine if pseudo labels are working or not. We tried generating labels with a few different models to determine if it’s working. Eventually we created two seed deberta v3 large models which seemed more stable than trying to determine with single deberta v3 large. Some of the models in our ensemble were with PL.
\nWe also did pseudo labeling inferred from the tensorflow Deberta v3 base model , which is trained with layer wise learning rate decay (LLRD) technique and used those PL inferred data into some of our models , which helps in improving the cv as well as LB.
Model Architecture:
\nJust CLS token, Mean pooling, Mean pooling + CLS token, MaskAddedAttentionHead, MaskAddedAttentionHead + CLS token(AutoModel and AutoModelForTokenClassification) , LSTM
Training strategies:
\nAWP worked, different max len per epoch, validation steps 20-250 were utilized for different models. Some of the models were trained using end of epoch validation as well.
\nCV was calculated with 5 folds models.
Diversity in ensemble:
\nFolds - By Abhishek - https://www.kaggle.com/code/abhishek/multi-label-stratified-folds
\nWe had ensemble of total 27 models (runtime ~7 hrs) - 7 Deberta v3 large models, 5 Deberta v3 base, Deberta v2 xlarge, Deberta xlarge, Deberta v3 small, Luke Large, Electra, Cocolm, Deberta v3 base squad, GPT2 medium, Longformer Large, Bart Large Squad, Distill Bart CNN, OPT 350M, Bart base, T5 large
Ridge Stacking
\nWe used Ridge for stacking our models, there were 2 deberta large (two seed models) which we combined so that generated only 1 ridge coefficient and it helped in ensemble.
Post processing
\nWe used “Nelder-mead” https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html for post processing - divided the predictions into 20 bins and multiplied the resulting array with predictions.
\nThis post processing is based on commonlit's 2nd place solution method.
\nhttps://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328
Our final ensemble is here - https://www.kaggle.com/code/aman1391/fb3-8th-place-solution-ensemble-27-models?scriptVersionId=112097448
\nMask Language modeling: - Tried many different ways for MLM to work, unsuccessful attempts. Tried to match similarity between FB3 data and previous feedback data for MLM which also didn’t work.
Other models: - Various other models from HuggingFace didn’t work. A few did work however it wouldn’t work with our ensemble so they were unutilized.
Different loss functions: - Focal Loss, Ordinal regression from public kernel, Asymmetric loss - Asymmetric loss did work with a few ensembles but it wasn’t part of our best submission.
Ensemble with weights: - In the last 2 days, we tried ensemble with weight tuning but Ridge did give us better results. This needed more tuning or work for us to feel confident in selecting a submission.
Many different model architectures, hyperparameters, token dropout, 4 fold model, dividing folds by number of tokens.
Compute: A100s, 4090 (3090 and Colab Pro+ prior to 4090)
Please see this post for a single model (average 2 seeds) gold solution - https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369368 and corresponding training code
Thanks to our hosts for hosting the competition and to teammates. Thanks also to the Kaggler's for sharing their helpful notebooks and discussions.
\nThank you and Congratulations to my team mates - @aman1391, @kanbehmw, @fightingmuscle
\nTeam Members
\n@rashmibanthia
\n@aman1391
\n@kanbehmw
\n@fightingmuscle
@rashmibanthia
\nThank you for your solution! and congratulation for being competition master!!!
\nI'm so glad to work with you!!!!
Likewise, It was pleasure working with you and Congratulations to you as well 🎉🎉
", "votes": 3, "replies": [] } ] }, { "id": 2054678, "author_name": "Marília Prata", "author_url": "", "post_date": "2022-12-04T10:39:46.287000", "content": "Thank you for sharing your valuable solution RB.
", "votes": 1, "replies": [] }, { "id": 2051853, "author_name": "AshAsh", "author_url": "", "post_date": "2022-12-01T16:50:32.610000", "content": "@rashmibanthia Thanks for the solution!
\n\n\nSo we decided to create one additional special token by replacing \\n\\n with [PARAGRAPH]. Also resized embeddings to take care of this one extra token.
\n
Would you mind sharing a code snippet on how to resize deberta embeddings to take care of this token exactly? Would like to try this but it's not obvious to me how to implement resizing
", "votes": 2, "replies": [ { "id": 2051900, "author_name": "RB", "author_url": "", "post_date": "2022-12-01T17:39:38.197000", "content": "Code for adding special token:
\nCFG.tokenizer = AutoTokenizer.from_pretrained(model)\nCFG.tokenizer.add_special_tokens({: []})\nThen, within model class, you need self.model.resize_token_embeddings(len(CFG.tokenizer)) (see below code)
(nn.Module):\n ():\n ().__init__()\n\n self.model_config = AutoConfig.from_pretrained(model_name)\n\n self.model_config.update(\n {\n : ,\n : ,\n : ,\n : CFG.num_labels,\n }\n )\n\n self.model = AutoModel.from_pretrained(model_name, config=self.model_config)\n self.model.resize_token_embeddings((CFG.tokenizer))\n\n self.output = nn.Linear(self.model_config.hidden_size, CFG.num_labels)\n\n\n ():\n transformer_out = self.model(input_ids=ids, attention_mask=mask)\n sequence_output = transformer_out.last_hidden_state[:, , :]\n\n logits = self.output(sequence_output)\n\n logits \nOur team ensembled 26 models in the following table. (including knowledge distilled and pseudo labeled ones) Ridge regression was used to integrate the prediction results of each model. Post-processing was applied to adjust the output values of the final ensemble's predictions. For the single model, the following techniques contributed to score improvement.
\n| \n | Model | \nCV / LB | \nDescription | \n
|---|---|---|---|
| 1 | \nDeberta v3 base | \n0.448 | \n512 Max length , Pseudo labels inferred from the tensorflow Deberta v3 base (LLRD technique) , and those plus competition data trained on the deberta v3 base (freezing+meanpooling) | \n
| 2 | \nDeberta v3 large | \n0.448 | \nSame as above , deberta v3 large trained on meanpooling + freezing | \n
| 3 | \nDeberta v3 small | \n0.447 | \nKnowledge distilled model | \n
| 4 | \nDeberta v2 xlarge | \n0.448 / 0.44 | \nLSTM + mean pooling +freezing (Same pseudo label from tensorflow) | \n
| 5 | \nDeberta v3 large | \n0.4495 / 0.43 | \nMax len = 470, trained with decreasing max len - 768/512/470/470 | \n
| 6 | \nDeberta v3 base | \n0.4515 / 0.43 | \nMax len =470, Mean pooling, AWP, val_steps = 250, reinit layers | \n
| 7 | \nDeberta xlarge | \n0.4534 / 0.43 | \nCLS token, val steps =20 | \n
| 8 | \ndeberta v3 large | \n0.4495 / 0.43 | \nCLS Token, AWP, val steps = 250, max len 768 | \n
| 9 | \nElectra large | \n0.4545 | \nMeanpool, LSTM | \n
| 10 | \ndeepset/deberta-v3-bbase-squad2 | \n0.4522 | \n\n |
| 11 | \nLuke Large | \n0.4551 | \nMax len 512, Mean pooling, bidirectional-LSTM | \n
| 12 | \nDeberta v3 large | \n0.4467 | \n\n |
| 13 | \nDeberta v3 base | \n0.4471 | \nAutoModelForTokenClassification, last 4 layer concatenate, mean pooling,max_len=768 | \n
| 14 | \nDeberta v3 large | \n0.4469 | \nPL Model, 2 seed averaged, CLS Token | \n
| 16 | \nCocolm large | \n0.4568 | \nNo [PARAGRAPH], CLS Token, AWP | \n
| 17 | \nGPT2 Medium | \n0.4648 | \nmax_len=1024, Mean pooling, SWA | \n
| 18 | \nLongformer Large | \n0.4592 | \nmax_len=1536, Mean pooling | \n
| 19 | \nBart Large Squad | \n0.4649 | \nmax_len=1024, Mean pooling | \n
| 20 | \nDistill Bart CNN | \n0.4607 | \nmax_len=1024, Mean pooling | \n
| 21 | \nOPT-350M | \n0.4639 | \nmax_len=1536, Mean pooling | \n
| 22 | \nBart Base | \n0.4788 | \nmax_len=1024, Mean pooling, SWA | \n
| 23 | \nT5 Large | \n0.4717 | \nmax_len=1024, Mean pooling | \n
| 24 | \nDeberta v3 base | \n0.4490 | \nmax_len=512 | \n
| 25 | \nDeberta v3 base | \n0.4484 | \nmax_len =384 | \n
| 26 | \nDeberta v3 large | \n0.4498 | \nmax_len=1429, Mean pooling, Pseudo Labeling | \n
Preprocessing: - replacing “\\n\\n” with [PARAGRAPH] There are about 17.5K instances of \\n\\n in the dataset. Deberta v3 large tokenizer simply ignores these. So we decided to create one additional special token by replacing \\n\\n with [PARAGRAPH]. Also resized embeddings to take care of this one extra token. Just this change moved deberta v3 large from 0.44 to 0.43.
Knowledge Distillation: We had one deberta v3 x small model with knowledge distillation.
\nWe decided to distill knowledge i.e. (get OOF data) from an ensemble of 9 models and utilize these to train the model. This is only utilizing current competition data not previous feedback data.
Pseudo labels (PL)
\nThis was challenging to determine if pseudo labels are working or not. We tried generating labels with a few different models to determine if it’s working. Eventually we created two seed deberta v3 large models which seemed more stable than trying to determine with single deberta v3 large. Some of the models in our ensemble were with PL.
\nWe also did pseudo labeling inferred from the tensorflow Deberta v3 base model , which is trained with layer wise learning rate decay (LLRD) technique and used those PL inferred data into some of our models , which helps in improving the cv as well as LB.
Model Architecture:
\nJust CLS token, Mean pooling, Mean pooling + CLS token, MaskAddedAttentionHead, MaskAddedAttentionHead + CLS token(AutoModel and AutoModelForTokenClassification) , LSTM
Training strategies:
\nAWP worked, different max len per epoch, validation steps 20-250 were utilized for different models. Some of the models were trained using end of epoch validation as well.
\nCV was calculated with 5 folds models.
Diversity in ensemble:
\nFolds - By Abhishek - https://www.kaggle.com/code/abhishek/multi-label-stratified-folds
\nWe had ensemble of total 27 models (runtime ~7 hrs) - 7 Deberta v3 large models, 5 Deberta v3 base, Deberta v2 xlarge, Deberta xlarge, Deberta v3 small, Luke Large, Electra, Cocolm, Deberta v3 base squad, GPT2 medium, Longformer Large, Bart Large Squad, Distill Bart CNN, OPT 350M, Bart base, T5 large
Ridge Stacking
\nWe used Ridge for stacking our models, there were 2 deberta large (two seed models) which we combined so that generated only 1 ridge coefficient and it helped in ensemble.
Post processing
\nWe used “Nelder-mead” https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html for post processing - divided the predictions into 20 bins and multiplied the resulting array with predictions.
\nThis post processing is based on commonlit's 2nd place solution method.
\nhttps://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328
Our final ensemble is here - https://www.kaggle.com/code/aman1391/fb3-8th-place-solution-ensemble-27-models?scriptVersionId=112097448
\nMask Language modeling: - Tried many different ways for MLM to work, unsuccessful attempts. Tried to match similarity between FB3 data and previous feedback data for MLM which also didn’t work.
Other models: - Various other models from HuggingFace didn’t work. A few did work however it wouldn’t work with our ensemble so they were unutilized.
Different loss functions: - Focal Loss, Ordinal regression from public kernel, Asymmetric loss - Asymmetric loss did work with a few ensembles but it wasn’t part of our best submission.
Ensemble with weights: - In the last 2 days, we tried ensemble with weight tuning but Ridge did give us better results. This needed more tuning or work for us to feel confident in selecting a submission.
Many different model architectures, hyperparameters, token dropout, 4 fold model, dividing folds by number of tokens.
Compute: A100s, 4090 (3090 and Colab Pro+ prior to 4090)
Please see this post for a single model (average 2 seeds) gold solution - https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369368 and corresponding training code
Thanks to our hosts for hosting the competition and to teammates. Thanks also to the Kaggler's for sharing their helpful notebooks and discussions.
\nThank you and Congratulations to my team mates - @aman1391, @kanbehmw, @fightingmuscle
\nTeam Members
\n@rashmibanthia
\n@aman1391
\n@kanbehmw
\n@fightingmuscle
@rashmibanthia
\nThank you for your solution! and congratulation for being competition master!!!
\nI'm so glad to work with you!!!!
Likewise, It was pleasure working with you and Congratulations to you as well 🎉🎉
", "rawMarkdown": "Likewise, It was pleasure working with you and Congratulations to you as well 🎉🎉", "votes": 3 } ] }, { "id": 2054678, "postDate": "2022-12-04T10:39:46.287Z", "content": "Thank you for sharing your valuable solution RB.
", "rawMarkdown": "Thank you for sharing your valuable solution RB.", "votes": 1 }, { "id": 2051853, "postDate": "2022-12-01T16:50:32.610Z", "content": "@rashmibanthia Thanks for the solution!
\n\n\nSo we decided to create one additional special token by replacing \\n\\n with [PARAGRAPH]. Also resized embeddings to take care of this one extra token.
\n
Would you mind sharing a code snippet on how to resize deberta embeddings to take care of this token exactly? Would like to try this but it's not obvious to me how to implement resizing
", "rawMarkdown": "@rashmibanthia Thanks for the solution!\n\n> So we decided to create one additional special token by replacing \\n\\n with [PARAGRAPH]. Also resized embeddings to take care of this one extra token.\n\nWould you mind sharing a code snippet on how to resize deberta embeddings to take care of this token exactly? Would like to try this but it's not obvious to me how to implement resizing", "votes": 2, "replies": [ { "id": 2051900, "postDate": "2022-12-01T17:39:38.197Z", "content": "Code for adding special token:
\nCFG.tokenizer = AutoTokenizer.from_pretrained(model)\nCFG.tokenizer.add_special_tokens({: []})\nThen, within model class, you need self.model.resize_token_embeddings(len(CFG.tokenizer)) (see below code)
(nn.Module):\n ():\n ().__init__()\n\n self.model_config = AutoConfig.from_pretrained(model_name)\n\n self.model_config.update(\n {\n : ,\n : ,\n : ,\n : CFG.num_labels,\n }\n )\n\n self.model = AutoModel.from_pretrained(model_name, config=self.model_config)\n self.model.resize_token_embeddings((CFG.tokenizer))\n\n self.output = nn.Linear(self.model_config.hidden_size, CFG.num_labels)\n\n\n ():\n transformer_out = self.model(input_ids=ids, attention_mask=mask)\n sequence_output = transformer_out.last_hidden_state[:, , :]\n\n logits = self.output(sequence_output)\n\n logits \n