First, I would like to thank the competition hosts for organizing this competition and also the participants.
\nAs a novice regarding NLP competitions and Kaggle, everyone's codes and discussions have been very helpful to me.
\nLuckily, I was able to get my first medal.
I did a weighted average ensemble of \"models trained on Feedback3 data\" + \"models trained on a mixture of Feedback1 pseudo-labeled data and Feedback3 data\" + \"models trained on Rapids SVR using my finetuned model embedding\" + \"models trained on Rapids SVR using huggingface pre-trained model embedding.\"
\nThe CV strategy used MultiLabelStratifiedKFold with K = 5.
\nIncidentally, my use of this MultiLabelStratifiedKFold was buggy according to Discussion (see: https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/368437), but the correlation between CV and LB was good.
Several steps were taken to improve the model and will be explained.
\nFirst I tried to develop a good single model. After tuning by hand, the following configuration worked well.
\nMany people have achieved Public 0.43 with a single model, and that was my goal, but I just couldn't achieve it, so I decided to rely on an ensemble.
\nThe highest Local CV at this step was 0.4502 for Deberta-xlarge (it was not Deberta-v3 series)
\nI trained the following models in the above configuration and Pseudo Labeled the train and test data for feedback prize1 with a weighted average ensemble of these models.
\nI was careful to avoid leaks and labeled each of the 5 folds to generate a pseudo label. I also removed duplicates in feedback 1 data and feedback 3 data. It seems that preventing this leak was quite important in this competition.
\nUsing the pretrained-model developed in step 1, I mixed the data from feedback 1 with the data from feedback 3 and fine-tuned it just for one epoch.
\nThe highest Local CV at this step was 0.4472 for Deberta-xlarge. Here, the Public score for the single model reached 0.43, and furthermore, the Public Score for the ensemble model entered the bronze medal zone.
\nFirst, I applied Rapids SVR to the embedding of the model developed in step 2, and although the CV was not good, both CV and LB increased when ensemble with the previous models, so I adopted it.
\nNext, I applied the model to a pretrained model available on huggingface. This model was also adopted since both CV and LB increased when ensemble with the previous model. At this stage, the ensemble model was able to enter the Public silver medal zone.
\nThe ensemble is a weighted average. The weights are optimized using the Nelder-Mead method. Since there were six targets in this case, I adjusted the weights for each target.
\nI also used the same weights for all 5 folds to prevent overfit to the local CV.
\nThe final ensemble was the following model.
\n| Model | \nLocal CV Score | \n
|---|---|
| deberta_v3_base_4096 | \n0.4536 | \n
| deberta_large_512 | \n0.4513 | \n
| muppet_large_512 | \n0.4571 | \n
| deberta_xlarge_512 | \n0.4502 | \n
| deberta_v3_base_512_pseudo | \n0.4489 | \n
| deberta_v3_large_4096_pseudo | \n0.448 | \n
| deberta_xlarge_512_pseudo | \n0.4472 | \n
| deberta_v3_base_512_pseudo_svr | \n0.4529 | \n
| deberta_v3_large_4096_pseudo_svr | \n0.4531 | \n
| deberta_xlarge_512_pseudo_svr | \n0.4538 | \n
| deberta_v3_large_4096_svr | \n0.4519 | \n
| deberta_large_512_svr | \n0.4553 | \n
| deberta_v3_base_4096_svr | \n0.4542 | \n
| ensemble | \n0.4436 | \n
In fact, I trained many more models, but these models were chosen as a result of manual and automatic selections.
\nFinally, I did an all data train using the procedure introduced here and took the average of the predictions of the 5 fold model and the all data train model as the final sub. I was able to choose the last sub without hesitation because Local CV and LB were quite correlated and Best Local CV = Best Public LB.
\nAgain, thank you to the organizers and participants of this competition, and I will continue to work hard to become a kaggle expert and a master.
", "comments": [ { "id": 2050267, "author_name": "Mark Baushenko", "author_url": "", "post_date": "2022-11-30T16:09:29.043000", "content": "Congrats u! Nice work🙌
", "votes": 1, "replies": [ { "id": 2050296, "author_name": "moritake04", "author_url": "", "post_date": "2022-11-30T16:30:56.510000", "content": "Thank you very much. It was a fun competition. Good luck to each other!
", "votes": 0, "replies": [] } ] } ], "comment_count": 2, "messages": [ { "id": 2050190, "postDate": "2022-11-30T15:21:14.877Z", "content": "First, I would like to thank the competition hosts for organizing this competition and also the participants.
\nAs a novice regarding NLP competitions and Kaggle, everyone's codes and discussions have been very helpful to me.
\nLuckily, I was able to get my first medal.
I did a weighted average ensemble of \"models trained on Feedback3 data\" + \"models trained on a mixture of Feedback1 pseudo-labeled data and Feedback3 data\" + \"models trained on Rapids SVR using my finetuned model embedding\" + \"models trained on Rapids SVR using huggingface pre-trained model embedding.\"
\nThe CV strategy used MultiLabelStratifiedKFold with K = 5.
\nIncidentally, my use of this MultiLabelStratifiedKFold was buggy according to Discussion (see: https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/368437), but the correlation between CV and LB was good.
Several steps were taken to improve the model and will be explained.
\nFirst I tried to develop a good single model. After tuning by hand, the following configuration worked well.
\nMany people have achieved Public 0.43 with a single model, and that was my goal, but I just couldn't achieve it, so I decided to rely on an ensemble.
\nThe highest Local CV at this step was 0.4502 for Deberta-xlarge (it was not Deberta-v3 series)
\nI trained the following models in the above configuration and Pseudo Labeled the train and test data for feedback prize1 with a weighted average ensemble of these models.
\nI was careful to avoid leaks and labeled each of the 5 folds to generate a pseudo label. I also removed duplicates in feedback 1 data and feedback 3 data. It seems that preventing this leak was quite important in this competition.
\nUsing the pretrained-model developed in step 1, I mixed the data from feedback 1 with the data from feedback 3 and fine-tuned it just for one epoch.
\nThe highest Local CV at this step was 0.4472 for Deberta-xlarge. Here, the Public score for the single model reached 0.43, and furthermore, the Public Score for the ensemble model entered the bronze medal zone.
\nFirst, I applied Rapids SVR to the embedding of the model developed in step 2, and although the CV was not good, both CV and LB increased when ensemble with the previous models, so I adopted it.
\nNext, I applied the model to a pretrained model available on huggingface. This model was also adopted since both CV and LB increased when ensemble with the previous model. At this stage, the ensemble model was able to enter the Public silver medal zone.
\nThe ensemble is a weighted average. The weights are optimized using the Nelder-Mead method. Since there were six targets in this case, I adjusted the weights for each target.
\nI also used the same weights for all 5 folds to prevent overfit to the local CV.
\nThe final ensemble was the following model.
\n| Model | \nLocal CV Score | \n
|---|---|
| deberta_v3_base_4096 | \n0.4536 | \n
| deberta_large_512 | \n0.4513 | \n
| muppet_large_512 | \n0.4571 | \n
| deberta_xlarge_512 | \n0.4502 | \n
| deberta_v3_base_512_pseudo | \n0.4489 | \n
| deberta_v3_large_4096_pseudo | \n0.448 | \n
| deberta_xlarge_512_pseudo | \n0.4472 | \n
| deberta_v3_base_512_pseudo_svr | \n0.4529 | \n
| deberta_v3_large_4096_pseudo_svr | \n0.4531 | \n
| deberta_xlarge_512_pseudo_svr | \n0.4538 | \n
| deberta_v3_large_4096_svr | \n0.4519 | \n
| deberta_large_512_svr | \n0.4553 | \n
| deberta_v3_base_4096_svr | \n0.4542 | \n
| ensemble | \n0.4436 | \n
In fact, I trained many more models, but these models were chosen as a result of manual and automatic selections.
\nFinally, I did an all data train using the procedure introduced here and took the average of the predictions of the 5 fold model and the all data train model as the final sub. I was able to choose the last sub without hesitation because Local CV and LB were quite correlated and Best Local CV = Best Public LB.
\nAgain, thank you to the organizers and participants of this competition, and I will continue to work hard to become a kaggle expert and a master.
", "rawMarkdown": "First, I would like to thank the competition hosts for organizing this competition and also the participants.\nAs a novice regarding NLP competitions and Kaggle, everyone's codes and discussions have been very helpful to me.\nLuckily, I was able to get my first medal.\n\n### Overview\nI did a weighted average ensemble of \"models trained on Feedback3 data\" + \"models trained on a mixture of Feedback1 pseudo-labeled data and Feedback3 data\" + \"models trained on Rapids SVR using my finetuned model embedding\" + \"models trained on Rapids SVR using huggingface pre-trained model embedding.\"\n\nThe CV strategy used MultiLabelStratifiedKFold with K = 5.\nIncidentally, my use of this MultiLabelStratifiedKFold was buggy according to Discussion (see: https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/368437), but the correlation between CV and LB was good.\n\n- [My inference code](https://www.kaggle.com/code/moritake04/private22nd-final-sub)\n\n### What Worked\nSeveral steps were taken to improve the model and will be explained.\n\n##### Step 1. Train a good single model [Medium Impact]\nFirst I tried to develop a good single model. After tuning by hand, the following configuration worked well.\n\n- Epoch:\n - base model: 4\n - large, xlarge model: 2\n- Learning Rate:\n - base model: encoder -> 1e-5, decoder -> 1e-4\n - large, xlarge model: encoder -> 1e-5, decoder -> 1e-5\n - Using Layerwise LR Decay\n- Batch Size: 8\n- Initialize the last layer\n- Optimizer: AdamW (weight_decay = 1e-2)\n- Scheduler: cosine_schedule_with_warmup (warmup_ratio = 0.1, num_cycles = 0.5)\n- Criterion: SmoothL1Loss\n- Head: Mean Pooling\n- When using the xlarge model, freeze half of the layers\n- gradient_clip_val = 1.0\n- token length = 512 (and 4096 for ensemble, 512 was better)\n\nMany people have achieved Public 0.43 with a single model, and that was my goal, but I just couldn't achieve it, so I decided to rely on an ensemble.\n\nThe highest Local CV at this step was 0.4502 for Deberta-xlarge (it was not Deberta-v3 series)\n\n##### Step 2. Pseudo Labeling [High Impact]\nI trained the following models in the above configuration and Pseudo Labeled the train and test data for feedback prize1 with a weighted average ensemble of these models.\n\n- deberta-v3-base (token_length = 512)\n- deberta-v3-base (token_length = 4096)\n- deberta-v3-large (token_length = 512)\n- deberta-v3-large (token_length = 4096)\n- deberta-large (token_length = 512)\n- deberta-xlarge (token_length = 512)\n- muppet-large (token_length = 512)\n\nI was careful to avoid leaks and labeled each of the 5 folds to generate a pseudo label. I also removed duplicates in feedback 1 data and feedback 3 data. It seems that preventing this leak was quite important in this competition.\n\nUsing the pretrained-model developed in step 1, I mixed the data from feedback 1 with the data from feedback 3 and fine-tuned it just for one epoch.\n\nThe highest Local CV at this step was 0.4472 for Deberta-xlarge. Here, the Public score for the single model reached 0.43, and furthermore, the Public Score for the ensemble model entered the bronze medal zone.\n\n##### Step 3. Rapids SVR [Medium Impact]\nFirst, I applied Rapids SVR to the embedding of the model developed in step 2, and although the CV was not good, both CV and LB increased when ensemble with the previous models, so I adopted it.\n\nNext, I applied the model to a pretrained model available on huggingface. This model was also adopted since both CV and LB increased when ensemble with the previous model. At this stage, the ensemble model was able to enter the Public silver medal zone.\n\n##### Step 4. Ensemble [High Impact]\nThe ensemble is a weighted average. The weights are optimized using the Nelder-Mead method. Since there were six targets in this case, I adjusted the weights for each target.\n\nI also used the same weights for all 5 folds to prevent overfit to the local CV.\n\nThe final ensemble was the following model.\n\n| Model | Local CV Score |\n| ---- | ---- |\n| deberta_v3_base_4096 | 0.4536 |\n| deberta_large_512 | 0.4513 |\n| muppet_large_512 | 0.4571 |\n| deberta_xlarge_512 | 0.4502 |\n| deberta_v3_base_512_pseudo | 0.4489 |\n| deberta_v3_large_4096_pseudo | 0.448 |\n| deberta_xlarge_512_pseudo | 0.4472 |\n| deberta_v3_base_512_pseudo_svr | 0.4529 |\n| deberta_v3_large_4096_pseudo_svr | 0.4531 |\n| deberta_xlarge_512_pseudo_svr | 0.4538 |\n| deberta_v3_large_4096_svr | 0.4519 |\n| deberta_large_512_svr | 0.4553 |\n| deberta_v3_base_4096_svr| 0.4542 |\n| ensemble | 0.4436 |\n\nIn fact, I trained many more models, but these models were chosen as a result of manual and automatic selections.\n\nFinally, I did an all data train using the procedure introduced here and took the average of the predictions of the 5 fold model and the all data train model as the final sub. I was able to choose the last sub without hesitation because Local CV and LB were quite correlated and Best Local CV = Best Public LB.\n\n### What Didn’t Worked\n- MLM\n- AWP\n - I think it is my problem because some people are doing well.\n- Initialization of two or more layers\n- Stacking using predictions or embedded + text metadata (number of misspellings, number of words, number of punctuation marks, etc.)\n- Add newline tokens\n- Heads such as Weighted Layer Pooling, Attention Pooling, 1dcnn, lstm\n\n### Important Citations\n- https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\n- https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference\n- https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\n\n### Thanks and Acknowledgements\nAgain, thank you to the organizers and participants of this competition, and I will continue to work hard to become a kaggle expert and a master.", "votes": 9 }, { "id": 2050267, "postDate": "2022-11-30T16:09:29.043Z", "content": "Congrats u! Nice work🙌
", "rawMarkdown": "Congrats u! Nice work🙌", "votes": 1, "replies": [ { "id": 2050296, "postDate": "2022-11-30T16:30:56.510Z", "content": "Thank you very much. It was a fun competition. Good luck to each other!
", "rawMarkdown": "Thank you very much. It was a fun competition. Good luck to each other!" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:01:21Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9096203, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/moritake04-22nd-place-solution" }