![\"Screen](\"https://user-images.githubusercontent.com/38504303/204792257-95877fde-4e82-45cd-852d-0df26a138adc.png\")
First of all, I'd like to thank kaggle team, host for hosting this competition and thank many other participants to share codes, ideas and datasets.
\nI'm really happy to get solo gold.
\nMy final submission is weighted avg model which is 0.5 * 11 models nelder-mead blended one + 11 models bayseian ridge stacking one.
\nEach models are trained with various hyperparameter such as seed, num_fold, max_length, and so on.
\n
My final Code is here. :
\nhttps://www.kaggle.com/code/kaerunantoka/fb3-private-14th-solution-code?scriptVersionId=112497983
I'm sorry that a lot of my notebooks are public because everyone can run this code.
My training repo is here. :
\n\n[High Impact] Idea
\nTrust CV
\nnelder-mead optimized weighted blending
\nlonger max length (1536) and use head-and-tail tokens
\nfreeze embedding layer and first 2 layers
\ndef freeze(module):\n for parameter in module.parameters():\n parameter.requires_grad = False\n\nif self.cfg.freezing:\n freeze(self.model.embeddings)\n freeze(self.model.encoder.layer[:2])\nevalulate twice per epoch
[Medium Impact] Idea
\nI replaced svr with xgboost and added final model. This is contributed to my model diversity.
add special token
\n\ndef setup_tokenizer(CFG):\n CFG.tokenizer.add_tokens([f\"\\n\"], special_tokens=True)\n CFG.tokenizer.add_tokens([f\"\\r\"], special_tokens=True)\nCV Strategy
\nModels
notebook
\ndiscussion
\ndataset
\nThanks and Acknowledgements:
\nFinally, I'd like to thank Kaggle, host for hosting such an iteresting competition and all who participated in lively discussions.
\nI wish my solution helps you in your future NLP competitions!
I am glad that my dataset is a little help to you. Congratulations on solo gold !
", "votes": 1, "replies": [ { "id": 2051698, "author_name": "kaerururu", "author_url": "", "post_date": "2022-12-01T15:23:00.640000", "content": "Thank you for sharing great dataset. Without your datasets, both my CV and LB are not improved !!
", "votes": 1, "replies": [] } ] }, { "id": 2049887, "author_name": "Mark Baushenko", "author_url": "", "post_date": "2022-11-30T12:03:37.950000", "content": "Congratulations on the gold medal, it's very nice to compete with such cool people like you.
", "votes": 0, "replies": [ { "id": 2049912, "author_name": "kaerururu", "author_url": "", "post_date": "2022-11-30T12:18:46.973000", "content": "Thank you for your encouraging comment :)
", "votes": 0, "replies": [] } ] }, { "id": 2049860, "author_name": "Naoism", "author_url": "", "post_date": "2022-11-30T11:40:32.160000", "content": "Thanks for sharing your solution and congratulations on the solo gold!! I was surprised that you didn't use pseudo labeling !
\nCan I ask 2 questions ?
Thanks for comment and question !!
\nMy answers are below.
1 . I'm sorry that my figure confused you. I fixed my figure. I got oof from both pretrained embedding models and finetuned models.
\n2 . Yes, there are a little leakage as you thought. Using the same split data (from the same seed, num_folds, … and so on) is better than as I did. By the way, we sometimes use seed averaging to avoid overfitting. I did it with that intention.
Thank you for your quick reply and update the figure. Perhaps it was important to use different seed not to avoid overfitting but also training data diversity…
", "votes": 0, "replies": [] }, { "id": 2049933, "author_name": "kaerururu", "author_url": "", "post_date": "2022-11-30T12:32:20.743000", "content": "Yes, I thought more diversity is important than more precise experiment in such a shakey cometition.
", "votes": 0, "replies": [] } ] }, { "id": 2049623, "author_name": "Shobhit Upadhyaya", "author_url": "", "post_date": "2022-11-30T08:42:50.177000", "content": "congrats on the solo gold @kaerunantoka
", "votes": 0, "replies": [ { "id": 2049659, "author_name": "kaerururu", "author_url": "", "post_date": "2022-11-30T09:05:55.177000", "content": "thank you for your comment !!
", "votes": 0, "replies": [] } ] }, { "id": 2050284, "author_name": "Quadeer Shaikh", "author_url": "", "post_date": "2022-11-30T16:21:40.200000", "content": "Congratulations ! thank you for sharing
", "votes": 0, "replies": [] } ], "comment_count": 11, "messages": [ { "id": 2049483, "postDate": "2022-11-30T07:18:40.043Z", "content": "First of all, I'd like to thank kaggle team, host for hosting this competition and thank many other participants to share codes, ideas and datasets.
\nI'm really happy to get solo gold.
\nMy final submission is weighted avg model which is 0.5 * 11 models nelder-mead blended one + 11 models bayseian ridge stacking one.
\nEach models are trained with various hyperparameter such as seed, num_fold, max_length, and so on.
\n
My final Code is here. :
\nhttps://www.kaggle.com/code/kaerunantoka/fb3-private-14th-solution-code?scriptVersionId=112497983
I'm sorry that a lot of my notebooks are public because everyone can run this code.
My training repo is here. :
\n\n[High Impact] Idea
\nTrust CV
\nnelder-mead optimized weighted blending
\nlonger max length (1536) and use head-and-tail tokens
\nfreeze embedding layer and first 2 layers
\ndef freeze(module):\n for parameter in module.parameters():\n parameter.requires_grad = False\n\nif self.cfg.freezing:\n freeze(self.model.embeddings)\n freeze(self.model.encoder.layer[:2])\nevalulate twice per epoch
[Medium Impact] Idea
\nI replaced svr with xgboost and added final model. This is contributed to my model diversity.
add special token
\n\ndef setup_tokenizer(CFG):\n CFG.tokenizer.add_tokens([f\"\\n\"], special_tokens=True)\n CFG.tokenizer.add_tokens([f\"\\r\"], special_tokens=True)\nCV Strategy
\nModels
notebook
\ndiscussion
\ndataset
\nThanks and Acknowledgements:
\nFinally, I'd like to thank Kaggle, host for hosting such an iteresting competition and all who participated in lively discussions.
\nI wish my solution helps you in your future NLP competitions!
\n\nMy final Code is here. : \n- https://www.kaggle.com/code/kaerunantoka/fb3-private-14th-solution-code?scriptVersionId=112497983\n\n- I'm sorry that a lot of my notebooks are public because everyone can run this code.\n\nMy training repo is here. :\n- https://github.com/osuossu8/FeedbackPrize3\n\n\n### What Worked\n- [High Impact] Idea\n - Trust CV\n - I added new model to ensemble in case oof cv score is improved.\n - [rapids SVR](https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x)\n\n - [nelder-mead optimized weighted blending](https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html#optimize-minimize-neldermead)\n - I heard this method first time in [this previous nlp compatitions solution.](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328)\n\n - longer max length (1536) and use head-and-tail tokens\n - many public notebooks used 512 max_length\n - however max token length in train full_text is around 1430.\n - I decided to use longer token length for my training.\n\n - freeze embedding layer and first 2 layers\n - ```\n def freeze(module):\n for parameter in module.parameters():\n parameter.requires_grad = False\n \n if self.cfg.freezing:\n freeze(self.model.embeddings)\n freeze(self.model.encoder.layer[:2])\n ```\n - evalulate twice per epoch\n\n- [Medium Impact] Idea\n - I replaced svr with xgboost and added final model. This is contributed to my model diversity.\n\n - add special token \n - I noticed full_text has many \"\\n\\n\".\n - ```\n def setup_tokenizer(CFG):\n CFG.tokenizer.add_tokens([f\"\\n\"], special_tokens=True)\n CFG.tokenizer.add_tokens([f\"\\r\"], special_tokens=True)\n ```\n\n### What Didn’t Work\n- Idea\n - Use my finetuned embeddings when I try rapids svr method.\n - Use Universal Sentence Encoder Embedding \n - Use text features as meta feature when I finetuneing hugging face models and my 2nd stage stacking models.\n\n\n### Additional Context:\n\n- CV Strategy\n - I used [abiheshark's cv strategy](https://www.kaggle.com/code/abhishek/multi-label-stratified-folds) with 5, 10 folds various seed.\n\n- Models\n\n| Name | CV | max length | num_fold | link |\n| ---- | ---- | ---- | ---- | ---- |\n| deberta-v3-large | 0.4584 | 1536 | 5 | https://github.com/osuossu8/FeedbackPrize3/blob/main/exp/024.py |\n| deberta-v3-large | 0.4585 | 1536 | 5 | https://github.com/osuossu8/FeedbackPrize3/blob/main/exp/026.py |\n| deberta-large | 0.4617 | 512 | 5 | https://www.kaggle.com/code/kaerunantoka/feedback3-n004 |\n| roberta-base | 0.4622 | 512 | 5 | https://www.kaggle.com/code/kaerunantoka/feedback3-n005 | \n| longformer-large-4096 | 0.4670 | 512 | 5 | https://www.kaggle.com/code/kaerunantoka/feedback3-n009 |\n| deberta-v3-base | 0.4554 | 512 | 5 | [original](https://www.kaggle.com/code/batprem/deberta-layerwiselr-lastlayerreini-infer), [my copy to save oof](https://www.kaggle.com/code/kaerunantoka/fb3-public-save-oof) |\n| deberta-v3-large | 0.4670 | 512 | 10 | https://www.kaggle.com/datasets/kojimar/20221012-123357-deberta-v3-large |\n| deberta-v2-xlarge-mnli | 0.4675 | 512 | 10 | https://www.kaggle.com/datasets/kojimar/0919-deberta-v2-xlarge-mnli |\n| deberta-v3-large | 0.4548 | 512 | 10 | https://www.kaggle.com/datasets/kojimar/0926-deberta-v3-large-unscale |\n| pretrained hugging face embedding & rapids SVR | 0.4499 | - | 15 | [original](https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x), [my copy to save oof and models](https://www.kaggle.com/code/kaerunantoka/fb3-public-svr-024-infer) |\n| pretrained hugging face embedding & XGB | 0.4552 | - | 15 | https://www.kaggle.com/code/kaerunantoka/fb3-pretrained-bert-family-xgb-train |\n| bayseian ridge | 0.4446 | - | 10 | https://www.kaggle.com/code/kaerunantoka/fb3-stacking-nelder-mead |\n\n### Important Citations:\n- notebook\n - https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\n - https://www.kaggle.com/code/batprem/deberta-layerwiselr-lastlayerreini-infer\n - https://www.kaggle.com/code/quangphm/lb-0-43-simple-ensemble-deberta-base-svr\n\n- discussion\n - https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328\n\n- dataset\n - https://www.kaggle.com/datasets/kojimar/20221012-123357-deberta-v3-large\n - https://www.kaggle.com/datasets/kojimar/0926-deberta-v3-large-unscale\n - https://www.kaggle.com/datasets/kojimar/0919-deberta-v2-xlarge-mnli\n\n\nThanks and Acknowledgements:\n\nFinally, I'd like to thank Kaggle, host for hosting such an iteresting competition and all who participated in lively discussions.\nI wish my solution helps you in your future NLP competitions!",
"votes": 43
},
{
"id": 2051480,
"postDate": "2022-12-01T13:04:00.103Z",
"content": "I am glad that my dataset is a little help to you. Congratulations on solo gold !
", "rawMarkdown": "I am glad that my dataset is a little help to you. Congratulations on solo gold !", "votes": 1, "replies": [ { "id": 2051698, "postDate": "2022-12-01T15:23:00.640Z", "content": "Thank you for sharing great dataset. Without your datasets, both my CV and LB are not improved !!
", "rawMarkdown": "Thank you for sharing great dataset. Without your datasets, both my CV and LB are not improved !!", "votes": 1 } ] }, { "id": 2049887, "postDate": "2022-11-30T12:03:37.950Z", "content": "Congratulations on the gold medal, it's very nice to compete with such cool people like you.
", "rawMarkdown": "Congratulations on the gold medal, it's very nice to compete with such cool people like you.", "replies": [ { "id": 2049912, "postDate": "2022-11-30T12:18:46.973Z", "content": "Thank you for your encouraging comment :)
", "rawMarkdown": "Thank you for your encouraging comment :)" } ] }, { "id": 2049860, "postDate": "2022-11-30T11:40:32.160Z", "content": "Thanks for sharing your solution and congratulations on the solo gold!! I was surprised that you didn't use pseudo labeling !
\nCan I ask 2 questions ?
Thanks for comment and question !!
\nMy answers are below.
1 . I'm sorry that my figure confused you. I fixed my figure. I got oof from both pretrained embedding models and finetuned models.
\n2 . Yes, there are a little leakage as you thought. Using the same split data (from the same seed, num_folds, … and so on) is better than as I did. By the way, we sometimes use seed averaging to avoid overfitting. I did it with that intention.
Thank you for your quick reply and update the figure. Perhaps it was important to use different seed not to avoid overfitting but also training data diversity…
", "rawMarkdown": "Thank you for your quick reply and update the figure. Perhaps it was important to use different seed not to avoid overfitting but also training data diversity..." }, { "id": 2049933, "postDate": "2022-11-30T12:32:20.743Z", "content": "Yes, I thought more diversity is important than more precise experiment in such a shakey cometition.
", "rawMarkdown": "Yes, I thought more diversity is important than more precise experiment in such a shakey cometition." } ] }, { "id": 2049623, "postDate": "2022-11-30T08:42:50.177Z", "content": "congrats on the solo gold @kaerunantoka
", "rawMarkdown": "congrats on the solo gold @kaerunantoka ", "replies": [ { "id": 2049659, "postDate": "2022-11-30T09:05:55.177Z", "content": "thank you for your comment !!
", "rawMarkdown": "thank you for your comment !!" } ] }, { "id": 2050284, "postDate": "2022-11-30T16:21:40.200Z", "content": "Congratulations ! thank you for sharing
", "rawMarkdown": "Congratulations ! thank you for sharing" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:58:22Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9091281, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/kaerururu-13th-place-solution" }