{ "id": 369621, "title": "4th place solutioin", "content": "

Overview

\n

My solution is stacking a total of 22 models. All the models trained in this competition are the same folds (4fold).

\n

\"\"

\n

What Worked

\n\n

What Didn’t Worked

\n\n

Important Citations:

\n\n

Thanks and Acknowledgements:

\n

Thanks to the organizers and Kaggle for a very exciting competition!
\nI will use what I learned here to do my best in the next competition.

\n

Team Members:

\n", "comments": [ { "id": 2050459, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-30T18:46:42.913000", "content": "

Awesome! Congratulations on fantastic solo 4th place finish!

\n
\n

Extracted features from the last 4 layers of the 38 pre-trained models and used forward feature selection to explorate the best SVR

\n
\n

Great SVR model !!

", "votes": 1, "replies": [ { "id": 2051583, "author_name": "hakubishin3", "author_url": "", "post_date": "2022-12-01T14:02:49.123000", "content": "

Thanks for your great sharing!

", "votes": 1, "replies": [] }, { "id": 2051972, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-12-01T19:12:34.960000", "content": "

@shuheigoda after forward feature selection, how many features did you input into SVR?

", "votes": 0, "replies": [] } ] }, { "id": 2465665, "author_name": "phnghiapro", "author_url": "", "post_date": "2023-10-03T07:06:27.367000", "content": "

Thanks you so much for such nice describe your work. I try hiddhidden_dropout_proben_dr = 0 for CommonLit competition and it work like charm

", "votes": 0, "replies": [] }, { "id": 2225979, "author_name": "Marília Prata", "author_url": "", "post_date": "2023-04-18T15:07:04.803000", "content": "

Though I'm late 5 months: congratulations for your amazing accomplishment. Thanks also for sharing your solution.

", "votes": 0, "replies": [] }, { "id": 2056507, "author_name": "Apa", "author_url": "", "post_date": "2022-12-06T07:15:49.570000", "content": "

Hi. Can you provide the training details of the ridge and lightgbm models? I also tried something like this but it didn't work very well

", "votes": 0, "replies": [] }, { "id": 2050633, "author_name": "Man of the year", "author_url": "", "post_date": "2022-11-30T22:11:05.447000", "content": "

@shuheigoda how do you use blending as an input to the red models in the step2? Didn't really understand this part (same with step4).

", "votes": 0, "replies": [ { "id": 2051587, "author_name": "hakubishin3", "author_url": "", "post_date": "2022-12-01T14:05:54.953000", "content": "

In step 3, the predictions for the FB1 data of models trained in step1 and step2 were blended and used as pseudo labels. Step 4 is also the same.

", "votes": 0, "replies": [] } ] } ], "comment_count": 8, "messages": [ { "id": 2050375, "postDate": "2022-11-30T17:39:43.267Z", "content": "

Overview

\n

My solution is stacking a total of 22 models. All the models trained in this competition are the same folds (4fold).

\n

\"\"

\n

What Worked

\n\n

What Didn’t Worked

\n\n

Important Citations:

\n\n

Thanks and Acknowledgements:

\n

Thanks to the organizers and Kaggle for a very exciting competition!
\nI will use what I learned here to do my best in the next competition.

\n

Team Members:

\n", "rawMarkdown": "## Overview\n\nMy solution is stacking a total of 22 models. All the models trained in this competition are the same folds (4fold).\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1403936%2F9a491f9c96cbbe035ed12ee656ba812b%2F2.jpeg?generation=1669829854528737&alt=media)\n\n## What Worked \n\n- [High Impact] Train SVR / Ridge using pre-trained model embeddings\n - Extracted features from the last 4 layers of the 38 pre-trained models and used forward feature selection to explorate the best SVR\n - Trained ridge model with optimal embeddings combination for SVR\n - SVR is my best single model (CV: 0.4467)\n- [High Impact] Pseudo Labeling\n - Two patterns are used for each models\n - Pre-train with pseudo labels and fine-tune with only the given train data afterwards\n - Concatenate pseudo labels with the given train data, and train all this data\n - The amount of FB1 data used as pseudo label\n - use only FB1 data similar to FB3\n - use all FB1 data (but only few model)\n - Repeat over and over again\n- [High Impact] Ridge and LGB stacking\n - CV: 0.4425(Ridge), 0.4443(LGB), 0.4423(Weighted Average of Ridge and LGB)\n - Train Ridge using the predictions of fine-tuned models as input values\n - Train LGB using the predictions and meta-features created by [readability](https://pypi.org/project/readability/)\n- [Middle Impact] Add special token ('\\n')\n - CV increase (0.001)\n- [Low Impact] Shorten sequence length\n - Almost use 512 length\n - Inference time can be reduced\n - For model diversity, train few models with 1500 length\n- [Low Impact] Tips for stable training\n - Full precision training\n - Layer wise learning rate decay\n - Cosine learning rate scheduler\n - Hyperparameter tuning\n\n## What Didn’t Worked \n\n- AWP, FGM\n- Train SVR using fine-tuned model embeddings\n- Last Layer Re-initialization\n- Regular MLM\n- Predict punctuation errors and their statistics use as meta-feature in the stacking model\n - https://www.kaggle.com/code/nulldata/deep-learning-powered-punctuation-corrector/notebook\n- Predict the quality rating of discourse element (FB2 task targets) and their statistics use as meta-feature in the stacking model\n - CV score and public score increase, but private score decrease (0.0002)\n- Fine-tune spelling correction model as convention-specific model\n - https://huggingface.co/oliverguhr/spelling-correction-english-base?text=lets+do+a+comparsion\n- Fine-tune pre-trained weights from top solution in the past competition\n - https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424\n\n## Important Citations: \n\n- [Petfinder.my - Pawpularity Contest 1st place solution](https://www.kaggle.com/c/petfinder-pawpularity-score/discussion/300938)\n- [CommonLiit Readability Prize 2nd place solution](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/258328)\n- [Feedback Prize - Predicting Effective Arguments 1st place solution](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347536)\n- [RAPIDS SVR - CV 0.450 - LB 0.44x](https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x)\n- [FB3 / Deberta-v3-base baseline [train]](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)\n\n## Thanks and Acknowledgements: \n\nThanks to the organizers and Kaggle for a very exciting competition!\nI will use what I learned here to do my best in the next competition.\n\n## Team Members:\n\n- @shuheigoda\n", "votes": 50 }, { "id": 2050459, "postDate": "2022-11-30T18:46:42.913Z", "content": "

Awesome! Congratulations on fantastic solo 4th place finish!

\n
\n

Extracted features from the last 4 layers of the 38 pre-trained models and used forward feature selection to explorate the best SVR

\n
\n

Great SVR model !!

", "rawMarkdown": "Awesome! Congratulations on fantastic solo 4th place finish!\n>Extracted features from the last 4 layers of the 38 pre-trained models and used forward feature selection to explorate the best SVR\n\nGreat SVR model !!", "votes": 1, "replies": [ { "id": 2051583, "postDate": "2022-12-01T14:02:49.123Z", "content": "

Thanks for your great sharing!

", "rawMarkdown": "Thanks for your great sharing!", "votes": 1 }, { "id": 2051972, "postDate": "2022-12-01T19:12:34.960Z", "content": "

@shuheigoda after forward feature selection, how many features did you input into SVR?

", "rawMarkdown": "@shuheigoda after forward feature selection, how many features did you input into SVR?" } ] }, { "id": 2465665, "postDate": "2023-10-03T07:06:27.367Z", "content": "

Thanks you so much for such nice describe your work. I try hiddhidden_dropout_proben_dr = 0 for CommonLit competition and it work like charm

", "rawMarkdown": "Thanks you so much for such nice describe your work. I try hiddhidden_dropout_proben_dr = 0 for CommonLit competition and it work like charm " }, { "id": 2225979, "postDate": "2023-04-18T15:07:04.803Z", "content": "

Though I'm late 5 months: congratulations for your amazing accomplishment. Thanks also for sharing your solution.

", "rawMarkdown": "Though I'm late 5 months: congratulations for your amazing accomplishment. Thanks also for sharing your solution." }, { "id": 2056507, "postDate": "2022-12-06T07:15:49.570Z", "content": "

Hi. Can you provide the training details of the ridge and lightgbm models? I also tried something like this but it didn't work very well

", "rawMarkdown": "Hi. Can you provide the training details of the ridge and lightgbm models? I also tried something like this but it didn't work very well" }, { "id": 2050633, "postDate": "2022-11-30T22:11:05.447Z", "content": "

@shuheigoda how do you use blending as an input to the red models in the step2? Didn't really understand this part (same with step4).

", "rawMarkdown": "@shuheigoda how do you use blending as an input to the red models in the step2? Didn't really understand this part (same with step4).", "replies": [ { "id": 2051587, "postDate": "2022-12-01T14:05:54.953Z", "content": "

In step 3, the predictions for the FB1 data of models trained in step1 and step2 were blended and used as pseudo labels. Step 4 is also the same.

", "rawMarkdown": "In step 3, the predictions for the FB1 data of models trained in step1 and step2 were blended and used as pseudo labels. Step 4 is also the same." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:02:29Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9091395, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/hakubishin3-4th-place-solutioin" }