Acknowledgement
\nPhew, the shakeup was massive! :) I want to thank the organizers, the Kaggle community, and the Kaggle team for this competition. Competitions like this are great opportunities for me to learn new things and update my machine-learning knowledge.
\nI also want to thank my teammates, @runningz, @kunihikofurugori, and @habedi, for their efforts and cooperation in this competition.
\nQuick Summary
\nOur final submission was a blend of two submissions files:
\niterativestratification with the number of folds set to four and the seed set to 42.10th Place Solution (Additional content)
\n@kainsama already gave an overview of our team’s top solution. Nevertheless, I added some extra information in the format mentioned in this post.
\nOverview
\nOur top solution is the blend of the output of two ensembles. The first ensemble consists of three models: one deberta-v3-base model and two deberta-v3-large. Models were tuned separately with different hyper-parameters, such as different learning rates. Furthermore, the data used to tune them were augmented with pseudo-labels created using a separate good-performing single model. The predictions by the deberta-v3-base were given a weight of 0.4, and the predictions by the two remaining models were given a weight of 0.3 (per model) to make the out of the first ensemble. Moreover, detailed information about the second ensemble can be found in the parent post (top).
\nMain things that worked and didn’t work
\nAlthough pseudo-labelling was used in the final solution, it didn’t perform best in most cases. (See this notebook; PB:~0.438 in single debate-v3-base.) Moreover, hyper-parameter tuning using Optuna also didn’t significantly boost the score on the CV. (See this notebook; an improvement of about 0.001). For the models used in the stacking, their CVs ranged from 0.45 to 0.46. The models were trained and tuned separately, so model diversity was influential in improving the stacked models' average CV, which was about 0.447.
\nThanks and Acknowledgements
\nPlease see @kainsama’s quick summary post (parent post; top).
Team Members
\n@habedi
\n@runningz
\n@kainsama
\n@kunihikofurugori
Congratulations @kainsama ! It seems like your team had a lot of base/small models. Did you try larger models?
", "votes": 1, "replies": [] }, { "id": 2049336, "author_name": "happymentee", "author_url": "", "post_date": "2022-11-30T04:06:45.327000", "content": "Great Work! Congrats, and Thanks for your sharing!
\nI am new to Kaggle, so I want to reimplement your solution. So what is the minimum GPU needed to train what your team did?
We had access to an NVIDIA RTX 3090 GPU and a small cluster of V100 GPUs. Plus, Kaggle environments and Google Colab Pro. I think having an RTX 3090 is a good minimum requirement. (For training or tuning larger language models you will need a cluster probbaly.)
", "votes": 2, "replies": [] }, { "id": 2049422, "author_name": "happymentee", "author_url": "", "post_date": "2022-11-30T06:16:44.633000", "content": "Thanks for your reply.
\nDoes GPU T4x2 only enough to train your team's model?
\nDoes it take a long training time, or is it unable to train?
Congratulation!
\nInteresting solution. I did not spent much time in fb3, and not join discussion, so Idk I can update regressor. It's look like ML regressor can align linear output. Before this time, I still think CNN/RNN/ML cannot learn well to MLM's embedding output, use pooling/attention + linear that' ok. BUT maybe that only classification, regression is different.
LB is inaccurate
\nCV is inaccurate
\nMight come true the old saying: Older ginger is more pungent.
\n\nCV is inaccurate
\n
It wasnt inaccurate though. I am not sure how you were measuring this but CV and private LB had good correlation for us.
", "votes": 1, "replies": [] } ] } ], "comment_count": 8, "messages": [ { "id": 2049179, "postDate": "2022-11-30T01:33:39.093Z", "content": "Acknowledgement
\nPhew, the shakeup was massive! :) I want to thank the organizers, the Kaggle community, and the Kaggle team for this competition. Competitions like this are great opportunities for me to learn new things and update my machine-learning knowledge.
\nI also want to thank my teammates, @runningz, @kunihikofurugori, and @habedi, for their efforts and cooperation in this competition.
\nQuick Summary
\nOur final submission was a blend of two submissions files:
\niterativestratification with the number of folds set to four and the seed set to 42.10th Place Solution (Additional content)
\n@kainsama already gave an overview of our team’s top solution. Nevertheless, I added some extra information in the format mentioned in this post.
\nOverview
\nOur top solution is the blend of the output of two ensembles. The first ensemble consists of three models: one deberta-v3-base model and two deberta-v3-large. Models were tuned separately with different hyper-parameters, such as different learning rates. Furthermore, the data used to tune them were augmented with pseudo-labels created using a separate good-performing single model. The predictions by the deberta-v3-base were given a weight of 0.4, and the predictions by the two remaining models were given a weight of 0.3 (per model) to make the out of the first ensemble. Moreover, detailed information about the second ensemble can be found in the parent post (top).
\nMain things that worked and didn’t work
\nAlthough pseudo-labelling was used in the final solution, it didn’t perform best in most cases. (See this notebook; PB:~0.438 in single debate-v3-base.) Moreover, hyper-parameter tuning using Optuna also didn’t significantly boost the score on the CV. (See this notebook; an improvement of about 0.001). For the models used in the stacking, their CVs ranged from 0.45 to 0.46. The models were trained and tuned separately, so model diversity was influential in improving the stacked models' average CV, which was about 0.447.
\nThanks and Acknowledgements
\nPlease see @kainsama’s quick summary post (parent post; top).
Team Members
\n@habedi
\n@runningz
\n@kainsama
\n@kunihikofurugori
Congratulations @kainsama ! It seems like your team had a lot of base/small models. Did you try larger models?
", "rawMarkdown": "Congratulations @kainsama ! It seems like your team had a lot of base/small models. Did you try larger models? ", "votes": 1 }, { "id": 2049336, "postDate": "2022-11-30T04:06:45.327Z", "content": "Great Work! Congrats, and Thanks for your sharing!
\nI am new to Kaggle, so I want to reimplement your solution. So what is the minimum GPU needed to train what your team did?
We had access to an NVIDIA RTX 3090 GPU and a small cluster of V100 GPUs. Plus, Kaggle environments and Google Colab Pro. I think having an RTX 3090 is a good minimum requirement. (For training or tuning larger language models you will need a cluster probbaly.)
", "rawMarkdown": "We had access to an NVIDIA RTX 3090 GPU and a small cluster of V100 GPUs. Plus, Kaggle environments and Google Colab Pro. I think having an RTX 3090 is a good minimum requirement. (For training or tuning larger language models you will need a cluster probbaly.)", "votes": 2 }, { "id": 2049422, "postDate": "2022-11-30T06:16:44.633Z", "content": "Thanks for your reply.
\nDoes GPU T4x2 only enough to train your team's model?
\nDoes it take a long training time, or is it unable to train?
Congratulation!
\nInteresting solution. I did not spent much time in fb3, and not join discussion, so Idk I can update regressor. It's look like ML regressor can align linear output. Before this time, I still think CNN/RNN/ML cannot learn well to MLM's embedding output, use pooling/attention + linear that' ok. BUT maybe that only classification, regression is different.
LB is inaccurate
\nCV is inaccurate
\nMight come true the old saying: Older ginger is more pungent.
\n\nCV is inaccurate
\n
It wasnt inaccurate though. I am not sure how you were measuring this but CV and private LB had good correlation for us.
", "rawMarkdown": "> CV is inaccurate\n\nIt wasnt inaccurate though. I am not sure how you were measuring this but CV and private LB had good correlation for us. \n", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:58:03Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9253614, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/team-meliora-10th-place-solution-my-quick-write-up" }