Firstly, I would like to competition's organizers for such a great competition and congrats to the winners, you did amazing efforts! Secondly, I would like to thank my great teammates: Mark ( @e0xextazy ) , Otabek ( @obatek ), and Sanskar ( @odins0n ) ! Thank you guys for the spent time, experience and emotions!
\nHere, I am going to describe our final solution, which could reach 0.594 and 0.592 on Public Leaderboard and Private Leaderboard leaderboard respectively, thereby getting us to 88th place and giving us the bronze medal (I and Otabek will become Kaggle Experts 😄).
\nUnfortunately, in the middle of the competition, we switched to the AI4Code competition and when we switched back we only had ~1 week to try out some other approaches and ideas. We used only 4x models in our ensemble: one of the high-scoring public notebooks, Token Classification, a model with fine-tuned weights from the previous Feedback Prize competition, and a model with quite strong pre-processing. Such diversity of models permanently improves our Public Leaderboard from 0.610 to 0.594.
\nFor classification, we used grouping by essay_id and stratify by discourse_effectiveness. Multi-label Stratified Cross-validation was used for Token Classification, i.e stratify by counting the number of entries of each discourse_effectiveness in the given essay.
Masked Language Modeling pre-training
Pseudo-Labeling
\nWe tried it yesterday when we had quite a strong ensemble, but pseudo-labels generating and then training took ~20 hours, so we just tried Hard Pseudo-Labeling, but it does not work well. Unfortunately, after reading other teams' solutions, I am very sad, that we didn't have much time two try more approaches with Pseudo-Labeling, especially Soft Pseudo-Labeling, which could give us the silver medal.
Final ensemble: https://www.kaggle.com/code/vad13irt/fp-final-ensemble/notebook
", "comments": [ { "id": 1911638, "author_name": "Mark Baushenko", "author_url": "", "post_date": "2022-08-24T08:04:18.373000", "content": "It was a very cool competition! Thank you all.
", "votes": 4, "replies": [] }, { "id": 1911739, "author_name": "Danila Berezin", "author_url": "", "post_date": "2022-08-24T09:20:10.593000", "content": "Amazing job! Congrats ✨
", "votes": 3, "replies": [] }, { "id": 1911733, "author_name": "Mark Baushenko", "author_url": "", "post_date": "2022-08-24T09:16:56.563000", "content": "Vadim forgot to specify that we had to train models with a sequence length of 4096 tokens
", "votes": 3, "replies": [] }, { "id": 1911769, "author_name": "Mike Mazurov", "author_url": "", "post_date": "2022-08-24T09:50:25.763000", "content": "Congrats, thanks for sharing your solution! It seems that soft pseudo-labeling would actually give you a strong result if you did that.
", "votes": 2, "replies": [ { "id": 1911773, "author_name": "Vadim Irtlach", "author_url": "", "post_date": "2022-08-24T09:54:12.333000", "content": "Sorrowfully 😪
", "votes": 2, "replies": [] } ] }, { "id": 1911649, "author_name": "Raja Biswas", "author_url": "", "post_date": "2022-08-24T08:08:20.187000", "content": "Congratulations @vad13irt, cool idea on making use to weights from a similar task 😊
\nWoking on 2 competitions at the same time is extremely challenging, even more so when both are compute heavy!
Thanks to Otabek for the computing resources
", "votes": 2, "replies": [] }, { "id": 1911658, "author_name": "Vadim Irtlach", "author_url": "", "post_date": "2022-08-24T08:13:58.783000", "content": "Thank you!
", "votes": 3, "replies": [] } ] } ], "comment_count": 8, "messages": [ { "id": 1911637, "postDate": "2022-08-24T08:03:24.383Z", "content": "Firstly, I would like to competition's organizers for such a great competition and congrats to the winners, you did amazing efforts! Secondly, I would like to thank my great teammates: Mark ( @e0xextazy ) , Otabek ( @obatek ), and Sanskar ( @odins0n ) ! Thank you guys for the spent time, experience and emotions!
\nHere, I am going to describe our final solution, which could reach 0.594 and 0.592 on Public Leaderboard and Private Leaderboard leaderboard respectively, thereby getting us to 88th place and giving us the bronze medal (I and Otabek will become Kaggle Experts 😄).
\nUnfortunately, in the middle of the competition, we switched to the AI4Code competition and when we switched back we only had ~1 week to try out some other approaches and ideas. We used only 4x models in our ensemble: one of the high-scoring public notebooks, Token Classification, a model with fine-tuned weights from the previous Feedback Prize competition, and a model with quite strong pre-processing. Such diversity of models permanently improves our Public Leaderboard from 0.610 to 0.594.
\nFor classification, we used grouping by essay_id and stratify by discourse_effectiveness. Multi-label Stratified Cross-validation was used for Token Classification, i.e stratify by counting the number of entries of each discourse_effectiveness in the given essay.
Masked Language Modeling pre-training
Pseudo-Labeling
\nWe tried it yesterday when we had quite a strong ensemble, but pseudo-labels generating and then training took ~20 hours, so we just tried Hard Pseudo-Labeling, but it does not work well. Unfortunately, after reading other teams' solutions, I am very sad, that we didn't have much time two try more approaches with Pseudo-Labeling, especially Soft Pseudo-Labeling, which could give us the silver medal.
Final ensemble: https://www.kaggle.com/code/vad13irt/fp-final-ensemble/notebook
", "rawMarkdown": "Firstly, I would like to competition's organizers for such a great competition and congrats to the winners, you did amazing efforts! Secondly, I would like to thank my great teammates: Mark ( @e0xextazy ) , Otabek ( @obatek ), and Sanskar ( @odins0n ) ! Thank you guys for the spent time, experience and emotions! \n\nHere, I am going to describe our final solution, which could reach 0.594 and 0.592 on Public Leaderboard and Private Leaderboard leaderboard respectively, thereby getting us to 88th place and giving us the bronze medal (I and Otabek will become Kaggle Experts 😄).\n\nUnfortunately, in the middle of the competition, we switched to the AI4Code competition and when we switched back we only had ~1 week to try out some other approaches and ideas. We used only 4x models in our ensemble: [one of the high-scoring public notebooks](https://www.kaggle.com/code/brandonhu0215/feedback-deberta-large-lb0-619), Token Classification, a model with fine-tuned weights from the previous Feedback Prize competition, and a model with quite strong pre-processing. Such diversity of models permanently improves our Public Leaderboard from 0.610 to 0.594. \n\n# Cross-Validation\nFor classification, we used grouping by `essay_id` and stratify by `discourse_effectiveness`. Multi-label Stratified Cross-validation was used for Token Classification, i.e stratify by counting the number of entries of each `discourse_effectiveness` in the given essay.\n\n# Useful attempts\n\n- Longer training for 4-5 epochs\n- Using pre-trained weights from the previous Feedback Prize competition.\n We used @cdeotte's fine-tuned weights of the DeBERTa BiLSTM model.\n- Models diversity\n 3x classification models 0.61 -> replace one classification model with token-classification model 0.598 -> add one more token classification model 0.594. \n\n\n# Useless attempts\n- Masked Language Modeling pre-training\n- Pseudo-Labeling\n\n We tried it yesterday when we had quite a strong ensemble, but pseudo-labels generating and then training took ~20 hours, so we just tried Hard Pseudo-Labeling, but it does not work well. Unfortunately, after reading other teams' solutions, I am very sad, that we didn't have much time two try more approaches with Pseudo-Labeling, especially Soft Pseudo-Labeling, which could give us the silver medal.\n\n# Code\nFinal ensemble: https://www.kaggle.com/code/vad13irt/fp-final-ensemble/notebook", "votes": 30 }, { "id": 1911638, "postDate": "2022-08-24T08:04:18.373Z", "content": "It was a very cool competition! Thank you all.
", "rawMarkdown": "It was a very cool competition! Thank you all.", "votes": 4 }, { "id": 1911739, "postDate": "2022-08-24T09:20:10.593Z", "content": "Amazing job! Congrats ✨
", "rawMarkdown": "Amazing job! Congrats ✨", "votes": 3 }, { "id": 1911733, "postDate": "2022-08-24T09:16:56.563Z", "content": "Vadim forgot to specify that we had to train models with a sequence length of 4096 tokens
", "rawMarkdown": "Vadim forgot to specify that we had to train models with a sequence length of 4096 tokens", "votes": 3 }, { "id": 1911769, "postDate": "2022-08-24T09:50:25.763Z", "content": "Congrats, thanks for sharing your solution! It seems that soft pseudo-labeling would actually give you a strong result if you did that.
", "rawMarkdown": "Congrats, thanks for sharing your solution! It seems that soft pseudo-labeling would actually give you a strong result if you did that.", "votes": 2, "replies": [ { "id": 1911773, "postDate": "2022-08-24T09:54:12.333Z", "content": "Sorrowfully 😪
", "rawMarkdown": "Sorrowfully 😪", "votes": 2 } ] }, { "id": 1911649, "postDate": "2022-08-24T08:08:20.187Z", "content": "Congratulations @vad13irt, cool idea on making use to weights from a similar task 😊
\nWoking on 2 competitions at the same time is extremely challenging, even more so when both are compute heavy!
Thanks to Otabek for the computing resources
", "rawMarkdown": "Thanks to Otabek for the computing resources ", "votes": 2 }, { "id": 1911658, "postDate": "2022-08-24T08:13:58.783Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 3 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:55:12Z", "competition_id": 35308, "competition_slug": "feedback-prize-effectiveness", "team_id": 8675708, "solution_url": "/competitions/feedback-prize-effectiveness/writeups/bestfitting-or-overfitting-88th-place-solution" }