First of all, I'd like to thank organizers for creating this nice competition and the whole kaggle community for sharing a lot of fruitful ideas during the competition. I learned a lot about modern NLP techniques used for analysis of large texts.
\nIn particular, I'd like to thank @cdeotte for his comprehensive starter kernel, which really helped me to get going, @abhishek for a strong baseline, which was probably used by majority of the top teams in this competition, as well as @hengck23, whose model ensembling approach I used and who also generously shared a lot of results from his experiments. Please, give their kernels an up-vote!
\nMy solution is using an ensemble of 5 models: 2x deberta-large-1024, 2x deberta-v3-large-1024 and longformer-large-1536 trained on 5 folds each.
\nMy training code: https://github.com/akuritsyn/feedback-prize-2021
\nSubmission kernel: https://www.kaggle.com/akuritsyn/feedback-model-ensemble
Good luck everyone!
", "comments": [ { "id": 1728990, "author_name": "Kohecchi", "author_url": "", "post_date": "2022-03-19T14:26:00.113000", "content": "Congrats on the silver medal!
\nI am new on NLP and learned a lot from your solution. I also don't have large GPU, so struggled to onboard models on my 15GB GPU. I am so surprized you set batch size = 1 cos I thought I cannot get good score with it. In the next NLP competition, I will try this. Thanks.
Thanks, @kohecchi . Yes, since you can use gradient accumulation (and gradient checkpointing out of the box for some models or other tricks like DeepSpeed discussed in this forum for deberta-large models) to effectively get a larger batch size, you do not necessarily need a GPU card with a lot of memory, although it would of course make your life easier. In fact, I did most of the initial experimentation using two 1080Ti cards with 11GB of memory and initially trained deberta-large models using max_len=512. I only retrained deberta-large-1024 models using 15GB card at the very end of the competition. I think it was still possible to get a gold-medal-zone result even with a 11GB card if one were smart enough to come up with a post-processing approach used by one of the top teams.
", "votes": 0, "replies": [] } ] }, { "id": 1725593, "author_name": "GUNER", "author_url": "", "post_date": "2022-03-17T08:39:31.663000", "content": "So you got almost the best barebones result (just transformers ensemble with thresholding), except maybe no:2 team (they have 72.1), one more step would have taken you to the top 10… Congrats!
", "votes": 1, "replies": [ { "id": 1726240, "author_name": "AlexeyK", "author_url": "", "post_date": "2022-03-17T19:34:02.127000", "content": "Thanks, @abdulkadirguner! My plan was to team up with someone towards the end of the competition, however I got distracted and was somewhere in the middle of the leaderboard before the team merger deadline so I was not in a position to team up with anyone on top and had to finish solo, which limited amount of things I could try, but all in all I am happy with the final result :)
", "votes": 0, "replies": [] }, { "id": 1729771, "author_name": "vialactea", "author_url": "", "post_date": "2022-03-20T13:49:39.827000", "content": "Thanks for your post. I learned that we can train deberta in a Kaggle GPU by reducing max_len. I thought the memory usage was driven by the model weights and the input didn't matter. You made my day 👍
\nThis was my first time with NLP models, but I quickly realized that memory and processing were going to be key. I spent my first weeks training different models in a TPU. At some point I concluded that all the models I could train there were significantly worse than the longformers shared by @abhishek and decided to focus on post-processing.
\nAfter a quick analysis of the predictions, I came up with some simple ideas to improve results. Rather than code them manually, I assumed that models could do a much better job than I could. Writing the code for those models, especially for building the target variables, took a while. I sometimes would take a quick break and try one of the simple manual ideas. Usually it would take anywhere from 10m to 1h to do that. My main goal was to use submissions to build an understanding of how much my CV would align with the LB. Interestingly those quick things got me an improvement of +0.007 compared to Abhishek's models (+0.008 in the private LB).
\nAt some point I realized from a @hengck23 post that deberta was way better than longformer. I made an attempt at training it in a GPU but hit a wall with out of memory errors. My lack of experience with pytorch (also first time using it) was a big handicap.
\nEach of my main ideas improved the CV but not the LB, so I just moved from one to another without spending time in optimizations: basic feature engineering, simple and small models, few epochs, basic scheduler, no hyperparameter optimization.
\nIn the last 3 weeks I was over loaded with work and personal matters I could no longer postpone, which left me stuck with the early submissions of my simple ideas. When the competition ended, I was very surprised to see that although my main ideas didn't improve the public LB, they did improve the private LB: +0.002, for a total gain of +0.010 on top of Abhishek's models. I wonder what the improvement would have been with optimizations.
\nI wish I had your models to complement my postprocessing. Maybe some other time 😃.
", "votes": 0, "replies": [] }, { "id": 1730977, "author_name": "AlexeyK", "author_url": "", "post_date": "2022-03-21T21:39:42.383000", "content": "@vialactea Re: I was very surprised to see that although my main ideas didn't improve the public LB, they did improve the private LB: +0.002, for a total gain of +0.010 on top of Abhishek's models.
\nIt is important to create your own validation set and not to overfit to the LB when building a model. While in this competition I observed good correlation between my offline validation results and the public LB and there was almost no shake up at the top after the private LB results were released, I still picked up 11 positions on the LB meaning that people above me over-fitted to the public LB.
", "votes": 0, "replies": [] } ] } ], "comment_count": 6, "messages": [ { "id": 1725482, "postDate": "2022-03-17T06:25:24.937Z", "content": "First of all, I'd like to thank organizers for creating this nice competition and the whole kaggle community for sharing a lot of fruitful ideas during the competition. I learned a lot about modern NLP techniques used for analysis of large texts.
\nIn particular, I'd like to thank @cdeotte for his comprehensive starter kernel, which really helped me to get going, @abhishek for a strong baseline, which was probably used by majority of the top teams in this competition, as well as @hengck23, whose model ensembling approach I used and who also generously shared a lot of results from his experiments. Please, give their kernels an up-vote!
\nMy solution is using an ensemble of 5 models: 2x deberta-large-1024, 2x deberta-v3-large-1024 and longformer-large-1536 trained on 5 folds each.
\nMy training code: https://github.com/akuritsyn/feedback-prize-2021
\nSubmission kernel: https://www.kaggle.com/akuritsyn/feedback-model-ensemble
Good luck everyone!
", "rawMarkdown": "First of all, I'd like to thank organizers for creating this nice competition and the whole kaggle community for sharing a lot of fruitful ideas during the competition. I learned a lot about modern NLP techniques used for analysis of large texts. \n\nIn particular, I'd like to thank @cdeotte for his comprehensive [starter kernel](https://www.kaggle.com/cdeotte/tensorflow-longformer-ner-cv-0-633), which really helped me to get going, @abhishek for a strong [baseline](https://www.kaggle.com/abhishek/two-longformers-are-better-than-1), which was probably used by majority of the top teams in this competition, as well as @hengck23, whose [model ensembling](https://www.kaggle.com/hengck23/1-birdformer-1-longformer-one-fold/) approach I used and who also generously shared a lot of results from his experiments. Please, give their kernels an up-vote!\n\nMy solution is using an ensemble of 5 models: 2x deberta-large-1024, 2x deberta-v3-large-1024 and longformer-large-1536 trained on 5 folds each.\n- I used Abhishek's 5-fold split and trained all the deberta-large models using max_len=1024, since due to GPU memory restrictions I was not able to go any higher and deberta models do not seem to support gradient checkpointing out of the box. I used max_len=1536 for longformer-large.\n- OOF validation scores showed good correlation with the public leaderboard scores, which was very nice to see and gave me confidence in my modeling approach.\n- **Deberta-large-1024 models gave the best performance. I did not verify this with a LB submit, but I think a simple ensemble of two deberta-large models using vanila Abhishek's code with properly tuned thresholds would be enough to get a silver medal. I estimate that one would get a public LB score of 0.702-0.703 and 0.713-0.714 on the private LB.**\n- Adding 2x deberta-v3-large-1024 and longformer-large-1536 allowed me to get ~+0.004 boost and the public LB score of 0.707 (or 0.719 on the private LB). This ensemble appeared to be fairly robust as I picked up 11 positions on the LB after the shake up.\n- I trained the models using Abhishek's code with batch_size=1, where I added gradient_accumulation=4 (so effectively I used batch_size=4) to fit them on my 15GB GPU. I used half a cycle cosine schedule with warm up of 0.1 and 5 epochs and AdamW optimizer. Training converged after 3-4 epochs for the most part. \n- I observed that slightly larger hidden_dropout=0.15 gives better validation scores than hidden_dropout=0.1, although I included both cases into the final ensemble for deberta. I also used LR=2e-5, however there were a couple of folds, when training did not converge in which case I dropped LR to 1e-5 on a subsequent attempt.\n- For ensembling different models, I used @hengck23's idea of predicting probabilities for every symbol in the text, then averaging probabilities of different models before predicting discourse.\n- I tuned the probability and length thresholds for different discourse classes using OOF data and computed average thresholds using the 5 folds, which gave me +0.004 boost compared to the thresholds posted in public kernels.\n- Inference of the 5 models, having 5 folds each, took a little over 8 hours in the Kaggle kernel.\n- I tried different ensembles by adding BigBird-Large, Funnel-Large, and medium-size deberta models to the mix, however this did not improve the score.\n- I also tried applying ranking to the probabilities before averaging to correct for any systematic model prediction shifts, however that made the scores worse.\n- Unfortunately I ran out of time and did not get a chance to try to build a better 2nd level post-processing model myself, but I am currently looking at the amazing solutions from the top teams. Thanks for sharing!\n\nMy training code: https://github.com/akuritsyn/feedback-prize-2021\nSubmission kernel: https://www.kaggle.com/akuritsyn/feedback-model-ensemble\n\nGood luck everyone!", "votes": 24 }, { "id": 1728990, "postDate": "2022-03-19T14:26:00.113Z", "content": "Congrats on the silver medal!
\nI am new on NLP and learned a lot from your solution. I also don't have large GPU, so struggled to onboard models on my 15GB GPU. I am so surprized you set batch size = 1 cos I thought I cannot get good score with it. In the next NLP competition, I will try this. Thanks.
Thanks, @kohecchi . Yes, since you can use gradient accumulation (and gradient checkpointing out of the box for some models or other tricks like DeepSpeed discussed in this forum for deberta-large models) to effectively get a larger batch size, you do not necessarily need a GPU card with a lot of memory, although it would of course make your life easier. In fact, I did most of the initial experimentation using two 1080Ti cards with 11GB of memory and initially trained deberta-large models using max_len=512. I only retrained deberta-large-1024 models using 15GB card at the very end of the competition. I think it was still possible to get a gold-medal-zone result even with a 11GB card if one were smart enough to come up with a post-processing approach used by one of the top teams.
", "rawMarkdown": "Thanks, @kohecchi . Yes, since you can use gradient accumulation (and gradient checkpointing out of the box for some models or other tricks like DeepSpeed discussed in this forum for deberta-large models) to effectively get a larger batch size, you do not necessarily need a GPU card with a lot of memory, although it would of course make your life easier. In fact, I did most of the initial experimentation using two 1080Ti cards with 11GB of memory and initially trained deberta-large models using max_len=512. I only retrained deberta-large-1024 models using 15GB card at the very end of the competition. I think it was still possible to get a gold-medal-zone result even with a 11GB card if one were smart enough to come up with a post-processing approach used by one of the top teams." } ] }, { "id": 1725593, "postDate": "2022-03-17T08:39:31.663Z", "content": "So you got almost the best barebones result (just transformers ensemble with thresholding), except maybe no:2 team (they have 72.1), one more step would have taken you to the top 10… Congrats!
", "rawMarkdown": "So you got almost the best barebones result (just transformers ensemble with thresholding), except maybe no:2 team (they have 72.1), one more step would have taken you to the top 10... Congrats!", "votes": 1, "replies": [ { "id": 1726240, "postDate": "2022-03-17T19:34:02.127Z", "content": "Thanks, @abdulkadirguner! My plan was to team up with someone towards the end of the competition, however I got distracted and was somewhere in the middle of the leaderboard before the team merger deadline so I was not in a position to team up with anyone on top and had to finish solo, which limited amount of things I could try, but all in all I am happy with the final result :)
", "rawMarkdown": "Thanks, @abdulkadirguner! My plan was to team up with someone towards the end of the competition, however I got distracted and was somewhere in the middle of the leaderboard before the team merger deadline so I was not in a position to team up with anyone on top and had to finish solo, which limited amount of things I could try, but all in all I am happy with the final result :)" }, { "id": 1729771, "postDate": "2022-03-20T13:49:39.827Z", "content": "Thanks for your post. I learned that we can train deberta in a Kaggle GPU by reducing max_len. I thought the memory usage was driven by the model weights and the input didn't matter. You made my day 👍
\nThis was my first time with NLP models, but I quickly realized that memory and processing were going to be key. I spent my first weeks training different models in a TPU. At some point I concluded that all the models I could train there were significantly worse than the longformers shared by @abhishek and decided to focus on post-processing.
\nAfter a quick analysis of the predictions, I came up with some simple ideas to improve results. Rather than code them manually, I assumed that models could do a much better job than I could. Writing the code for those models, especially for building the target variables, took a while. I sometimes would take a quick break and try one of the simple manual ideas. Usually it would take anywhere from 10m to 1h to do that. My main goal was to use submissions to build an understanding of how much my CV would align with the LB. Interestingly those quick things got me an improvement of +0.007 compared to Abhishek's models (+0.008 in the private LB).
\nAt some point I realized from a @hengck23 post that deberta was way better than longformer. I made an attempt at training it in a GPU but hit a wall with out of memory errors. My lack of experience with pytorch (also first time using it) was a big handicap.
\nEach of my main ideas improved the CV but not the LB, so I just moved from one to another without spending time in optimizations: basic feature engineering, simple and small models, few epochs, basic scheduler, no hyperparameter optimization.
\nIn the last 3 weeks I was over loaded with work and personal matters I could no longer postpone, which left me stuck with the early submissions of my simple ideas. When the competition ended, I was very surprised to see that although my main ideas didn't improve the public LB, they did improve the private LB: +0.002, for a total gain of +0.010 on top of Abhishek's models. I wonder what the improvement would have been with optimizations.
\nI wish I had your models to complement my postprocessing. Maybe some other time 😃.
", "rawMarkdown": "Thanks for your post. I learned that we can train deberta in a Kaggle GPU by reducing max_len. I thought the memory usage was driven by the model weights and the input didn't matter. You made my day 👍\n\nThis was my first time with NLP models, but I quickly realized that memory and processing were going to be key. I spent my first weeks training different models in a TPU. At some point I concluded that all the models I could train there were significantly worse than the longformers shared by @abhishek and decided to focus on post-processing.\n\nAfter a quick analysis of the predictions, I came up with some simple ideas to improve results. Rather than code them manually, I assumed that models could do a much better job than I could. Writing the code for those models, especially for building the target variables, took a while. I sometimes would take a quick break and try one of the simple manual ideas. Usually it would take anywhere from 10m to 1h to do that. My main goal was to use submissions to build an understanding of how much my CV would align with the LB. Interestingly those quick things got me an improvement of +0.007 compared to Abhishek's models (+0.008 in the private LB).\n\nAt some point I realized from a @hengck23 post that deberta was way better than longformer. I made an attempt at training it in a GPU but hit a wall with out of memory errors. My lack of experience with pytorch (also first time using it) was a big handicap.\n\nEach of my main ideas improved the CV but not the LB, so I just moved from one to another without spending time in optimizations: basic feature engineering, simple and small models, few epochs, basic scheduler, no hyperparameter optimization. \n\nIn the last 3 weeks I was over loaded with work and personal matters I could no longer postpone, which left me stuck with the early submissions of my simple ideas. When the competition ended, I was very surprised to see that although my main ideas didn't improve the public LB, they did improve the private LB: +0.002, for a total gain of +0.010 on top of Abhishek's models. I wonder what the improvement would have been with optimizations.\n\nI wish I had your models to complement my postprocessing. Maybe some other time 😃." }, { "id": 1730977, "postDate": "2022-03-21T21:39:42.383Z", "content": "@vialactea Re: I was very surprised to see that although my main ideas didn't improve the public LB, they did improve the private LB: +0.002, for a total gain of +0.010 on top of Abhishek's models.
\nIt is important to create your own validation set and not to overfit to the LB when building a model. While in this competition I observed good correlation between my offline validation results and the public LB and there was almost no shake up at the top after the private LB results were released, I still picked up 11 positions on the LB meaning that people above me over-fitted to the public LB.
", "rawMarkdown": "@vialactea Re: I was very surprised to see that although my main ideas didn't improve the public LB, they did improve the private LB: +0.002, for a total gain of +0.010 on top of Abhishek's models.\n\nIt is important to create your own validation set and not to overfit to the LB when building a model. While in this competition I observed good correlation between my offline validation results and the public LB and there was almost no shake up at the top after the private LB results were released, I still picked up 11 positions on the LB meaning that people above me over-fitted to the public LB." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:35:30Z", "competition_id": 31779, "competition_slug": "feedback-prize-2021", "team_id": 7913185, "solution_url": "/competitions/feedback-prize-2021/writeups/alexeyk-ods-ai-36th-place-deberta-large-1024-longf" }