Thanks to Kaggle and hosts for hosting the 3rd addition of the Feedback competition.
\nMy solution is based on an ensemble of multiple finetuned NLP transformer models. Additionally, I employ two rounds of pseudo tagging on old Feedback data.
\nI follow a lot of our advice from the second Feedback competition described here and worked for around two weeks on this competition. I split my time working around 50% on accuracy, and 50% on efficiency solution.
\nIn general, I observed very good correlation between local CV and public LB. As the data is very small and the metric is RMSE, the local scores can be quite shaky. To that end, for each experiment I was running, I trained three unique seeds and always only compared the average of these three seeds. So for example, if I would want to compare LR=1e-5 vs. LR=2e-5 I would run for each of those two experiments three separate seeds for a single fold, and only if the average of the three seeds improves, I would run on all my 5-folds, and then again compare 3-seed blends to make sure.
\nThis allows to bring more trust to my experiments and as the data is really small, this was in general possible for me to do.
\nThe problem at hand is very much straight forward, feed in the text to a transformer model, apply some pooling, add a linear head, and predict regression targets. I used combinations of the following variations of the training routine for my final ensemble:
\nToken length:
\nAll my models are trained and predicted with dynamic padding.
\nPooling:
\nBackbones:
\nI usually run 3 epochs for most of my models, all with cosine decay learning rate always picking the last epoch. I use differential learning rate for backbone and head of the model. I do not use any other techniques suggested in forums like differential lr across layers of backbone or reinitialization.
\nAs always, I retrained my models on full data for final subs, but also blended some fold models in as I had lots of runtime left.
\nI followed our routine from 2nd FB competition and employed two stages of pseudo labeling following these steps:
\nBy doing pre-training and finetuning, I did not need to adjust the distribution of the pseudo labels, because I am doing the final adjustment on this competition's data. This allowed me to use all prior data without issues.
\nFor most of my subs I just did usual average across seeds and models. My final best sub is a Nelder-Mead optimized ensemble of models, where I optimize the ensemble weights separately per target column. To not overfit too much on CV, I added weight bounds between 1 and 3 on the weights.
\nActually, I could have trusted the local optimization even more, I have an unselected sub from couple of days ago with best local CV that would score #2 on private, which has unrestricted weights for the ensemble, also with negative weights, but it felt a bit too risky and as I only did one sub for best CV, I chose a bit more of a conservative one.
\nI did one sub based on my conservative best CV score, which also was my best selected private LB sub (although I had a better one unselected). I did one sub based on best public LB which was clearly worse on private LB. And I spent one sub on efficiency.
", "comments": [ { "id": 2050192, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-30T15:23:24.107000", "content": "Congratulations Psi! Nice jump from 166th to 5th.
\n\n\nUse pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards.
\n
this is a good trick. The distribution between FP1 and FP3 was different.
\n\n\nActually, I could have trusted the local optimization even more, I have an unselected sub from couple of days ago with best local CV that would score #2 on private, which has unrestricted weights for the ensemble, also with negative weights, but it felt a bit too risky and as I only did one sub for best CV, I chose a bit more of a conservative one.
\n
We were also wondering whether to include negative weights in our ensemble of 24 models. Luckily we chose our ensemble with negative weights which ranked private 3rd whereas the one with only positive weights ranked 11th. This was my first time using negative weights in an ensemble. It worked well. I guess it is just like stacking with linear regression stage 2 model which should be safe.
", "votes": 6, "replies": [ { "id": 2050224, "author_name": "Psi", "author_url": "", "post_date": "2022-11-30T15:39:19.513000", "content": "We actually used negative weights successfully in FB2, I should have trusted it :)
", "votes": 3, "replies": [] }, { "id": 2050305, "author_name": "Trushant Kalyanpur", "author_url": "", "post_date": "2022-11-30T16:37:02.213000", "content": "Great work as usual @philippsinger ! We also used Nelder Mead (NM) in our pipeline but that sub didnt end up in gold even though the CV was pretty good (0.439x) . I am wondering if there is some over-optimistic CV if we mix N-fold models. Do you always make sure you use same folds when doing NM? I dont see why it breaks if we have N folds and are using OOF's to optimize the weights.
", "votes": 1, "replies": [] }, { "id": 2050308, "author_name": "Psi", "author_url": "", "post_date": "2022-11-30T16:38:24.567000", "content": "I always 100% make sure to have same folds.
\nMy final CV was btw ~0.445
", "votes": 2, "replies": [] }, { "id": 2050318, "author_name": "Raja Biswas", "author_url": "", "post_date": "2022-11-30T16:46:39.183000", "content": "Even after having same folds for all models, if we tune NM weights over all oof predictions, it would have risk of overfitting to training data right?
", "votes": 0, "replies": [] }, { "id": 2050330, "author_name": "Psi", "author_url": "", "post_date": "2022-11-30T16:56:57.683000", "content": "Yes absolutely, which is why I tried to use the bounds.
", "votes": 2, "replies": [] }, { "id": 2050345, "author_name": "Raja Biswas", "author_url": "", "post_date": "2022-11-30T17:16:59.723000", "content": "Thanks, makes perfect sense! Congratulations & great work as always ππ Will try to follow your cross validation strategy now onwards, I feel it's inspired by how results are reported in papers.
", "votes": 1, "replies": [] } ] }, { "id": 2050174, "author_name": "Feng Qilong", "author_url": "", "post_date": "2022-11-30T15:14:49.647000", "content": "Congrats! For \"by pre-train models on the pseudo labels and finetune it only on the given train data afterwards\", do you mean training the models with PL for x epochs then save the model and train with original data for y epochs?
", "votes": 1, "replies": [ { "id": 2050175, "author_name": "Psi", "author_url": "", "post_date": "2022-11-30T15:16:12.910000", "content": "yes exactly
", "votes": 1, "replies": [] }, { "id": 2056560, "author_name": "Apa", "author_url": "", "post_date": "2022-12-06T08:33:04.957000", "content": "\"Use pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards.\" In this step, you will not combine the data of this competition for pre-training ?
", "votes": 0, "replies": [] }, { "id": 2056629, "author_name": "Psi", "author_url": "", "post_date": "2022-12-06T10:02:08.623000", "content": "No, just old data
", "votes": 1, "replies": [] } ] }, { "id": 2050527, "author_name": "CPMP", "author_url": "", "post_date": "2022-11-30T20:18:04.947000", "content": "Congrats!
\n\n\nI trained three unique seeds and always only compared the average of these three seeds.
\n
I did the same, LOL. Before that, using one seed led to unreliable CV.
\nNice idea to only used pseudo labels for pretraining.
", "votes": 0, "replies": [] }, { "id": 2050424, "author_name": "Hazrul Akmal", "author_url": "", "post_date": "2022-11-30T18:14:17.067000", "content": "\n\nif I would want to compare LR=1e-5 vs. LR=2e-5 I would run for each of those two experiments three separate seeds for a single fold, and only if the average of the three seeds improves, I would run on all my 5-folds, and then again compare 3-seed blends to make sure.
\n
If I get your CV approach correctly, it seems that your approach is computationally expensive. 3-seed validation + 5-fold CV with 3 different seeds = 18 runs for a single full experiment. How long did it take for you to do it, especially when training a large language model like deberta-L/XL/XXL?
", "votes": 0, "replies": [ { "id": 2050460, "author_name": "Psi", "author_url": "", "post_date": "2022-11-30T18:48:32.870000", "content": "Deberta-v3-large is very fast. XL and XXL can take a while, but nothing I couldnt run overnight. The data is very small. I did not use XL and XXL for pseudos to save time.
", "votes": 0, "replies": [] }, { "id": 2050498, "author_name": "Hazrul Akmal", "author_url": "", "post_date": "2022-11-30T19:52:36.713000", "content": "I see. I was wondering because it took me roughly 7 hours to train Deberta-Large for 4-fold with 5 epochs. Not sure if this is normal. Anyway, congratulations on your stunning performance in the competition!
", "votes": 0, "replies": [] }, { "id": 2050502, "author_name": "Psi", "author_url": "", "post_date": "2022-11-30T19:54:28.683000", "content": "What GPU do you have? Also 5 epochs is probably too much, I was fine with 2-3 epochs.
\nYou also need to utilize all the runtime tricks, mixed precision, dynamic padding, etc.
", "votes": 1, "replies": [] } ] }, { "id": 2050385, "author_name": "ErtuΔrul Demir", "author_url": "", "post_date": "2022-11-30T17:43:14.560000", "content": "Congratulations @philippsinger! These are good tricks to remember. One simple question though:
\n\n\nAs always, I retrained my models on full data for final subs, but also blended some fold models in as I had lots of runtime left.
\n
Have you used exact same hyperparameters to train whole model again or applied some little changes?
\nCongrats again!
", "votes": 0, "replies": [ { "id": 2050410, "author_name": "Psi", "author_url": "", "post_date": "2022-11-30T17:59:07.030000", "content": "Exact same paras.
", "votes": 1, "replies": [] } ] }, { "id": 2050312, "author_name": "Muhammad Hilmi Asyrofi", "author_url": "", "post_date": "2022-11-30T16:40:20.133000", "content": "Congrats for the win @philippsinger !
\nThe idea of using 3 different seeds to ensure the actual model improvement is mind blowing π Thank you for sharing!
\nI wanna ask, why do you choose GeM Pooling? what's the rationale behind? have you tried different pooling strategies (e.g. attention)?
", "votes": 0, "replies": [ { "id": 2050326, "author_name": "Psi", "author_url": "", "post_date": "2022-11-30T16:53:21.667000", "content": "\n\nI wanna ask, why do you choose GeM Pooling? what's the rationale behind?
\n
It is my favorite type of pooling, I always use it :)
\nAttention pooling does not make much sense to me personally, so I never try it.
Hahaha I will try GeM pooling next time, it's kind of magic pooling used by a GM π
\nWell, I need to read the paper on it to know more detail.
\nMany thanks!
Congratulations and thanks for the detailed write-up! I was amazed by the strong mind of top teams for trusting CVβ¦
\nI have two questions.
\n\nI trained three unique seeds and always only compared the average of these three seeds.
\n
Is there any reason that you take the average instead of the score of the ensemble?
\n\n\nUse pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards.
\n
When you start training for the training data, did you resume cosine decay as a continuous one from the pseudo label or start another one again?
", "votes": 0, "replies": [ { "id": 2050297, "author_name": "Psi", "author_url": "", "post_date": "2022-11-30T16:32:13.707000", "content": "\n\nIs there any reason that you take the average instead of the score of the ensemble?
\n
I am always just averaging seeds of same model
\n\n\nWhen you start training for the training data, did you resume cosine decay as a continuous one from the pseudo label or start another one again?
\n
Start another one
", "votes": 1, "replies": [] }, { "id": 2050320, "author_name": "knshnb", "author_url": "", "post_date": "2022-11-30T16:47:37.087000", "content": "Thanks for the quick reply!
\nFor the first one, I mean the average of three scores vs the score of the ensemble of three.
I see, I am checking the ensemble score of the average of the three seeds.
", "votes": 0, "replies": [] } ] }, { "id": 2050219, "author_name": "kaerururu", "author_url": "", "post_date": "2022-11-30T15:37:47.933000", "content": "Congratulations !!
\nI have a simple question.
\nI calculated max token length in train full_text is around 1430.
\nHowever, you used 2048 token length.
\nIs this for unseen test full_text ?
As I use dynamic padding it is just to go up to the largest one.
", "votes": 3, "replies": [] }, { "id": 2050237, "author_name": "kaerururu", "author_url": "", "post_date": "2022-11-30T15:48:32.323000", "content": "Thank you for quick reply, I see.
", "votes": 0, "replies": [] } ] } ], "comment_count": 28, "messages": [ { "id": 2050161, "postDate": "2022-11-30T15:07:12.043Z", "content": "Thanks to Kaggle and hosts for hosting the 3rd addition of the Feedback competition.
\nMy solution is based on an ensemble of multiple finetuned NLP transformer models. Additionally, I employ two rounds of pseudo tagging on old Feedback data.
\nI follow a lot of our advice from the second Feedback competition described here and worked for around two weeks on this competition. I split my time working around 50% on accuracy, and 50% on efficiency solution.
\nIn general, I observed very good correlation between local CV and public LB. As the data is very small and the metric is RMSE, the local scores can be quite shaky. To that end, for each experiment I was running, I trained three unique seeds and always only compared the average of these three seeds. So for example, if I would want to compare LR=1e-5 vs. LR=2e-5 I would run for each of those two experiments three separate seeds for a single fold, and only if the average of the three seeds improves, I would run on all my 5-folds, and then again compare 3-seed blends to make sure.
\nThis allows to bring more trust to my experiments and as the data is really small, this was in general possible for me to do.
\nThe problem at hand is very much straight forward, feed in the text to a transformer model, apply some pooling, add a linear head, and predict regression targets. I used combinations of the following variations of the training routine for my final ensemble:
\nToken length:
\nAll my models are trained and predicted with dynamic padding.
\nPooling:
\nBackbones:
\nI usually run 3 epochs for most of my models, all with cosine decay learning rate always picking the last epoch. I use differential learning rate for backbone and head of the model. I do not use any other techniques suggested in forums like differential lr across layers of backbone or reinitialization.
\nAs always, I retrained my models on full data for final subs, but also blended some fold models in as I had lots of runtime left.
\nI followed our routine from 2nd FB competition and employed two stages of pseudo labeling following these steps:
\nBy doing pre-training and finetuning, I did not need to adjust the distribution of the pseudo labels, because I am doing the final adjustment on this competition's data. This allowed me to use all prior data without issues.
\nFor most of my subs I just did usual average across seeds and models. My final best sub is a Nelder-Mead optimized ensemble of models, where I optimize the ensemble weights separately per target column. To not overfit too much on CV, I added weight bounds between 1 and 3 on the weights.
\nActually, I could have trusted the local optimization even more, I have an unselected sub from couple of days ago with best local CV that would score #2 on private, which has unrestricted weights for the ensemble, also with negative weights, but it felt a bit too risky and as I only did one sub for best CV, I chose a bit more of a conservative one.
\nI did one sub based on my conservative best CV score, which also was my best selected private LB sub (although I had a better one unselected). I did one sub based on best public LB which was clearly worse on private LB. And I spent one sub on efficiency.
", "rawMarkdown": "Thanks to Kaggle and hosts for hosting the 3rd addition of the Feedback competition.\n\n# Overview \n\nMy solution is based on an ensemble of multiple finetuned NLP transformer models. Additionally, I employ two rounds of pseudo tagging on old Feedback data.\n\nI follow a lot of our advice from the second Feedback competition described [here](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347536) and worked for around two weeks on this competition. I split my time working around 50% on accuracy, and 50% on efficiency solution. \n\n## Cross validation\n\nIn general, I observed very good correlation between local CV and public LB. As the data is very small and the metric is RMSE, the local scores can be quite shaky. To that end, for each experiment I was running, I trained three unique seeds and always only compared the average of these three seeds. So for example, if I would want to compare LR=1e-5 vs. LR=2e-5 I would run for each of those two experiments three separate seeds for a single fold, and only if the average of the three seeds improves, I would run on all my 5-folds, and then again compare 3-seed blends to make sure.\n\nThis allows to bring more trust to my experiments and as the data is really small, this was in general possible for me to do.\n\n## Modeling\n\nThe problem at hand is very much straight forward, feed in the text to a transformer model, apply some pooling, add a linear head, and predict regression targets. I used combinations of the following variations of the training routine for my final ensemble:\n\nToken length:\n- 512\n- 1024\n- 2048\n\nAll my models are trained and predicted with dynamic padding.\n\nPooling:\n- CLS Token\n- GeM Pooling\n\nBackbones:\n- Deberta-V3-Base\n- Deberta-V3-Large\n- Deberta-V2-XL\n- Deberta-V2-XXL\n- Longformer Large\n- Roberta-Large\n\nI usually run 3 epochs for most of my models, all with cosine decay learning rate always picking the last epoch. I use differential learning rate for backbone and head of the model. I do not use any other techniques suggested in forums like differential lr across layers of backbone or reinitialization.\n\nAs always, I retrained my models on full data for final subs, but also blended some fold models in as I had lots of runtime left.\n\n## Pseudo labels\n\nI followed our routine from 2nd FB competition and employed two stages of pseudo labeling following these steps:\n\n1. Train an ensemble of models only on the given train data\n1. Run predictions on the previous Feedback competition data excluding this competition's data\n3. Use pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards. \n4. Repeat steps 1-3 three times using an ensemble of models trained on pseudo labels now\n\nBy doing pre-training and finetuning, I did not need to adjust the distribution of the pseudo labels, because I am doing the final adjustment on this competition's data. This allowed me to use all prior data without issues.\n\n## Ensembling\n\nFor most of my subs I just did usual average across seeds and models. My final best sub is a Nelder-Mead optimized ensemble of models, where I optimize the ensemble weights separately per target column. To not overfit too much on CV, I added weight bounds between 1 and 3 on the weights.\n\nActually, I could have trusted the local optimization even more, I have an unselected sub from couple of days ago with best local CV that would score #2 on private, which has unrestricted weights for the ensemble, also with negative weights, but it felt a bit too risky and as I only did one sub for best CV, I chose a bit more of a conservative one.\n\n## What did not work\n\n- Augmentations (as always, specifically tricky with regression)\n- Different losses\n- TFIDF\n- Other backbones (such as T5, GPT, etc) - Deberta is so strong\n- 2nd stage models / stacker models\n\n## Final subs\n\nI did one sub based on my conservative best CV score, which also was my best selected private LB sub (although I had a better one unselected). I did one sub based on best public LB which was clearly worse on private LB. And I spent one sub on efficiency.", "votes": 74 }, { "id": 2050192, "postDate": "2022-11-30T15:23:24.107Z", "content": "Congratulations Psi! Nice jump from 166th to 5th.
\n\n\nUse pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards.
\n
this is a good trick. The distribution between FP1 and FP3 was different.
\n\n\nActually, I could have trusted the local optimization even more, I have an unselected sub from couple of days ago with best local CV that would score #2 on private, which has unrestricted weights for the ensemble, also with negative weights, but it felt a bit too risky and as I only did one sub for best CV, I chose a bit more of a conservative one.
\n
We were also wondering whether to include negative weights in our ensemble of 24 models. Luckily we chose our ensemble with negative weights which ranked private 3rd whereas the one with only positive weights ranked 11th. This was my first time using negative weights in an ensemble. It worked well. I guess it is just like stacking with linear regression stage 2 model which should be safe.
", "rawMarkdown": "Congratulations Psi! Nice jump from 166th to 5th.\n>Use pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards.\n\nthis is a good trick. The distribution between FP1 and FP3 was different.\n\n>Actually, I could have trusted the local optimization even more, I have an unselected sub from couple of days ago with best local CV that would score #2 on private, which has unrestricted weights for the ensemble, also with negative weights, but it felt a bit too risky and as I only did one sub for best CV, I chose a bit more of a conservative one.\n\nWe were also wondering whether to include negative weights in our ensemble of 24 models. Luckily we chose our ensemble with negative weights which ranked private 3rd whereas the one with only positive weights ranked 11th. This was my first time using negative weights in an ensemble. It worked well. I guess it is just like stacking with linear regression stage 2 model which should be safe.", "votes": 6, "replies": [ { "id": 2050224, "postDate": "2022-11-30T15:39:19.513Z", "content": "We actually used negative weights successfully in FB2, I should have trusted it :)
", "rawMarkdown": "We actually used negative weights successfully in FB2, I should have trusted it :)", "votes": 3 }, { "id": 2050305, "postDate": "2022-11-30T16:37:02.213Z", "content": "Great work as usual @philippsinger ! We also used Nelder Mead (NM) in our pipeline but that sub didnt end up in gold even though the CV was pretty good (0.439x) . I am wondering if there is some over-optimistic CV if we mix N-fold models. Do you always make sure you use same folds when doing NM? I dont see why it breaks if we have N folds and are using OOF's to optimize the weights.
", "rawMarkdown": "Great work as usual @philippsinger ! We also used Nelder Mead (NM) in our pipeline but that sub didnt end up in gold even though the CV was pretty good (0.439x) . I am wondering if there is some over-optimistic CV if we mix N-fold models. Do you always make sure you use same folds when doing NM? I dont see why it breaks if we have N folds and are using OOF's to optimize the weights. ", "votes": 1 }, { "id": 2050308, "postDate": "2022-11-30T16:38:24.567Z", "content": "I always 100% make sure to have same folds.
\nMy final CV was btw ~0.445
", "rawMarkdown": "I always 100% make sure to have same folds.\n\nMy final CV was btw ~0.445", "votes": 2 }, { "id": 2050318, "postDate": "2022-11-30T16:46:39.183Z", "content": "Even after having same folds for all models, if we tune NM weights over all oof predictions, it would have risk of overfitting to training data right?
", "rawMarkdown": "Even after having same folds for all models, if we tune NM weights over all oof predictions, it would have risk of overfitting to training data right?" }, { "id": 2050330, "postDate": "2022-11-30T16:56:57.683Z", "content": "Yes absolutely, which is why I tried to use the bounds.
", "rawMarkdown": "Yes absolutely, which is why I tried to use the bounds.", "votes": 2 }, { "id": 2050345, "postDate": "2022-11-30T17:16:59.723Z", "content": "Thanks, makes perfect sense! Congratulations & great work as always ππ Will try to follow your cross validation strategy now onwards, I feel it's inspired by how results are reported in papers.
", "rawMarkdown": "Thanks, makes perfect sense! Congratulations & great work as always ππ Will try to follow your cross validation strategy now onwards, I feel it's inspired by how results are reported in papers. ", "votes": 1 } ] }, { "id": 2050174, "postDate": "2022-11-30T15:14:49.647Z", "content": "Congrats! For \"by pre-train models on the pseudo labels and finetune it only on the given train data afterwards\", do you mean training the models with PL for x epochs then save the model and train with original data for y epochs?
", "rawMarkdown": "Congrats! For \"by pre-train models on the pseudo labels and finetune it only on the given train data afterwards\", do you mean training the models with PL for x epochs then save the model and train with original data for y epochs?", "votes": 1, "replies": [ { "id": 2050175, "postDate": "2022-11-30T15:16:12.910Z", "content": "yes exactly
", "rawMarkdown": "yes exactly", "votes": 1 }, { "id": 2056560, "postDate": "2022-12-06T08:33:04.957Z", "content": "\"Use pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards.\" In this step, you will not combine the data of this competition for pre-training ?
", "rawMarkdown": "\"Use pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards.\" In this step, you will not combine the data of this competition for pre-training ?" }, { "id": 2056629, "postDate": "2022-12-06T10:02:08.623Z", "content": "No, just old data
", "rawMarkdown": "No, just old data", "votes": 1 } ] }, { "id": 2050527, "postDate": "2022-11-30T20:18:04.947Z", "content": "Congrats!
\n\n\nI trained three unique seeds and always only compared the average of these three seeds.
\n
I did the same, LOL. Before that, using one seed led to unreliable CV.
\nNice idea to only used pseudo labels for pretraining.
", "rawMarkdown": "Congrats!\n\n> I trained three unique seeds and always only compared the average of these three seeds.\n\nI did the same, LOL. Before that, using one seed led to unreliable CV.\n\nNice idea to only used pseudo labels for pretraining." }, { "id": 2050424, "postDate": "2022-11-30T18:14:17.067Z", "content": "\n\nif I would want to compare LR=1e-5 vs. LR=2e-5 I would run for each of those two experiments three separate seeds for a single fold, and only if the average of the three seeds improves, I would run on all my 5-folds, and then again compare 3-seed blends to make sure.
\n
If I get your CV approach correctly, it seems that your approach is computationally expensive. 3-seed validation + 5-fold CV with 3 different seeds = 18 runs for a single full experiment. How long did it take for you to do it, especially when training a large language model like deberta-L/XL/XXL?
", "rawMarkdown": "> if I would want to compare LR=1e-5 vs. LR=2e-5 I would run for each of those two experiments three separate seeds for a single fold, and only if the average of the three seeds improves, I would run on all my 5-folds, and then again compare 3-seed blends to make sure.\n\nIf I get your CV approach correctly, it seems that your approach is computationally expensive. 3-seed validation + 5-fold CV with 3 different seeds = 18 runs for a single full experiment. How long did it take for you to do it, especially when training a large language model like deberta-L/XL/XXL? ", "replies": [ { "id": 2050460, "postDate": "2022-11-30T18:48:32.870Z", "content": "Deberta-v3-large is very fast. XL and XXL can take a while, but nothing I couldnt run overnight. The data is very small. I did not use XL and XXL for pseudos to save time.
", "rawMarkdown": "Deberta-v3-large is very fast. XL and XXL can take a while, but nothing I couldnt run overnight. The data is very small. I did not use XL and XXL for pseudos to save time. " }, { "id": 2050498, "postDate": "2022-11-30T19:52:36.713Z", "content": "I see. I was wondering because it took me roughly 7 hours to train Deberta-Large for 4-fold with 5 epochs. Not sure if this is normal. Anyway, congratulations on your stunning performance in the competition!
", "rawMarkdown": "I see. I was wondering because it took me roughly 7 hours to train Deberta-Large for 4-fold with 5 epochs. Not sure if this is normal. Anyway, congratulations on your stunning performance in the competition!" }, { "id": 2050502, "postDate": "2022-11-30T19:54:28.683Z", "content": "What GPU do you have? Also 5 epochs is probably too much, I was fine with 2-3 epochs.
\nYou also need to utilize all the runtime tricks, mixed precision, dynamic padding, etc.
", "rawMarkdown": "What GPU do you have? Also 5 epochs is probably too much, I was fine with 2-3 epochs.\n\nYou also need to utilize all the runtime tricks, mixed precision, dynamic padding, etc.", "votes": 1 } ] }, { "id": 2050385, "postDate": "2022-11-30T17:43:14.560Z", "content": "Congratulations @philippsinger! These are good tricks to remember. One simple question though:
\n\n\nAs always, I retrained my models on full data for final subs, but also blended some fold models in as I had lots of runtime left.
\n
Have you used exact same hyperparameters to train whole model again or applied some little changes?
\nCongrats again!
", "rawMarkdown": "Congratulations @philippsinger! These are good tricks to remember. One simple question though:\n\n> As always, I retrained my models on full data for final subs, but also blended some fold models in as I had lots of runtime left.\n\nHave you used exact same hyperparameters to train whole model again or applied some little changes?\n\nCongrats again!", "replies": [ { "id": 2050410, "postDate": "2022-11-30T17:59:07.030Z", "content": "Exact same paras.
", "rawMarkdown": "Exact same paras.", "votes": 1 } ] }, { "id": 2050312, "postDate": "2022-11-30T16:40:20.133Z", "content": "Congrats for the win @philippsinger !
\nThe idea of using 3 different seeds to ensure the actual model improvement is mind blowing π Thank you for sharing!
\nI wanna ask, why do you choose GeM Pooling? what's the rationale behind? have you tried different pooling strategies (e.g. attention)?
", "rawMarkdown": "Congrats for the win @philippsinger !\n\nThe idea of using 3 different seeds to ensure the actual model improvement is mind blowing π Thank you for sharing!\n\nI wanna ask, why do you choose GeM Pooling? what's the rationale behind? have you tried different pooling strategies (e.g. attention)?", "replies": [ { "id": 2050326, "postDate": "2022-11-30T16:53:21.667Z", "content": "\n\nI wanna ask, why do you choose GeM Pooling? what's the rationale behind?
\n
It is my favorite type of pooling, I always use it :)
\nAttention pooling does not make much sense to me personally, so I never try it.
Hahaha I will try GeM pooling next time, it's kind of magic pooling used by a GM π
\nWell, I need to read the paper on it to know more detail.
\nMany thanks!
Congratulations and thanks for the detailed write-up! I was amazed by the strong mind of top teams for trusting CVβ¦
\nI have two questions.
\n\nI trained three unique seeds and always only compared the average of these three seeds.
\n
Is there any reason that you take the average instead of the score of the ensemble?
\n\n\nUse pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards.
\n
When you start training for the training data, did you resume cosine decay as a continuous one from the pseudo label or start another one again?
", "rawMarkdown": "Congratulations and thanks for the detailed write-up! I was amazed by the strong mind of top teams for trusting CV...\nI have two questions. \n\n> I trained three unique seeds and always only compared the average of these three seeds.\n\nIs there any reason that you take the average instead of the score of the ensemble?\n\n> Use pseudo labels from this extra dataset and apply it to modeling by pre-train models on the pseudo labels and finetune it only on the given train data afterwards.\n\nWhen you start training for the training data, did you resume cosine decay as a continuous one from the pseudo label or start another one again?", "replies": [ { "id": 2050297, "postDate": "2022-11-30T16:32:13.707Z", "content": "\n\nIs there any reason that you take the average instead of the score of the ensemble?
\n
I am always just averaging seeds of same model
\n\n\nWhen you start training for the training data, did you resume cosine decay as a continuous one from the pseudo label or start another one again?
\n
Start another one
", "rawMarkdown": "> Is there any reason that you take the average instead of the score of the ensemble?\n\nI am always just averaging seeds of same model\n\n> When you start training for the training data, did you resume cosine decay as a continuous one from the pseudo label or start another one again?\n\nStart another one", "votes": 1 }, { "id": 2050320, "postDate": "2022-11-30T16:47:37.087Z", "content": "Thanks for the quick reply!
\nFor the first one, I mean the average of three scores vs the score of the ensemble of three.
I see, I am checking the ensemble score of the average of the three seeds.
", "rawMarkdown": "I see, I am checking the ensemble score of the average of the three seeds." } ] }, { "id": 2050219, "postDate": "2022-11-30T15:37:47.933Z", "content": "Congratulations !!
\nI have a simple question.
\nI calculated max token length in train full_text is around 1430.
\nHowever, you used 2048 token length.
\nIs this for unseen test full_text ?
As I use dynamic padding it is just to go up to the largest one.
", "rawMarkdown": "As I use dynamic padding it is just to go up to the largest one.", "votes": 3 }, { "id": 2050237, "postDate": "2022-11-30T15:48:32.323Z", "content": "Thank you for quick reply, I see.
", "rawMarkdown": "Thank you for quick reply, I see." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:01:16Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9121689, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/psi-5th-place-solution" }