\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F304662%2Fbdd644ba5cf65aba562fa2a2611e6183%2FLB.png?generation=1661353133551573&alt=media\")
Thanks a lot to the hosts and Kaggle for hosting this interesting competition, we had great fun working on both the accuracy and efficiency tracks. Also congratulations to all other competitors for the great solutions and results. Our solution is the result of perfect teamwork.
\nOur solution is based on a two-stage approach ensembling multiple transformer-based models with different techniques and adding second level models on top. We additionally employ multiple rounds of pseudo tagging and add pseudo labels with different techniques to our models.
\nThroughout this competition we had near perfect correlation between CV and LB. Whenever we saw some improvement on CV, we saw it reflected in a similar manner on the LB with very small random range. For splitting the folds, we just used an efficiency-stratified split on essays.
\nAs the data is small to medium size and the metric is log loss, the scores can vary between different runs. This is typical for deep learning models as they are quite dependent on the seed at hand that influences weight initializations, batching, or augmentations. Yet, this means one should not judge model performance on single seeds, and it is better to always evaluate on multiple seeds.
\nGiven that model training was quite fast, we thus only relied on checking blends of 3 seeds for each model. Also, single model scores did not correlate well here with their ability to blend into larger ensembles. So a better individual model could have quite a worse performance in the blend, diversity really mattered here. Consequently, we also always checked models in the blend, even if they did not seem too promising on an individual basis. Similar to how we checked CV, we then always subbed a blend of 3 seeds of models trained on the full data.
\nOur correlation looked like follows:
\n
Our final solution is a combination of different modeling approaches. Most of them are based on the idea of training on all discourses from a single essay at the same time. This not only made training and inference much faster, but also improved accuracy significantly. In the following we want to describe our main approaches in more detail. For backbones, we could only get deberta-(v3)-large to work. Other backbones did not improve the ensemble.
\nThe main idea of this approach is to feed a full essay into the model, and pool each discourse separately and then feed it through the final linear layer for prediction. The main approach here is similar to what others shared, but there are some peculiarities and different sub-approaches.
\nOur main version has an input as follows:
\nLead Position Claim Evidence Counterclaim Rebuttal Evidence Counterclaim Concluding Statement [SEP] [START] Hi, i'm Isaac, i'm going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn't know if the landform was created by life on Mars, or if it is just a natural landform. [END] [START] On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I'll be talking about how I think that is is a natural landform [END] … more text follows here\nAs mentioned, one sample includes one essay. We start by adding a list of all types of the discourses in the essay with a SEP token and then we mark the individual discourses with custom START and END tokens. We then run this sample through the backbone, and pool between START and END tokens for each discourse. The input batch size is always 1, and this gets transformed to a batch size that depends on the number of discourses within the essay. These pooled embeddings then run through a final linear layer predicting the class.
\nYou can see that in this example we do not specifically add the type to each discourse, but we use an additional auxiliary loss to predict the type of each one. This helped with regularizing the model and allowed for a bit longer training.
\nAn additional sub-approach does not have this auxiliary loss and trains a model based on the following input:
\nLead Position Claim Evidence Counterclaim Rebuttal Evidence Counterclaim Concluding Statement [SEP] [START_Lead] Hi, i'm Isaac, i'm going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn't know if the landform was created by life on Mars, or if it is just a natural landform. [END_Lead] [START_Position] On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I'll be talking about how I think that is is a natural landform [END_Position] … more text follows here\nWhile the latter approach was better individually on CV, the former approach blended significantly better in our large ensemble.
\nIn this approach the chunks of the essays are treated as separate tokens. Individual chunks either get the efficiency label or “O” label if they are outside of the annotated essay text. And the subsequent process is similar to the idea above: pass the whole essay through the backbone, apply local average poolings for each chunk and add a dense classification layer on top. The input essay didn’t have any extra special tokens apart from discourse type in front of all the chunks.
\nFor diversity, we added the following models with minor impact to our solution:
\nWe spent significant efforts on hyperparameter tuning and playing with various regularization techniques. We implemented quite a few augmentation techniques, but as always they were not really helpful for NLP finetuning. Only mask augmentations worked with decent results as it was bringing some diversity for the ensemble. Hyperparameter tuning was very important though, and it was time well spent.
\nThe log loss metric is always only optimal if the mean prediction per column matches the mean of the target columns for that label. Our different models (specifically if trained in batch-wise manner on essays) do not always reflect this mean very well, which is why we added an additional optimization after each model to adjust to the train mean. This additionally makes the scores more reliable and comparable. We then also employ these learned scaling factors on LB.
\nFor ensembling different models we resorted to directly optimizing the blending weights between the models. Interestingly, we also had several models with negative weights, but this worked for us both on CV as well as LB.
\nWe additionally trained several 2nd level models to further improve our predictions.
\nFor the 2nd level LightGBM model we took the weighted ensemble prediction, together with individual models predictions, and generated some aggregate features based on the whole essay. We trained 2 LightGBM versions with different features and parameters.
\nWe tuned two types of neural networks here. The first takes the weighted ensemble prediction, as well as an average across the essay and across the type within an essay for each of the three target columns as input and trains a three-layer DNN. The second one takes the same features, but on an individual model basis and then uses a three-layer Conv1d with average pooling afterwards.
\nAll together, 2nd level models were consistently bringing us about 0.003-0.005 points on CV and the leaderboard throughout the competition.
\nAnother major part of our solution is pseudo labeling. We applied 3 stages of pseudo labeling on the extra data from the previous Feedback competition. It was done in a leak-free manner for the individual folds and additionally for our models trained on all the data (6 versions of pseudo labels in total). The process consisted of the following steps:
\nApart from using previous Feedback competition data for pseudo labels, it was also used in some models as a pre-training dataset. The model was warmed up on the old data predicting the type of the chunk and further finetuned on the given train data.
\nCongratulations Psi and Yauhen! Great solution!
\n\n\nPre-train models on the pseudo labels and finetune it only on the given train data afterwards
\n
This is what i just did in Amex competition. It worked great there too discussion.
", "votes": 11, "replies": [] }, { "id": 1912323, "author_name": "Trushant Kalyanpur", "author_url": "", "post_date": "2022-08-24T16:38:16.397000", "content": "Congratulations @ybabakhin and @philippsinger ! Amazing finishing 1st in both tracks! It was a great battle in both tracks.. Your team definitely kept us on our feet!
", "votes": 5, "replies": [] }, { "id": 1917265, "author_name": "patriot", "author_url": "", "post_date": "2022-08-28T15:33:28.987000", "content": "Congratulations on winning 1st place!
\nCould you please provide some additional explanation with the code for the following part?
\n\n", "votes": 3, "replies": [] }, { "id": 1927252, "author_name": "Peal", "author_url": "", "post_date": "2022-09-05T13:33:54.047000", "content": "The log loss metric is always only optimal if the mean prediction per column matches the mean of the target columns for that label. Our different models (specifically if trained in batch-wise manner on essays) do not always reflect this mean very well, which is why we added an additional optimization after each model to adjust to the train mean. This additionally makes the scores more reliable and comparable.
\n
Congratulations! could you please tell me more about the details of mask augmentation? how to do it?
", "votes": 1, "replies": [] }, { "id": 1912900, "author_name": "Apa", "author_url": "", "post_date": "2022-08-25T03:08:07.880000", "content": "Congratulations on your first place, will you open source your training code? Want to learn the details of training from code
", "votes": 1, "replies": [ { "id": 1913935, "author_name": "Yauhen Babakhin", "author_url": "", "post_date": "2022-08-25T15:30:30.143000", "content": "Thank you! We will be preparing our solution for the competition organizers, and we might open source it afterwards. However, you can already see the main ideas in our inference kernel here: https://www.kaggle.com/code/philippsinger/team-hydrogen-efficiency-prize-1st-place
", "votes": 3, "replies": [] }, { "id": 1914052, "author_name": "", "author_url": "", "post_date": "2022-08-25T17:22:21.033000", "content": "", "votes": 0, "replies": [] }, { "id": 1919396, "author_name": "Apa", "author_url": "", "post_date": "2022-08-30T11:21:08.760000", "content": "`
\nfor j in range(x.shape[0]):
idx0 = torch.where(\n (input_ids[j] >= min(cfg._tokenizer_start_token_id))\n & (input_ids[j] <= max(cfg._tokenizer_start_token_id))\n)[0]\nidx1 = torch.where(\n (input_ids[j] >= min(cfg._tokenizer_end_token_id))\n & (input_ids[j] <= max(cfg._tokenizer_end_token_id))\n)[0]\nxx = []\nfor jj in range(len(idx0)):\n xx0 = x[j, idx0[jj]]\n xx1 = x[j, idx1[jj]]\n xx2 = x[j, idx0[jj] + 1 : idx1[jj]].mean(dim=0)\n xxx = torch.cat([xx0, xx1, xx2]).unsqueeze(0)\n xx.append(xxx)\nxx = torch.cat(xx)\nret.append(xx)\n`
\nCan you give an example of this code? Thank you very much
Congratulations!
\nCan you post the 1st-level performance of the main version (single model)?
Thank you! The performance of the ensemble of the 1st level models on Private LB is 0.558. So, 2nd level LightGBM and NN bring extra 0.004. And our best single model is the most efficient one with the score of the same 0.558 without 2nd level models.
", "votes": 1, "replies": [] } ] }, { "id": 1912161, "author_name": "leemop", "author_url": "", "post_date": "2022-08-24T14:47:05.910000", "content": "Congraturation for 1st prize! Thanks for great sharing too, :)
", "votes": 1, "replies": [] }, { "id": 1913605, "author_name": "Yuri Barbashov", "author_url": "", "post_date": "2022-08-25T11:52:41.617000", "content": "Congratulations! Can you please elaborate on regularization techniques you tried. Did you use smaller backbone to test hyperparameters? Did you use layerwise learning rate and dropouts?
", "votes": 2, "replies": [ { "id": 1913938, "author_name": "Yauhen Babakhin", "author_url": "", "post_date": "2022-08-25T15:35:32.427000", "content": "Thank you! There were only a few things that worked for us. We didn't use smaller backbones for the tuning, but directly optimized deberta-(v3)-large; for some models we used different learning rates for the backbone and the head; for some models we changed dropout; and for some we applied masking augmentation. An interesting thing is that there is no intermediate dropout in the backbone in our most efficient model, it made results considerably better.
", "votes": 6, "replies": [] }, { "id": 1914089, "author_name": "Yuri Barbashov", "author_url": "", "post_date": "2022-08-25T18:04:42.567000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 1912544, "author_name": "Darragh", "author_url": "", "post_date": "2022-08-24T19:54:04.473000", "content": "Congrats Yauhen, great work and diligence to detail from Team Hydrogen. Nice job 👍
", "votes": 2, "replies": [] }, { "id": 2189217, "author_name": "highscoreman", "author_url": "", "post_date": "2023-03-20T10:31:59.267000", "content": "Congratulations!
", "votes": 0, "replies": [] }, { "id": 2002717, "author_name": "Mathew Murdock", "author_url": "", "post_date": "2022-10-25T03:16:02.937000", "content": "May I ask how much score do you improved when you use pseudo labels method?
", "votes": 0, "replies": [] }, { "id": 1977804, "author_name": "william.wu", "author_url": "", "post_date": "2022-10-08T09:25:23.110000", "content": "Congratulations @ybabakhin, and thanks for sharing your solutions. I have a few questions about the seed blending:
\n\n\nthe data is small to medium size and the metric is log loss, the scores can vary between different runs. This is typical for deep learning models as they are quite dependent on the seed at hand that influences weight initializations, batching, or augmentations. Yet, this means one should not judge model performance on single seeds, and it is better to always evaluate on multiple seeds…
\n
@ybabakhin I wanted to clarify, when you finetuned models, did you train only last linear models or did you train the whole model (all layers)? And how should one choose one of these methods?
", "votes": 0, "replies": [ { "id": 1975526, "author_name": "Psi", "author_url": "", "post_date": "2022-10-06T20:24:31.510000", "content": "ALways finetune all layers.
", "votes": 0, "replies": [] } ] }, { "id": 1912445, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2022-08-24T18:15:27.220000", "content": "Didnt get the logic behind training with 1 batch size?
\nAlso as you know training on this data was complex, the gap between Val and training loss became quickly large, how did you stabilised the training? any tips?
Congrats @ybabakhin and @philippsinger on such an awesome accomplishment! Could you help me understand how this quote works or point me in a direction where I can learn more
\nAbout it?
\n", "votes": 0, "replies": [ { "id": 1912321, "author_name": "Psi", "author_url": "", "post_date": "2022-08-24T16:37:48.193000", "content": "We then run this sample through the backbone, and pool between START and END tokens for each discourse
\n
You just look up the indices of START and END tokens and pool the embeddings between them. You can find code in our efficiency sub: https://www.kaggle.com/code/philippsinger/team-hydrogen-efficiency-prize-1st-place
", "votes": 3, "replies": [] }, { "id": 1912335, "author_name": "RDizzl3", "author_url": "", "post_date": "2022-08-24T16:44:54.127000", "content": "Perfect thanks!
", "votes": 0, "replies": [] } ] }, { "id": 2027456, "author_name": "", "author_url": "", "post_date": "2022-11-12T21:00:28.330000", "content": "", "votes": 0, "replies": [] }, { "id": 1974214, "author_name": "", "author_url": "", "post_date": "2022-10-06T06:06:20.303000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 26, "messages": [ { "id": 1912155, "postDate": "2022-08-24T14:44:43.957Z", "content": "Thanks a lot to the hosts and Kaggle for hosting this interesting competition, we had great fun working on both the accuracy and efficiency tracks. Also congratulations to all other competitors for the great solutions and results. Our solution is the result of perfect teamwork.
\nOur solution is based on a two-stage approach ensembling multiple transformer-based models with different techniques and adding second level models on top. We additionally employ multiple rounds of pseudo tagging and add pseudo labels with different techniques to our models.
\nThroughout this competition we had near perfect correlation between CV and LB. Whenever we saw some improvement on CV, we saw it reflected in a similar manner on the LB with very small random range. For splitting the folds, we just used an efficiency-stratified split on essays.
\nAs the data is small to medium size and the metric is log loss, the scores can vary between different runs. This is typical for deep learning models as they are quite dependent on the seed at hand that influences weight initializations, batching, or augmentations. Yet, this means one should not judge model performance on single seeds, and it is better to always evaluate on multiple seeds.
\nGiven that model training was quite fast, we thus only relied on checking blends of 3 seeds for each model. Also, single model scores did not correlate well here with their ability to blend into larger ensembles. So a better individual model could have quite a worse performance in the blend, diversity really mattered here. Consequently, we also always checked models in the blend, even if they did not seem too promising on an individual basis. Similar to how we checked CV, we then always subbed a blend of 3 seeds of models trained on the full data.
\nOur correlation looked like follows:
\n
Our final solution is a combination of different modeling approaches. Most of them are based on the idea of training on all discourses from a single essay at the same time. This not only made training and inference much faster, but also improved accuracy significantly. In the following we want to describe our main approaches in more detail. For backbones, we could only get deberta-(v3)-large to work. Other backbones did not improve the ensemble.
\nThe main idea of this approach is to feed a full essay into the model, and pool each discourse separately and then feed it through the final linear layer for prediction. The main approach here is similar to what others shared, but there are some peculiarities and different sub-approaches.
\nOur main version has an input as follows:
\nLead Position Claim Evidence Counterclaim Rebuttal Evidence Counterclaim Concluding Statement [SEP] [START] Hi, i'm Isaac, i'm going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn't know if the landform was created by life on Mars, or if it is just a natural landform. [END] [START] On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I'll be talking about how I think that is is a natural landform [END] … more text follows here\nAs mentioned, one sample includes one essay. We start by adding a list of all types of the discourses in the essay with a SEP token and then we mark the individual discourses with custom START and END tokens. We then run this sample through the backbone, and pool between START and END tokens for each discourse. The input batch size is always 1, and this gets transformed to a batch size that depends on the number of discourses within the essay. These pooled embeddings then run through a final linear layer predicting the class.
\nYou can see that in this example we do not specifically add the type to each discourse, but we use an additional auxiliary loss to predict the type of each one. This helped with regularizing the model and allowed for a bit longer training.
\nAn additional sub-approach does not have this auxiliary loss and trains a model based on the following input:
\nLead Position Claim Evidence Counterclaim Rebuttal Evidence Counterclaim Concluding Statement [SEP] [START_Lead] Hi, i'm Isaac, i'm going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn't know if the landform was created by life on Mars, or if it is just a natural landform. [END_Lead] [START_Position] On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I'll be talking about how I think that is is a natural landform [END_Position] … more text follows here\nWhile the latter approach was better individually on CV, the former approach blended significantly better in our large ensemble.
\nIn this approach the chunks of the essays are treated as separate tokens. Individual chunks either get the efficiency label or “O” label if they are outside of the annotated essay text. And the subsequent process is similar to the idea above: pass the whole essay through the backbone, apply local average poolings for each chunk and add a dense classification layer on top. The input essay didn’t have any extra special tokens apart from discourse type in front of all the chunks.
\nFor diversity, we added the following models with minor impact to our solution:
\nWe spent significant efforts on hyperparameter tuning and playing with various regularization techniques. We implemented quite a few augmentation techniques, but as always they were not really helpful for NLP finetuning. Only mask augmentations worked with decent results as it was bringing some diversity for the ensemble. Hyperparameter tuning was very important though, and it was time well spent.
\nThe log loss metric is always only optimal if the mean prediction per column matches the mean of the target columns for that label. Our different models (specifically if trained in batch-wise manner on essays) do not always reflect this mean very well, which is why we added an additional optimization after each model to adjust to the train mean. This additionally makes the scores more reliable and comparable. We then also employ these learned scaling factors on LB.
\nFor ensembling different models we resorted to directly optimizing the blending weights between the models. Interestingly, we also had several models with negative weights, but this worked for us both on CV as well as LB.
\nWe additionally trained several 2nd level models to further improve our predictions.
\nFor the 2nd level LightGBM model we took the weighted ensemble prediction, together with individual models predictions, and generated some aggregate features based on the whole essay. We trained 2 LightGBM versions with different features and parameters.
\nWe tuned two types of neural networks here. The first takes the weighted ensemble prediction, as well as an average across the essay and across the type within an essay for each of the three target columns as input and trains a three-layer DNN. The second one takes the same features, but on an individual model basis and then uses a three-layer Conv1d with average pooling afterwards.
\nAll together, 2nd level models were consistently bringing us about 0.003-0.005 points on CV and the leaderboard throughout the competition.
\nAnother major part of our solution is pseudo labeling. We applied 3 stages of pseudo labeling on the extra data from the previous Feedback competition. It was done in a leak-free manner for the individual folds and additionally for our models trained on all the data (6 versions of pseudo labels in total). The process consisted of the following steps:
\nApart from using previous Feedback competition data for pseudo labels, it was also used in some models as a pre-training dataset. The model was warmed up on the old data predicting the type of the chunk and further finetuned on the given train data.
\n\n\n# Modeling\n\nOur final solution is a combination of different modeling approaches. Most of them are based on the idea of training on all discourses from a single essay at the same time. This not only made training and inference much faster, but also improved accuracy significantly. In the following we want to describe our main approaches in more detail. For backbones, we could only get deberta-(v3)-large to work. Other backbones did not improve the ensemble.\n\n#### Essay group model\n\nThe main idea of this approach is to feed a full essay into the model, and pool each discourse separately and then feed it through the final linear layer for prediction. The main approach here is similar to what others shared, but there are some peculiarities and different sub-approaches. \n\nOur main version has an input as follows:\n\n```\nLead Position Claim Evidence Counterclaim Rebuttal Evidence Counterclaim Concluding Statement [SEP] [START] Hi, i'm Isaac, i'm going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn't know if the landform was created by life on Mars, or if it is just a natural landform. [END] [START] On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I'll be talking about how I think that is is a natural landform [END] … more text follows here\n```\n\nAs mentioned, one sample includes one essay. We start by adding a list of all types of the discourses in the essay with a SEP token and then we mark the individual discourses with custom START and END tokens. We then run this sample through the backbone, and pool between START and END tokens for each discourse. The input batch size is always 1, and this gets transformed to a batch size that depends on the number of discourses within the essay. These pooled embeddings then run through a final linear layer predicting the class.\n\nYou can see that in this example we do not specifically add the type to each discourse, but we use an additional auxiliary loss to predict the type of each one. This helped with regularizing the model and allowed for a bit longer training.\n\nAn additional sub-approach does not have this auxiliary loss and trains a model based on the following input:\n\n```\nLead Position Claim Evidence Counterclaim Rebuttal Evidence Counterclaim Concluding Statement [SEP] [START_Lead] Hi, i'm Isaac, i'm going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn't know if the landform was created by life on Mars, or if it is just a natural landform. [END_Lead] [START_Position] On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I'll be talking about how I think that is is a natural landform [END_Position] … more text follows here\n```\n\nWhile the latter approach was better individually on CV, the former approach blended significantly better in our large ensemble.\n\n\n#### Token classification\n\nIn this approach the chunks of the essays are treated as separate tokens. Individual chunks either get the efficiency label or “O” label if they are outside of the annotated essay text. And the subsequent process is similar to the idea above: pass the whole essay through the backbone, apply local average poolings for each chunk and add a dense classification layer on top. The input essay didn’t have any extra special tokens apart from discourse type in front of all the chunks.\n\n#### Further models\n\nFor diversity, we added the following models with minor impact to our solution:\n\n* Simple Deberta classification on Discourse input only\n* Bag-of-words LightGBM model \n\n# Regularization and hyperparameter tuning\n\nWe spent significant efforts on hyperparameter tuning and playing with various regularization techniques. We implemented quite a few augmentation techniques, but as always they were not really helpful for NLP finetuning. Only mask augmentations worked with decent results as it was bringing some diversity for the ensemble. Hyperparameter tuning was very important though, and it was time well spent.\n\n# Adjustment, ensembling & 2nd level models\n\nThe log loss metric is always only optimal if the mean prediction per column matches the mean of the target columns for that label. Our different models (specifically if trained in batch-wise manner on essays) do not always reflect this mean very well, which is why we added an additional optimization after each model to adjust to the train mean. This additionally makes the scores more reliable and comparable. We then also employ these learned scaling factors on LB.\n\nFor ensembling different models we resorted to directly optimizing the blending weights between the models. Interestingly, we also had several models with negative weights, but this worked for us both on CV as well as LB.\n\nWe additionally trained several 2nd level models to further improve our predictions.\n\n#### LightGBMs\n\nFor the 2nd level LightGBM model we took the weighted ensemble prediction, together with individual models predictions, and generated some aggregate features based on the whole essay. We trained 2 LightGBM versions with different features and parameters.\n\n#### Neural networks\n\nWe tuned two types of neural networks here. The first takes the weighted ensemble prediction, as well as an average across the essay and across the type within an essay for each of the three target columns as input and trains a three-layer DNN. The second one takes the same features, but on an individual model basis and then uses a three-layer Conv1d with average pooling afterwards.\n\nAll together, 2nd level models were consistently bringing us about 0.003-0.005 points on CV and the leaderboard throughout the competition.\n\n# Pseudo labels\n\nAnother major part of our solution is pseudo labeling. We applied 3 stages of pseudo labeling on the extra data from the previous Feedback competition. It was done in a leak-free manner for the individual folds and additionally for our models trained on all the data (6 versions of pseudo labels in total). The process consisted of the following steps:\n\n\n1. Train an ensemble of models only on the given train data\n2. Run predictions on the previous Feedback competition data using our full 2-stage pipeline\n3. Use soft pseudo labels from this extra dataset and apply it to modeling in two different ways:\n * Concatenate pseudo labels with the actual labels in the given train data, and train simultaneously on all this data\n * Pre-train models on the pseudo labels and finetune it only on the given train data afterwards. Similar to: https://arxiv.org/abs/1904.04445\n4. Repeat steps 1-3 three times using an ensemble of models trained on pseudo labels now\n\nApart from using previous Feedback competition data for pseudo labels, it was also used in some models as a pre-training dataset. The model was warmed up on the old data predicting the type of the chunk and further finetuned on the given train data.\n\n# Efficiency solution\n\n* Please refer to [this](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347537) post to read about our efficiency solution.\n* And [here](https://www.kaggle.com/code/philippsinger/team-hydrogen-efficiency-prize-1st-place) you can find our most efficient kernel that gets 0.557 Private LB scores in 5 minutes and 40 seconds!\n\n### Extra links\n\n* [Here](https://www.kaggle.com/code/ybabakhin/team-hydrogen-1st-place) you can find our final inference kernel\n* [Here](https://github.com/ybabakhin/kaggle-feedback-effectiveness-1st-place-solution) you can find our code to train the models\n",
"votes": 141
},
{
"id": 1915341,
"postDate": "2022-08-26T22:35:11.110Z",
"content": "Congratulations Psi and Yauhen! Great solution!
\n\n\nPre-train models on the pseudo labels and finetune it only on the given train data afterwards
\n
This is what i just did in Amex competition. It worked great there too discussion.
", "rawMarkdown": "Congratulations Psi and Yauhen! Great solution!\n> Pre-train models on the pseudo labels and finetune it only on the given train data afterwards\n\nThis is what i just did in Amex competition. It worked great there too [discussion][1].\n\n[1]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641", "votes": 11 }, { "id": 1912323, "postDate": "2022-08-24T16:38:16.397Z", "content": "Congratulations @ybabakhin and @philippsinger ! Amazing finishing 1st in both tracks! It was a great battle in both tracks.. Your team definitely kept us on our feet!
", "rawMarkdown": "Congratulations @ybabakhin and @philippsinger ! Amazing finishing 1st in both tracks! It was a great battle in both tracks.. Your team definitely kept us on our feet! ", "votes": 5 }, { "id": 1917265, "postDate": "2022-08-28T15:33:28.987Z", "content": "Congratulations on winning 1st place!
\nCould you please provide some additional explanation with the code for the following part?
\n\n", "rawMarkdown": "Congratulations on winning 1st place!\n\nCould you please provide some additional explanation with the code for the following part?\n\n> The log loss metric is always only optimal if the mean prediction per column matches the mean of the target columns for that label. Our different models (specifically if trained in batch-wise manner on essays) do not always reflect this mean very well, which is why we added an additional optimization after each model to adjust to the train mean. This additionally makes the scores more reliable and comparable.\n", "votes": 3 }, { "id": 1927252, "postDate": "2022-09-05T13:33:54.047Z", "content": "The log loss metric is always only optimal if the mean prediction per column matches the mean of the target columns for that label. Our different models (specifically if trained in batch-wise manner on essays) do not always reflect this mean very well, which is why we added an additional optimization after each model to adjust to the train mean. This additionally makes the scores more reliable and comparable.
\n
Congratulations! could you please tell me more about the details of mask augmentation? how to do it?
", "rawMarkdown": "Congratulations! could you please tell me more about the details of mask augmentation? how to do it?", "votes": 1 }, { "id": 1912900, "postDate": "2022-08-25T03:08:07.880Z", "content": "Congratulations on your first place, will you open source your training code? Want to learn the details of training from code
", "rawMarkdown": "Congratulations on your first place, will you open source your training code? Want to learn the details of training from code", "votes": 1, "replies": [ { "id": 1913935, "postDate": "2022-08-25T15:30:30.143Z", "content": "Thank you! We will be preparing our solution for the competition organizers, and we might open source it afterwards. However, you can already see the main ideas in our inference kernel here: https://www.kaggle.com/code/philippsinger/team-hydrogen-efficiency-prize-1st-place
", "rawMarkdown": "Thank you! We will be preparing our solution for the competition organizers, and we might open source it afterwards. However, you can already see the main ideas in our inference kernel here: https://www.kaggle.com/code/philippsinger/team-hydrogen-efficiency-prize-1st-place", "votes": 3 }, { "id": 1914052, "postDate": "2022-08-25T17:22:21.033Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1919396, "postDate": "2022-08-30T11:21:08.760Z", "content": "`
\nfor j in range(x.shape[0]):
idx0 = torch.where(\n (input_ids[j] >= min(cfg._tokenizer_start_token_id))\n & (input_ids[j] <= max(cfg._tokenizer_start_token_id))\n)[0]\nidx1 = torch.where(\n (input_ids[j] >= min(cfg._tokenizer_end_token_id))\n & (input_ids[j] <= max(cfg._tokenizer_end_token_id))\n)[0]\nxx = []\nfor jj in range(len(idx0)):\n xx0 = x[j, idx0[jj]]\n xx1 = x[j, idx1[jj]]\n xx2 = x[j, idx0[jj] + 1 : idx1[jj]].mean(dim=0)\n xxx = torch.cat([xx0, xx1, xx2]).unsqueeze(0)\n xx.append(xxx)\nxx = torch.cat(xx)\nret.append(xx)\n`
\nCan you give an example of this code? Thank you very much
Congratulations!
\nCan you post the 1st-level performance of the main version (single model)?
Thank you! The performance of the ensemble of the 1st level models on Private LB is 0.558. So, 2nd level LightGBM and NN bring extra 0.004. And our best single model is the most efficient one with the score of the same 0.558 without 2nd level models.
", "rawMarkdown": "Thank you! The performance of the ensemble of the 1st level models on Private LB is 0.558. So, 2nd level LightGBM and NN bring extra 0.004. And our best single model is the most [efficient](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347537) one with the score of the same 0.558 without 2nd level models.", "votes": 1 } ] }, { "id": 1912161, "postDate": "2022-08-24T14:47:05.910Z", "content": "Congraturation for 1st prize! Thanks for great sharing too, :)
", "rawMarkdown": "Congraturation for 1st prize! Thanks for great sharing too, :)", "votes": 1 }, { "id": 1913605, "postDate": "2022-08-25T11:52:41.617Z", "content": "Congratulations! Can you please elaborate on regularization techniques you tried. Did you use smaller backbone to test hyperparameters? Did you use layerwise learning rate and dropouts?
", "rawMarkdown": "Congratulations! Can you please elaborate on regularization techniques you tried. Did you use smaller backbone to test hyperparameters? Did you use layerwise learning rate and dropouts?", "votes": 2, "replies": [ { "id": 1913938, "postDate": "2022-08-25T15:35:32.427Z", "content": "Thank you! There were only a few things that worked for us. We didn't use smaller backbones for the tuning, but directly optimized deberta-(v3)-large; for some models we used different learning rates for the backbone and the head; for some models we changed dropout; and for some we applied masking augmentation. An interesting thing is that there is no intermediate dropout in the backbone in our most efficient model, it made results considerably better.
", "rawMarkdown": "Thank you! There were only a few things that worked for us. We didn't use smaller backbones for the tuning, but directly optimized deberta-(v3)-large; for some models we used different learning rates for the backbone and the head; for some models we changed dropout; and for some we applied masking augmentation. An interesting thing is that there is no intermediate dropout in the backbone in our most efficient model, it made results considerably better.", "votes": 6 }, { "id": 1914089, "postDate": "2022-08-25T18:04:42.567Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 1912544, "postDate": "2022-08-24T19:54:04.473Z", "content": "Congrats Yauhen, great work and diligence to detail from Team Hydrogen. Nice job 👍
", "rawMarkdown": "Congrats Yauhen, great work and diligence to detail from Team Hydrogen. Nice job 👍 ", "votes": 2 }, { "id": 2189217, "postDate": "2023-03-20T10:31:59.267Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations! " }, { "id": 2002717, "postDate": "2022-10-25T03:16:02.937Z", "content": "May I ask how much score do you improved when you use pseudo labels method?
", "rawMarkdown": "May I ask how much score do you improved when you use pseudo labels method?" }, { "id": 1977804, "postDate": "2022-10-08T09:25:23.110Z", "content": "Congratulations @ybabakhin, and thanks for sharing your solutions. I have a few questions about the seed blending:
\n\n\nthe data is small to medium size and the metric is log loss, the scores can vary between different runs. This is typical for deep learning models as they are quite dependent on the seed at hand that influences weight initializations, batching, or augmentations. Yet, this means one should not judge model performance on single seeds, and it is better to always evaluate on multiple seeds…
\n
@ybabakhin I wanted to clarify, when you finetuned models, did you train only last linear models or did you train the whole model (all layers)? And how should one choose one of these methods?
", "rawMarkdown": "@ybabakhin I wanted to clarify, when you finetuned models, did you train only last linear models or did you train the whole model (all layers)? And how should one choose one of these methods?", "replies": [ { "id": 1975526, "postDate": "2022-10-06T20:24:31.510Z", "content": "ALways finetune all layers.
", "rawMarkdown": "ALways finetune all layers." } ] }, { "id": 1912445, "postDate": "2022-08-24T18:15:27.220Z", "content": "Didnt get the logic behind training with 1 batch size?
\nAlso as you know training on this data was complex, the gap between Val and training loss became quickly large, how did you stabilised the training? any tips?
Congrats @ybabakhin and @philippsinger on such an awesome accomplishment! Could you help me understand how this quote works or point me in a direction where I can learn more
\nAbout it?
\n", "rawMarkdown": "Congrats @ybabakhin and @philippsinger on such an awesome accomplishment! Could you help me understand how this quote works or point me in a direction where I can learn more\nAbout it? \n\n> We then run this sample through the backbone, and pool between START and END tokens for each discourse", "replies": [ { "id": 1912321, "postDate": "2022-08-24T16:37:48.193Z", "content": "We then run this sample through the backbone, and pool between START and END tokens for each discourse
\n
You just look up the indices of START and END tokens and pool the embeddings between them. You can find code in our efficiency sub: https://www.kaggle.com/code/philippsinger/team-hydrogen-efficiency-prize-1st-place
", "rawMarkdown": "You just look up the indices of START and END tokens and pool the embeddings between them. You can find code in our efficiency sub: https://www.kaggle.com/code/philippsinger/team-hydrogen-efficiency-prize-1st-place", "votes": 3 }, { "id": 1912335, "postDate": "2022-08-24T16:44:54.127Z", "content": "Perfect thanks!
", "rawMarkdown": "Perfect thanks! " } ] }, { "id": 2027456, "postDate": "2022-11-12T21:00:28.330Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1974214, "postDate": "2022-10-06T06:06:20.303Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:52:15Z", "competition_id": 35308, "competition_slug": "feedback-prize-effectiveness", "team_id": 8682044, "solution_url": "/competitions/feedback-prize-effectiveness/writeups/team-hydrogen-team-hydrogen-1st-place-solution" }