Sept 10, 2021 - added link to submission notebook
\nAug 16, 2021 - added code link
First, I would like to thank Kaggle and CommonLit for hosting this competition and the community here at Kaggle for making this platform a wonderful place to learn. This is my second competition at Kaggle, with the first one being the Coleridge Initiative; I have to say the past three months or so has been a tremendous journey. If anyone told me three months ago that I would end up in 3rd place in a Kaggle competition, I would not believe it. None of this would be possible without Kaggle and its incredible community.
\nSpecial thanks to:
\n@rhtsingh for sharing so many great notebooks with detailed and excellent explanations. Those notebooks guided my first few weeks in this competition.
\n@andretugan for sharing a simple yet elegant solution. Some parts of my final model are from his approach.
Before I go over my approach, I'd like to share my point of view on the competition objective. Please correct me if what I say is wrong or doesn't make sense to you.
\nThis competition is characterized by small training data and noisy, non-intuitive to understand target values. The target is generated by the Bradley-Terry algorithm from a relatively small number of pair-wise comparison data. From my intuition, Bradley-Terry acts like an Elo rating system where match results are translated to raw numbers representing the \"strength\" of each player. (again, this could be wrong and is merely my point of view) In this competition, the texts are players, and the \"strength\" is \"easiness to read,\" where an easier-to-read text would score higher.
\nTraining data + public/private data nets to little less than 5k samples, and each sample is compared with others 46.47 times on average. That means each text was compared to only ~1% of the entire dataset on average. Imagine a chess player's Elo where the player has only played against 1% of the whole chess player population, and the opponents are selected at random. I believe most of the noise is due to the small number of comparisons.
\nThe objective of the competition is to predict where an arbitrary text would land on that leaderboard generated from 5k texts.
I've played with Roberta base mostly at the beginning of the competition, and with a gru+multihead attention on the last hidden layer + learning rate decay, I was able to get ~0.473 on both public and private LB.
\nI then switched to large and with the same setup I was at around 0.465-0.467 on both public and private LB.
\nAfter which I learned from @andretugan's notebook and incorporated 3 group lr decay, simple attention head, and validation step policy into my solution
\nRoberta-large + simple attention head + learning rate decay: 0.462 on both LB
0.462 + weighted average of all layers and multisample dropout (from google quest #1 solution) + separate lr for head: **0.459 on public **
\n0.459 + a 3 stage lr schedule: 0.454 on public
\n![\"custom](\"https://i.ibb.co/fNjh6pc/LR.png\")
\n0.454 + early stop at the beginning of 3rd epoch: 0.453 on public
\nHowever, the private did not agree with the improvement to 0.453, which I didn't know at the time. Private LB stayed at ~0.461
\nAn ensemble of 2 Roberta large and a Roberta base trained this way, however, got 0.454 on public and 0.456 on private.
I then used two of my 0.453 public models to label extra data consists of simple wiki dump, children's book test, and OneStopEnglishCropus.
\nUsing all folds to label lead to unrealistic low cv (due to leak) and no improvement to LB, so I used 5 fold pseudo label described in google quest #1 solution.
I first train the model on pseudo labeled data and then finish up with low lr 3 epoch and an lr scheduler with a decaying rate (1,0.5,0.25) for each epoch, no warmup.
\n2 cycles improved my public to 0.448: 0.453 -> 0.450 -> 0.448
\nyet private still stayed at ~0.461
I then used the average of two 0.448 models to label the data again and used that data to train a Deberta-large; Deberta ends up with 0.448 on public and 0.449 on private
\nensemble of a Roberta-large and a Deberta-large: 0.442 on public, 0.448 on private
\nAnother model trained with a further cycle labeled with 0.442 public was added to the blend and got 0.442/0.447 on public/private. This is the best scoring out of the two I selected.
\nBest private was 0.446, which I did not select because it didn't improve public LB. (I turned my eyes away from cv long ago)
\nThe difference of this one to the 0.447 private is the third model. The new model is trained with training data mixed into pseudo labeled extra data. The train data however, was relabeled and mixed with the original train data with weight of 0.5. I tested 0.8 and 1 but 0.5 performed the best.
1080ti at the beginning of the competition
\nbought a 3090 when switching to large models
what worked:
\ncode: https://github.com/Danielhuxc/CLRP-solution
\nnotebook: https://www.kaggle.com/potatoc5/final-sub
PS: I am looking for ML/DL job (preferably within Canada), please DM me if you would like to get in touch.
", "comments": [ { "id": 1433554, "author_name": "عثمان", "author_url": "", "post_date": "2021-08-03T14:05:50.033000", "content": "\n\n1080ti at the beginning of the competition
\n
\n bought a 3090 when switching to large models
That's the kind of dedication needed! I'm sure the job will come soon and congratulations on the well deserved strong finish!
", "votes": 3, "replies": [] }, { "id": 1426853, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-03T07:56:40.433000", "content": "Nice writeup!
\nYou forgot to mention you could have ended up with 2 solo golds in your first 3 months of Kaggle if you had not forgotten to submit your entries for Coleridge..
\nAnyway a tremendous achievement in such a short span!
Thank you for your kind words! I've learned so much in the past 3 months and a solo gold with a prize is more than I can dream of. I consider myself very lucky already. what a wild journey.
", "votes": 4, "replies": [] } ] }, { "id": 1435764, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-03T14:49:59.037000", "content": "Very impressive solution. I like how you did so well with few models. Awesome! Congrats on strong solo finish!
\nI particularly like your cycles of pseudo labeling. Great stuff
", "votes": 4, "replies": [] }, { "id": 1429795, "author_name": "Miguel González González", "author_url": "", "post_date": "2021-08-03T10:59:26.597000", "content": "Thanks for sharing! Apparently pseudo labelling extra data was quite helpful from what I'm reading in the forum solutions. Lesson learned!
", "votes": 1, "replies": [] }, { "id": 1426184, "author_name": "Salim Khazem", "author_url": "", "post_date": "2021-08-03T07:21:54.993000", "content": "Great job, very nice performance Congratulation, Thanks for sharing !
\nGood luck for your job search !
Great job, very nice performance Congratulation, Thanks for sharing !
\nGood luck for your job search !
Hi, as I was reading your code, I realised that u pretrained a MLM with the training data before fine tuning, will this cause leakage? As the model had already seen the data, when fine-tuning, wouldnt cv be unreliable?
", "votes": 0, "replies": [] }, { "id": 1507499, "author_name": "LuzhangCV", "author_url": "", "post_date": "2021-09-09T09:00:13.150000", "content": "First of all, I congratulate you on winning the gold medal. I am a novice at bert. I am participating in a text classification competition. I intend to reuse your skills, but I don’t know how to adjust the parameters properly. My f1-score has risen slightly, and the loss is also It is rising, and finally the loss continues to rise, but the f1-score is constantly oscillating.
", "votes": 0, "replies": [] }, { "id": 1489160, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2021-08-24T18:43:32.830000", "content": "@potatoc5 congrats. Very nice solution. Can you explain a bit about your lr_schedulers('custom' and '3stage') and how you come up with these schedulers with specific configurations? Can you share some resources or research papers that led you to design schedulers in this way?
There might be some paper out there about lr schedule but I did not follow one. I tuned the lr scheduler by experimenting with different settings and kept the one that's learning the fastest (faster decrease of val loss) with the least amount of data. Stability of val loss is also important.
", "votes": 0, "replies": [] } ] }, { "id": 1457091, "author_name": "Shibata", "author_url": "", "post_date": "2021-08-07T08:21:04.787000", "content": "@potatoc5 Thank you for sharing this great solution. I learned a lot from this. I was very surprised to see 3 stage learning rate is so effective in this competition👀
\nI too tried to pseudo labelling to Simple/English Wikipedia data and experienced low CV score (CV:0.445, LB:0.456). I assume this is caused by leakage you mentioned in this thread.
\nPlease let me know your CV/LB difference before/after below processing.
\n\nUsing all folds to label lead to unrealistic low cv (due to leak) and no improvement to LB, so I used 5 fold pseudo label described in google quest #1 solution.
\n
Moreover, I would like to know how you decided the weights of your weighted average of all BERT's layers.
\nThanks in advance !!
Before/after fold specific pseudo label:
\n| \n | CV | \nPrivate | \nPublic | \n
|---|---|---|---|
| Before | \n449 | \n457 | \n456 | \n
| After | \n479 | \n458 | \n450 | \n
Looking back at it, it did not help in generalization to private LB. Instead, it might have learned more biases from each fold than generalizable knowledge. If you want to use this method, I'd suggest you do your own testing. And if you ever do, I'd appreciate it if you share the result with me.
\nAs for weights for hidden layers, I set them as parameters and have the model learn them during training. I learned this from google quest 1st solution as well.
", "votes": 0, "replies": [] } ] }, { "id": 1450875, "author_name": "Frank", "author_url": "", "post_date": "2021-08-05T07:47:48.967000", "content": "Great job with the solo gold! @potatoc5
\nHow did you set 3 group decaying lr for large models? Did you use a uniform strategy or tune it for each model?
\nAnd how did you set the 3 stage lr schedule? Thanks.
Thank you! For 3 group decaying lr I tuned a single parameter decay_rate/layer on base model. When used on large models I use the square root of it to maintain the relative lr ratio between groups. 3 stage lr schedule is made by eyeballing validation loss, while it should be different for every data-model combination I find my schedule work reasonably well for all models I used. The only graph in the original post is a full cycle of my lr scheduler.
", "votes": 2, "replies": [] } ] }, { "id": 1449712, "author_name": "HinePo", "author_url": "", "post_date": "2021-08-05T00:14:03.973000", "content": "Thanks for letting us know your path and how you got so many improvements. In this comp I also decided to trust LB, and even so it was very hard and tricky to select only two submissions.
", "votes": 0, "replies": [] }, { "id": 1443040, "author_name": "takuoko", "author_url": "", "post_date": "2021-08-04T01:30:53.647000", "content": "Congrats! Thank you for your amazing writeup.
\nI have a question.
\n\nI then used the average of two 0.448 models to label the data again and used that data to train a Deberta-large; Deberta ends up with 0.448 on public and 0.449 on private
\n
In this part, are you doing relabel for train data or external data?
", "votes": 0, "replies": [ { "id": 1443161, "author_name": "hxcc", "author_url": "", "post_date": "2021-08-04T01:57:25.720000", "content": "Thank you for your question. In the part you quoted I relabeled external data only, however, I did make an attempt to relabel train data in the last day and it did boost my private to 0.446. I've updated the details about 0.446 private in the original post. I did not have the time to pursue this path but I think with a proper pipeline single model performance could be further boosted.
", "votes": 1, "replies": [] } ] }, { "id": 1428669, "author_name": "", "author_url": "", "post_date": "2021-08-03T09:41:55.947000", "content": "", "votes": 1, "replies": [] } ], "comment_count": 19, "messages": [ { "id": 1425582, "postDate": "2021-08-03T06:55:49.757Z", "content": "Sept 10, 2021 - added link to submission notebook
\nAug 16, 2021 - added code link
First, I would like to thank Kaggle and CommonLit for hosting this competition and the community here at Kaggle for making this platform a wonderful place to learn. This is my second competition at Kaggle, with the first one being the Coleridge Initiative; I have to say the past three months or so has been a tremendous journey. If anyone told me three months ago that I would end up in 3rd place in a Kaggle competition, I would not believe it. None of this would be possible without Kaggle and its incredible community.
\nSpecial thanks to:
\n@rhtsingh for sharing so many great notebooks with detailed and excellent explanations. Those notebooks guided my first few weeks in this competition.
\n@andretugan for sharing a simple yet elegant solution. Some parts of my final model are from his approach.
Before I go over my approach, I'd like to share my point of view on the competition objective. Please correct me if what I say is wrong or doesn't make sense to you.
\nThis competition is characterized by small training data and noisy, non-intuitive to understand target values. The target is generated by the Bradley-Terry algorithm from a relatively small number of pair-wise comparison data. From my intuition, Bradley-Terry acts like an Elo rating system where match results are translated to raw numbers representing the \"strength\" of each player. (again, this could be wrong and is merely my point of view) In this competition, the texts are players, and the \"strength\" is \"easiness to read,\" where an easier-to-read text would score higher.
\nTraining data + public/private data nets to little less than 5k samples, and each sample is compared with others 46.47 times on average. That means each text was compared to only ~1% of the entire dataset on average. Imagine a chess player's Elo where the player has only played against 1% of the whole chess player population, and the opponents are selected at random. I believe most of the noise is due to the small number of comparisons.
\nThe objective of the competition is to predict where an arbitrary text would land on that leaderboard generated from 5k texts.
I've played with Roberta base mostly at the beginning of the competition, and with a gru+multihead attention on the last hidden layer + learning rate decay, I was able to get ~0.473 on both public and private LB.
\nI then switched to large and with the same setup I was at around 0.465-0.467 on both public and private LB.
\nAfter which I learned from @andretugan's notebook and incorporated 3 group lr decay, simple attention head, and validation step policy into my solution
\nRoberta-large + simple attention head + learning rate decay: 0.462 on both LB
0.462 + weighted average of all layers and multisample dropout (from google quest #1 solution) + separate lr for head: **0.459 on public **
\n0.459 + a 3 stage lr schedule: 0.454 on public
\n
\n0.454 + early stop at the beginning of 3rd epoch: 0.453 on public
\nHowever, the private did not agree with the improvement to 0.453, which I didn't know at the time. Private LB stayed at ~0.461
\nAn ensemble of 2 Roberta large and a Roberta base trained this way, however, got 0.454 on public and 0.456 on private.
I then used two of my 0.453 public models to label extra data consists of simple wiki dump, children's book test, and OneStopEnglishCropus.
\nUsing all folds to label lead to unrealistic low cv (due to leak) and no improvement to LB, so I used 5 fold pseudo label described in google quest #1 solution.
I first train the model on pseudo labeled data and then finish up with low lr 3 epoch and an lr scheduler with a decaying rate (1,0.5,0.25) for each epoch, no warmup.
\n2 cycles improved my public to 0.448: 0.453 -> 0.450 -> 0.448
\nyet private still stayed at ~0.461
I then used the average of two 0.448 models to label the data again and used that data to train a Deberta-large; Deberta ends up with 0.448 on public and 0.449 on private
\nensemble of a Roberta-large and a Deberta-large: 0.442 on public, 0.448 on private
\nAnother model trained with a further cycle labeled with 0.442 public was added to the blend and got 0.442/0.447 on public/private. This is the best scoring out of the two I selected.
\nBest private was 0.446, which I did not select because it didn't improve public LB. (I turned my eyes away from cv long ago)
\nThe difference of this one to the 0.447 private is the third model. The new model is trained with training data mixed into pseudo labeled extra data. The train data however, was relabeled and mixed with the original train data with weight of 0.5. I tested 0.8 and 1 but 0.5 performed the best.
1080ti at the beginning of the competition
\nbought a 3090 when switching to large models
what worked:
\ncode: https://github.com/Danielhuxc/CLRP-solution
\nnotebook: https://www.kaggle.com/potatoc5/final-sub
PS: I am looking for ML/DL job (preferably within Canada), please DM me if you would like to get in touch.
", "rawMarkdown": "### update\nSept 10, 2021 - added link to submission notebook\nAug 16, 2021 - added code link\n\n###**tl;dr at the bottom**\n\nFirst, I would like to thank Kaggle and CommonLit for hosting this competition and the community here at Kaggle for making this platform a wonderful place to learn. This is my second competition at Kaggle, with the first one being the Coleridge Initiative; I have to say the past three months or so has been a tremendous journey. If anyone told me three months ago that I would end up in 3rd place in a Kaggle competition, I would not believe it. None of this would be possible without Kaggle and its incredible community.\n\nSpecial thanks to:\n@rhtsingh for sharing so many great notebooks with detailed and excellent explanations. Those notebooks guided my first few weeks in this competition. \n@andretugan for sharing a simple yet elegant solution. Some parts of my final model are from his approach.\n\n## Analysis of the objective\nBefore I go over my approach, I'd like to share my point of view on the competition objective. Please correct me if what I say is wrong or doesn't make sense to you.\nThis competition is characterized by small training data and noisy, non-intuitive to understand target values. The target is generated by the Bradley-Terry algorithm from a relatively small number of pair-wise comparison data. From my intuition, Bradley-Terry acts like an Elo rating system where match results are translated to raw numbers representing the \"strength\" of each player. (again, this could be wrong and is merely my point of view) In this competition, the texts are players, and the \"strength\" is \"easiness to read,\" where an easier-to-read text would score higher. \nTraining data + public/private data nets to little less than 5k samples, and each sample is compared with others 46.47 times on average. That means each text was compared to only ~1% of the entire dataset on average. Imagine a chess player's Elo where the player has only played against 1% of the whole chess player population, and the opponents are selected at random. I believe most of the noise is due to the small number of comparisons. \nThe objective of the competition is to predict where an arbitrary text would land on that leaderboard generated from 5k texts.\n\n##Path to public LB 0.442:\nI've played with Roberta base mostly at the beginning of the competition, and with a gru+multihead attention on the last hidden layer + learning rate decay, I was able to get **~0.473 on both public and private LB.**\nI then switched to large and with the same setup I was at around **0.465-0.467 on both public and private LB.**\nAfter which I learned from @andretugan's notebook and incorporated 3 group lr decay, simple attention head, and validation step policy into my solution\nRoberta-large + simple attention head + learning rate decay: **0.462 on both LB**\n\n0.462 + weighted average of all layers and multisample dropout (from google quest #1 solution) + separate lr for head: **0.459 on public **\n0.459 + a 3 stage lr schedule: **0.454 on public**\n\n0.454 + early stop at the beginning of 3rd epoch: **0.453 on public**\nHowever, **the private did not agree with the improvement to 0.453**, which I didn't know at the time. **Private LB stayed at ~0.461**\nAn ensemble of 2 Roberta large and a Roberta base trained this way, however, got **0.454 on public** and **0.456 on private**. \n\nI then used two of my 0.453 public models to label extra data consists of simple wiki dump, children's book test, and OneStopEnglishCropus.\nUsing all folds to label lead to unrealistic low cv (due to leak) and no improvement to LB, so I used 5 fold pseudo label described in google quest #1 solution. \n\nI first train the model on pseudo labeled data and then finish up with low lr 3 epoch and an lr scheduler with a decaying rate (1,0.5,0.25) for each epoch, no warmup.\n\n2 cycles improved my public to 0.448: **0.453 -> 0.450 -> 0.448**\n**yet private still stayed at ~0.461**\n\nI then used the average of two 0.448 models to label the data again and used that data to train a Deberta-large; Deberta ends up with **0.448 on public and 0.449 on private**\n\nensemble of a Roberta-large and a Deberta-large: **0.442 on public, 0.448 on private**\n\nAnother model trained with a further cycle labeled with 0.442 public was added to the blend and got **0.442/0.447 on public/private**. This is the best scoring out of the two I selected.\n\nBest private was **0.446**, which I did not select because it didn't improve public LB. (I turned my eyes away from cv long ago) \nThe difference of this one to the 0.447 private is the third model. The new model is trained with training data mixed into pseudo labeled extra data. The train data however, was relabeled and mixed with the original train data with weight of 0.5. I tested 0.8 and 1 but 0.5 performed the best.\n\n##what I wish I had time to do:\n- ensembling techniques, In my solution I only used simple average as I didn't have time to explore ensembling techniques\n- better pseudo-label pipeline. I only tried a few variants and I believe some are always better than others \n- pairwise comparison model which simply outputs which text is easier to read to generate raw data for Bradley-Terry algorithm, convert it to target score using Bradley-Terry algorithm, and then maybe recalibrate to competition training data distribution. (This might be a wild-goose chase, but seems like a very promising one to me)\n\n##hardware:\n1080ti at the beginning of the competition \nbought a 3090 when switching to large models\n\n##tl;dr:\nwhat worked:\n- 3 group decaying lr for encoder\n- simple attention head\n- separate lr for head\n- multi-sample dropout\n- weighted average of hidden layers\n- early stop\n- fold specific pseudo-label\n- Deberta-large\n\ncode: https://github.com/Danielhuxc/CLRP-solution\nnotebook: https://www.kaggle.com/potatoc5/final-sub\n\nPS: I am looking for ML/DL job (preferably within Canada), please DM me if you would like to get in touch.", "votes": 111 }, { "id": 1433554, "postDate": "2021-08-03T14:05:50.033Z", "content": "\n\n1080ti at the beginning of the competition
\n
\n bought a 3090 when switching to large models
That's the kind of dedication needed! I'm sure the job will come soon and congratulations on the well deserved strong finish!
", "rawMarkdown": "> 1080ti at the beginning of the competition\nbought a 3090 when switching to large models\n\nThat's the kind of dedication needed! I'm sure the job will come soon and congratulations on the well deserved strong finish!", "votes": 3 }, { "id": 1426853, "postDate": "2021-08-03T07:56:40.433Z", "content": "Nice writeup!
\nYou forgot to mention you could have ended up with 2 solo golds in your first 3 months of Kaggle if you had not forgotten to submit your entries for Coleridge..
\nAnyway a tremendous achievement in such a short span!
Thank you for your kind words! I've learned so much in the past 3 months and a solo gold with a prize is more than I can dream of. I consider myself very lucky already. what a wild journey.
", "rawMarkdown": "Thank you for your kind words! I've learned so much in the past 3 months and a solo gold with a prize is more than I can dream of. I consider myself very lucky already. what a wild journey. ", "votes": 4 } ] }, { "id": 1435764, "postDate": "2021-08-03T14:49:59.037Z", "content": "Very impressive solution. I like how you did so well with few models. Awesome! Congrats on strong solo finish!
\nI particularly like your cycles of pseudo labeling. Great stuff
", "rawMarkdown": "Very impressive solution. I like how you did so well with few models. Awesome! Congrats on strong solo finish!\n\nI particularly like your cycles of pseudo labeling. Great stuff", "votes": 4 }, { "id": 1429795, "postDate": "2021-08-03T10:59:26.597Z", "content": "Thanks for sharing! Apparently pseudo labelling extra data was quite helpful from what I'm reading in the forum solutions. Lesson learned!
", "rawMarkdown": "Thanks for sharing! Apparently pseudo labelling extra data was quite helpful from what I'm reading in the forum solutions. Lesson learned!", "votes": 1 }, { "id": 1426184, "postDate": "2021-08-03T07:21:54.993Z", "content": "Great job, very nice performance Congratulation, Thanks for sharing !
\nGood luck for your job search !
Great job, very nice performance Congratulation, Thanks for sharing !
\nGood luck for your job search !
Hi, as I was reading your code, I realised that u pretrained a MLM with the training data before fine tuning, will this cause leakage? As the model had already seen the data, when fine-tuning, wouldnt cv be unreliable?
", "rawMarkdown": "Hi, as I was reading your code, I realised that u pretrained a MLM with the training data before fine tuning, will this cause leakage? As the model had already seen the data, when fine-tuning, wouldnt cv be unreliable?" }, { "id": 1507499, "postDate": "2021-09-09T09:00:13.150Z", "content": "First of all, I congratulate you on winning the gold medal. I am a novice at bert. I am participating in a text classification competition. I intend to reuse your skills, but I don’t know how to adjust the parameters properly. My f1-score has risen slightly, and the loss is also It is rising, and finally the loss continues to rise, but the f1-score is constantly oscillating.
", "rawMarkdown": "First of all, I congratulate you on winning the gold medal. I am a novice at bert. I am participating in a text classification competition. I intend to reuse your skills, but I don’t know how to adjust the parameters properly. My f1-score has risen slightly, and the loss is also It is rising, and finally the loss continues to rise, but the f1-score is constantly oscillating." }, { "id": 1489160, "postDate": "2021-08-24T18:43:32.830Z", "content": "@potatoc5 congrats. Very nice solution. Can you explain a bit about your lr_schedulers('custom' and '3stage') and how you come up with these schedulers with specific configurations? Can you share some resources or research papers that led you to design schedulers in this way?
There might be some paper out there about lr schedule but I did not follow one. I tuned the lr scheduler by experimenting with different settings and kept the one that's learning the fastest (faster decrease of val loss) with the least amount of data. Stability of val loss is also important.
", "rawMarkdown": "There might be some paper out there about lr schedule but I did not follow one. I tuned the lr scheduler by experimenting with different settings and kept the one that's learning the fastest (faster decrease of val loss) with the least amount of data. Stability of val loss is also important." } ] }, { "id": 1457091, "postDate": "2021-08-07T08:21:04.787Z", "content": "@potatoc5 Thank you for sharing this great solution. I learned a lot from this. I was very surprised to see 3 stage learning rate is so effective in this competition👀
\nI too tried to pseudo labelling to Simple/English Wikipedia data and experienced low CV score (CV:0.445, LB:0.456). I assume this is caused by leakage you mentioned in this thread.
\nPlease let me know your CV/LB difference before/after below processing.
\n\nUsing all folds to label lead to unrealistic low cv (due to leak) and no improvement to LB, so I used 5 fold pseudo label described in google quest #1 solution.
\n
Moreover, I would like to know how you decided the weights of your weighted average of all BERT's layers.
\nThanks in advance !!
Before/after fold specific pseudo label:
\n| \n | CV | \nPrivate | \nPublic | \n
|---|---|---|---|
| Before | \n449 | \n457 | \n456 | \n
| After | \n479 | \n458 | \n450 | \n
Looking back at it, it did not help in generalization to private LB. Instead, it might have learned more biases from each fold than generalizable knowledge. If you want to use this method, I'd suggest you do your own testing. And if you ever do, I'd appreciate it if you share the result with me.
\nAs for weights for hidden layers, I set them as parameters and have the model learn them during training. I learned this from google quest 1st solution as well.
", "rawMarkdown": "Before/after fold specific pseudo label:\n| | CV | Private | Public |\n| --- | --- |\n| Before | 449 | 457 | 456 |\n| After | 479 | 458 | 450 |\n\nLooking back at it, it did not help in generalization to private LB. Instead, it might have learned more biases from each fold than generalizable knowledge. If you want to use this method, I'd suggest you do your own testing. And if you ever do, I'd appreciate it if you share the result with me.\n\nAs for weights for hidden layers, I set them as parameters and have the model learn them during training. I learned this from [google quest 1st solution](https://www.kaggle.com/c/google-quest-challenge/discussion/129840) as well." } ] }, { "id": 1450875, "postDate": "2021-08-05T07:47:48.967Z", "content": "Great job with the solo gold! @potatoc5
\nHow did you set 3 group decaying lr for large models? Did you use a uniform strategy or tune it for each model?
\nAnd how did you set the 3 stage lr schedule? Thanks.
Thank you! For 3 group decaying lr I tuned a single parameter decay_rate/layer on base model. When used on large models I use the square root of it to maintain the relative lr ratio between groups. 3 stage lr schedule is made by eyeballing validation loss, while it should be different for every data-model combination I find my schedule work reasonably well for all models I used. The only graph in the original post is a full cycle of my lr scheduler.
", "rawMarkdown": "Thank you! For 3 group decaying lr I tuned a single parameter decay_rate/layer on base model. When used on large models I use the square root of it to maintain the relative lr ratio between groups. 3 stage lr schedule is made by eyeballing validation loss, while it should be different for every data-model combination I find my schedule work reasonably well for all models I used. The only graph in the original post is a full cycle of my lr scheduler. ", "votes": 2 } ] }, { "id": 1449712, "postDate": "2021-08-05T00:14:03.973Z", "content": "Thanks for letting us know your path and how you got so many improvements. In this comp I also decided to trust LB, and even so it was very hard and tricky to select only two submissions.
", "rawMarkdown": "Thanks for letting us know your path and how you got so many improvements. In this comp I also decided to trust LB, and even so it was very hard and tricky to select only two submissions." }, { "id": 1443040, "postDate": "2021-08-04T01:30:53.647Z", "content": "Congrats! Thank you for your amazing writeup.
\nI have a question.
\n\nI then used the average of two 0.448 models to label the data again and used that data to train a Deberta-large; Deberta ends up with 0.448 on public and 0.449 on private
\n
In this part, are you doing relabel for train data or external data?
", "rawMarkdown": "Congrats! Thank you for your amazing writeup.\nI have a question.\n\n>I then used the average of two 0.448 models to label the data again and used that data to train a Deberta-large; Deberta ends up with 0.448 on public and 0.449 on private\n\nIn this part, are you doing relabel for train data or external data?\n", "replies": [ { "id": 1443161, "postDate": "2021-08-04T01:57:25.720Z", "content": "Thank you for your question. In the part you quoted I relabeled external data only, however, I did make an attempt to relabel train data in the last day and it did boost my private to 0.446. I've updated the details about 0.446 private in the original post. I did not have the time to pursue this path but I think with a proper pipeline single model performance could be further boosted.
", "rawMarkdown": "Thank you for your question. In the part you quoted I relabeled external data only, however, I did make an attempt to relabel train data in the last day and it did boost my private to 0.446. I've updated the details about 0.446 private in the original post. I did not have the time to pursue this path but I think with a proper pipeline single model performance could be further boosted.", "votes": 1 } ] }, { "id": 1428669, "postDate": "2021-08-03T09:41:55.947Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:46:24Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6925863, "solution_url": "/competitions/commonlitreadabilityprize/writeups/hxcc-3rd-place-solution-0-447-private-3-models-sim" }