{ "id": 258363, "title": "40th Place solution: How we improved our rank from 1100th three days ago", "content": "

First of all @shahules and I would like to start by heartily thanking the following people: @rhtsingh, @maunish, @jcesquiveld, @andretugan, @jacob34, @kurupical, @cdeotte and @markwijkhuizen. Without their great work and discussion it would not have been possible for us to have made it here. Additionally I would like to personally thank @radek1 for his wonderful book Meta learning. This book inspired me with a lot of ideas in general as well as I used them in this competition. I highly recommend this book to all fellow data scientists. This is my first serious Kaggle competition. I did one for a couple of days five years ago but didn't pursue it. But this time I survived :) Once again a very big hearty thank you to all of you. I personally had never worked with transformers two months ago. Thanks to this competition I am happy I discovered it. We were at 1100th place three days ago, so we are really happy that we made it to 40th.

\n

Getting inspired from @rhtsingh and @maunish I decided to develop everything publicly for the first two months of the competition. Later on @shahules and I met exactly one month ago and we formed a team.

\n

We understood that the key to succeed in this competition were the following:

\n
    \n
  1. Stabilizing the training of the transformer
    2. \n
  2. Leveraging massive external data
    3. \n
  3. A good ensemble of good models
    4. \n
  4. Definitely not to overfit
    5. \n
\n

Stabilizing the training of the transformer

\n

Of all these things the most difficult part was stabilizing the training. Our validation graphs looked like seismographs. Here is a snippet of the graph before and after stabilizing.
\n\"\"

\n

At least with the base model it was ok but with the large model it was just impossible. We also felt very uncomfortable with evaluating every n iterations. Therefore we spent about twenty days focusing on running experiments to stabilize the training. That's when we came across @jcesquiveld's models trained using differential learning rate. The first time when we implemented it felt magical. The training was extremely stable, no more seismographs ;) This helped us in avoiding frequent evaluation and also let us comfortably train our models on the full dataset without validation. Yes the idea of training on full data instead of Kfold CV came from @cdeotte. You can watch the wonderful video posted by @abhishek in his youtube channel where 4 grandmasters share their secrets of success

\n

Design of work

\n

We also realized a good ensemble was key to success and independence is the key to successful ensemble. Therefore we always designed and ran experiments independently and discussed results later on so we don't bias each other as well. In order to facilitate identical experimental conditions and also not to do double work we created a central repository of all our pipeline components adhering to good coding principles.

\n

Choosing the models and not overfitting

\n

As the CV and LB correlation was not great, we had two choices: we could either probe LB and understand more about the test data but we had almost no time/submissions left for that. Our other choice was choosing models not based on their CV scores but by looking at the validation graphs, evaluating them visually if the training and validation loss look stable, and then if the final model was close to the best model (with frequent evaluation) and so on. So this was more of the data scientist instinct that helped us. We never chose a model because the CV score was high.

\n

Model definition

\n

The attention head that were used in almost all the public notebook we came across took the pad tokens into account. Therefore we modified it to exclude the pad tokens in the attention calculation.

\n
class MaskAddedAttentionHead(nn.Module):\n    def __init__(self, input_dim, head_hidden_dim):\n        super(MaskAddedAttentionHead, self).__init__()\n        self.W = nn.Linear(input_dim, head_hidden_dim)\n        self.V = nn.Linear(head_hidden_dim, 1)\n\n    def forward(self, x, attention_mask):\n        attention_scores = self.V(torch.tanh(self.W(x)))\n        attention_scores = attention_scores + attention_mask\n        attention_scores = torch.softmax(attention_scores, dim=1)\n        attentive_x = attention_scores * x\n        attentive_x = attentive_x.sum(axis=1)\n        return attentive_x\n
\n

Once we stabilized the training, replacing RMSE loss with MSE loss gave us a good boost. We didn't dig deeper into the reason but I guess it had something to do with gradient flow.

\n

Leveraging massive external data

\n

One of our star differences was the pretraining of our models with the external data. Thanks to @markwijkhuizen we did not have to scrape Wikipedia. We converted the problem into a text ranking problem and trained the model with the ranking loss as the objective. We understood that Bradley Terry scores were essentially computed by comparing pairs of text excerpts therefore we have to do something similar. As many of the text excerpts come from Wikipedia and Simple Wikipedia we trained a ranker as follows: \"\"
\nOur initial experiments with pretraining did not give us good results so we dropped it. Only on the last day after running out of ideas we remembered abandoning this and thought why not try it again. The reason why transfer learning did not work well in the past for us but worked well now was because our transformer became a stable bed now.

\n

Now when we look back the key to rapid progress in three days comes from the fact that we invested all our effort in preparing a very stable experimental set-up.

\n

Thank you all for all your valuable discussion in the forums and notebooks. Without great work being made public, we couldn't have done this. I really cannot thank @maunish and @rhtsingh enough. This continues to inspire us to develop more and more publicly. Thank you.

\n

Here is our code: https://www.kaggle.com/vigneshbaskaran/commonlit-nn-kit

", "comments": [ { "id": 1441121, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-03T19:40:12.413000", "content": "

Great solution and great climb to 40th place. Stablizing training was indeed key!

\n

We also focused on this and plotted val loss curves. Differential learning rates and schedules, freezing backbones, heads with different losses, and adjusting dropout all helped. Here is a plot showing training with and without dropout.

", "votes": 3, "replies": [ { "id": 1445047, "author_name": "Vignesh Baskaran", "author_url": "", "post_date": "2021-08-04T07:41:37.987000", "content": "

Thank you Chris. We also experimented with dropout and observed similar effect so we also set all the dropouts to 0. We set both the hidden and the attention dropouts to 0. We didn't find this inmtuitive but nevertheless we couldn't investigate the reason.

\n
'roberta_hidden_dropout_prob': 0.0,\n'roberta_attention_probs_dropout_prob': 0.0,\n
", "votes": 2, "replies": [] }, { "id": 1472444, "author_name": "Gary Lai", "author_url": "", "post_date": "2021-08-14T21:40:14.980000", "content": "

Hi @cdeotte , what does heads with different losses and freezing backbones mean? Can you link me to resources / explanations please? Thank you so much!

", "votes": 0, "replies": [] } ] }, { "id": 1429578, "author_name": "Shahules", "author_url": "", "post_date": "2021-08-03T10:47:39.217000", "content": "

Great, It was a pleasure working with you @vigneshbaskaran . A lot happened during these last 30 days, learned a lot, and got one more good friend through kaggle ๐Ÿ’ฏ๐ŸŽ‰

", "votes": 3, "replies": [ { "id": 1431532, "author_name": "Vignesh Baskaran", "author_url": "", "post_date": "2021-08-03T12:35:25.593000", "content": "

It was the same working with you @shahules! I am extremely happy that I got a great friend through Kaggle! I learnt so much from you. Thank you Shahul!

", "votes": 1, "replies": [] } ] }, { "id": 1456312, "author_name": "Nabeel AlMusawy", "author_url": "", "post_date": "2021-08-06T21:23:39.357000", "content": "

Great work!

", "votes": 1, "replies": [] }, { "id": 1455592, "author_name": "Gary Lai", "author_url": "", "post_date": "2021-08-06T16:06:00.787000", "content": "

Hi Vignesh, thanks so much for this very thoughtful and inspiring post. I have a question about the ranking loss? How exactly does it work? What is the \"comparator\", \"difficulty ranking\" and \"readability ground truth\" (I couldn't seem to find it in your code)?

", "votes": 1, "replies": [ { "id": 1464225, "author_name": "Vignesh Baskaran", "author_url": "", "post_date": "2021-08-10T13:50:30.433000", "content": "

Hi Gary,
\nI am very sorry for the late response. I have released a small prototype of the comparator here: https://www.kaggle.com/vigneshbaskaran/commonlit-text-comparator-rmse-vs-ranking-loss

\n

Essentially comparator is a model that takes in a pair of inputs and identifies which one is more difficult to read in comparison to the other. Readability ground truth is the target of the text available from the training data. Please go through the notebook and please don't hesitate to write to me if you have follow up questions. Thank you

", "votes": 1, "replies": [] } ] }, { "id": 1450331, "author_name": "Shivu11", "author_url": "", "post_date": "2021-08-05T04:53:33.743000", "content": "

If you could only tell about how to survive till the end of the competition ? This is my first competition ,I really want to make it till the end.

", "votes": 1, "replies": [ { "id": 1454970, "author_name": "Vignesh Baskaran", "author_url": "", "post_date": "2021-08-06T12:01:48.550000", "content": "

Hi @shivu11
\nIt was the most difficult part of this competition. I made it very clear to myself it is ok if I don't make progress but its not ok to drop out. Here is a blog that inspired me to keep going: https://typesense.org/blog/the-unreasonable-effectiveness-of-just-showing-up-everyday/ . Hope this helps you as well! Good luck with future competitions

", "votes": 5, "replies": [] } ] }, { "id": 1448654, "author_name": "Amol Ambkar", "author_url": "", "post_date": "2021-08-04T18:15:27.107000", "content": "

Great oneโ€ฆ

", "votes": 1, "replies": [] }, { "id": 1441480, "author_name": "Shubham Singh", "author_url": "", "post_date": "2021-08-03T20:26:41.513000", "content": "

It is a great detailed explanation and thanks for the resources you mentioned .

", "votes": 1, "replies": [] }, { "id": 1432616, "author_name": "over watch", "author_url": "", "post_date": "2021-08-03T13:26:33.930000", "content": "

Nice Work, and congrats on your big improvement in the last few days of the competition!

\n

Quick question about differential learning rate- is it the same as layer-wise learning decay? I also read that you were training on the entire dataset. Can you elaborate on the advantages of doing this compared to k-fold? Thanks a lot!

", "votes": 1, "replies": [ { "id": 1445123, "author_name": "Vignesh Baskaran", "author_url": "", "post_date": "2021-08-04T07:53:33.383000", "content": "

Thanks @overwatcher. Yes its the same. I just noticed that my private score has improved by +0.003 just by training on full dataset. We know this is a great boost in this competition. So all credits goes to @cdeotte

", "votes": 1, "replies": [] } ] }, { "id": 1432018, "author_name": "Dave E", "author_url": "", "post_date": "2021-08-03T12:58:18.543000", "content": "

Thanks for sharing, great stuff, all really clear and logical and sensible. Read your discussion posts @vigneshbaskaran and happy to see you were able to achieve a great progression.

", "votes": 1, "replies": [ { "id": 1445125, "author_name": "Vignesh Baskaran", "author_url": "", "post_date": "2021-08-04T07:54:01.397000", "content": "

Thank you @davidedwards1

", "votes": 0, "replies": [] } ] }, { "id": 1445206, "author_name": "jcesquivel", "author_url": "", "post_date": "2021-08-04T08:07:12.563000", "content": "

Congratulations @vigneshbaskaran for your awesome work. I'm glad to have been helpful.

", "votes": 2, "replies": [ { "id": 1449049, "author_name": "Vignesh Baskaran", "author_url": "", "post_date": "2021-08-04T19:44:12.833000", "content": "

Thank you very much @jcesquiveld

", "votes": 0, "replies": [] } ] }, { "id": 1443151, "author_name": "Anshu Kumar Singh", "author_url": "", "post_date": "2021-08-04T01:55:46.403000", "content": "

Thank you for this nice explanation. Very much informative

", "votes": 2, "replies": [] }, { "id": 1464738, "author_name": "DIVYA MAHESHWARI1101", "author_url": "", "post_date": "2021-08-10T17:25:41.277000", "content": "

Really Nice work!

", "votes": 0, "replies": [] }, { "id": 1443522, "author_name": "spy1234567", "author_url": "", "post_date": "2021-08-04T03:16:05.290000", "content": "

Hi, Baskaran, Congrats!
\nCould you share from with notebook you have learned the differential learning rate? Thank you!

", "votes": 0, "replies": [ { "id": 1443790, "author_name": "Saiyan Warrior", "author_url": "", "post_date": "2021-08-04T04:10:15.867000", "content": "

Hi @spy1234567 this is how I used differential learning rate
\nhttps://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch?scriptVersionId=67087445&cellId=14
\nHope this help๐Ÿ˜Š

", "votes": 0, "replies": [] }, { "id": 1445105, "author_name": "Vignesh Baskaran", "author_url": "", "post_date": "2021-08-04T07:51:08.760000", "content": "

Hi @spy1234567
\nWe learnt about this technique from @jcesquiveld's discussion post and we implemented it ourselves. You may find our code here: https://www.kaggle.com/vigneshbaskaran/commonlit-nn-kit

\n

Under the function get_optimizer_parameters with group_mode=be_wd you may find our implementation.

", "votes": 1, "replies": [] }, { "id": 1446457, "author_name": "spy1234567", "author_url": "", "post_date": "2021-08-04T12:19:04.113000", "content": "

Hi, @saiyanwarrior and @vigneshbaskaran , thanks a lot for your kind help!

", "votes": 1, "replies": [] } ] }, { "id": 1440356, "author_name": "HRUSHIKESH SHELAR", "author_url": "", "post_date": "2021-08-03T18:11:29.363000", "content": "

Nice work!!!

", "votes": 0, "replies": [] }, { "id": 1447289, "author_name": "ABDELAZIZ Korba", "author_url": "", "post_date": "2021-08-04T14:36:01.587000", "content": "

Thank you for this nice explanation.

", "votes": 1, "replies": [] }, { "id": 1455418, "author_name": "Shivu11", "author_url": "", "post_date": "2021-08-06T15:02:20.167000", "content": "

Thank you so much @vigneshbaskaran

", "votes": 0, "replies": [] } ], "comment_count": 27, "messages": [ { "id": 1428215, "postDate": "2021-08-03T09:13:57.417Z", "content": "

First of all @shahules and I would like to start by heartily thanking the following people: @rhtsingh, @maunish, @jcesquiveld, @andretugan, @jacob34, @kurupical, @cdeotte and @markwijkhuizen. Without their great work and discussion it would not have been possible for us to have made it here. Additionally I would like to personally thank @radek1 for his wonderful book Meta learning. This book inspired me with a lot of ideas in general as well as I used them in this competition. I highly recommend this book to all fellow data scientists. This is my first serious Kaggle competition. I did one for a couple of days five years ago but didn't pursue it. But this time I survived :) Once again a very big hearty thank you to all of you. I personally had never worked with transformers two months ago. Thanks to this competition I am happy I discovered it. We were at 1100th place three days ago, so we are really happy that we made it to 40th.

\n

Getting inspired from @rhtsingh and @maunish I decided to develop everything publicly for the first two months of the competition. Later on @shahules and I met exactly one month ago and we formed a team.

\n

We understood that the key to succeed in this competition were the following:

\n
    \n
  1. Stabilizing the training of the transformer
    2. \n
  2. Leveraging massive external data
    3. \n
  3. A good ensemble of good models
    4. \n
  4. Definitely not to overfit
    5. \n
\n

Stabilizing the training of the transformer

\n

Of all these things the most difficult part was stabilizing the training. Our validation graphs looked like seismographs. Here is a snippet of the graph before and after stabilizing.
\n\"\"

\n

At least with the base model it was ok but with the large model it was just impossible. We also felt very uncomfortable with evaluating every n iterations. Therefore we spent about twenty days focusing on running experiments to stabilize the training. That's when we came across @jcesquiveld's models trained using differential learning rate. The first time when we implemented it felt magical. The training was extremely stable, no more seismographs ;) This helped us in avoiding frequent evaluation and also let us comfortably train our models on the full dataset without validation. Yes the idea of training on full data instead of Kfold CV came from @cdeotte. You can watch the wonderful video posted by @abhishek in his youtube channel where 4 grandmasters share their secrets of success

\n

Design of work

\n

We also realized a good ensemble was key to success and independence is the key to successful ensemble. Therefore we always designed and ran experiments independently and discussed results later on so we don't bias each other as well. In order to facilitate identical experimental conditions and also not to do double work we created a central repository of all our pipeline components adhering to good coding principles.

\n

Choosing the models and not overfitting

\n

As the CV and LB correlation was not great, we had two choices: we could either probe LB and understand more about the test data but we had almost no time/submissions left for that. Our other choice was choosing models not based on their CV scores but by looking at the validation graphs, evaluating them visually if the training and validation loss look stable, and then if the final model was close to the best model (with frequent evaluation) and so on. So this was more of the data scientist instinct that helped us. We never chose a model because the CV score was high.

\n

Model definition

\n

The attention head that were used in almost all the public notebook we came across took the pad tokens into account. Therefore we modified it to exclude the pad tokens in the attention calculation.

\n
class MaskAddedAttentionHead(nn.Module):\n    def __init__(self, input_dim, head_hidden_dim):\n        super(MaskAddedAttentionHead, self).__init__()\n        self.W = nn.Linear(input_dim, head_hidden_dim)\n        self.V = nn.Linear(head_hidden_dim, 1)\n\n    def forward(self, x, attention_mask):\n        attention_scores = self.V(torch.tanh(self.W(x)))\n        attention_scores = attention_scores + attention_mask\n        attention_scores = torch.softmax(attention_scores, dim=1)\n        attentive_x = attention_scores * x\n        attentive_x = attentive_x.sum(axis=1)\n        return attentive_x\n
\n

Once we stabilized the training, replacing RMSE loss with MSE loss gave us a good boost. We didn't dig deeper into the reason but I guess it had something to do with gradient flow.

\n

Leveraging massive external data

\n

One of our star differences was the pretraining of our models with the external data. Thanks to @markwijkhuizen we did not have to scrape Wikipedia. We converted the problem into a text ranking problem and trained the model with the ranking loss as the objective. We understood that Bradley Terry scores were essentially computed by comparing pairs of text excerpts therefore we have to do something similar. As many of the text excerpts come from Wikipedia and Simple Wikipedia we trained a ranker as follows: \"\"
\nOur initial experiments with pretraining did not give us good results so we dropped it. Only on the last day after running out of ideas we remembered abandoning this and thought why not try it again. The reason why transfer learning did not work well in the past for us but worked well now was because our transformer became a stable bed now.

\n

Now when we look back the key to rapid progress in three days comes from the fact that we invested all our effort in preparing a very stable experimental set-up.

\n

Thank you all for all your valuable discussion in the forums and notebooks. Without great work being made public, we couldn't have done this. I really cannot thank @maunish and @rhtsingh enough. This continues to inspire us to develop more and more publicly. Thank you.

\n

Here is our code: https://www.kaggle.com/vigneshbaskaran/commonlit-nn-kit

", "rawMarkdown": "First of all @shahules and I would like to start by heartily thanking the following people: @rhtsingh, @maunish, @jcesquiveld, @andretugan, @jacob34, @kurupical, @cdeotte and @markwijkhuizen. Without their great work and discussion it would not have been possible for us to have made it here. Additionally I would like to personally thank @radek1 for his wonderful book **Meta learning**. This book inspired me with a lot of ideas in general as well as I used them in this competition. I highly recommend this book to all fellow data scientists. This is my first serious Kaggle competition. I did one for a couple of days five years ago but didn't pursue it. But this time I survived :) Once again a very big hearty thank you to all of you. I personally had never worked with transformers two months ago. Thanks to this competition I am happy I discovered it. We were at 1100th place three days ago, so we are really happy that we made it to 40th.\n\nGetting inspired from @rhtsingh and @maunish I decided to develop everything publicly for the first two months of the competition. Later on @shahules and I met exactly one month ago and we formed a team. \n\nWe understood that the key to succeed in this competition were the following:\n1. Stabilizing the training of the transformer\n2. Leveraging massive external data\n3. A good ensemble of good models\n4. Definitely not to overfit\n\n### Stabilizing the training of the transformer\nOf all these things the most difficult part was stabilizing the training. Our validation graphs looked like seismographs. Here is a snippet of the graph before and after stabilizing.\n![](https://i.ibb.co/x8q5NGy/image-3.png)\n\nAt least with the base model it was ok but with the large model it was just impossible. We also felt very uncomfortable with evaluating every n iterations. Therefore we spent about twenty days focusing on running experiments to stabilize the training. That's when we came across @jcesquiveld's models trained using **differential learning rate**. The first time when we implemented it felt magical. The training was extremely stable, no more seismographs ;) This helped us in avoiding frequent evaluation and also let us comfortably train our models on the full dataset without validation. Yes the idea of training on full data instead of Kfold CV came from @cdeotte. You can watch the wonderful video posted by @abhishek in his youtube channel where 4 grandmasters share their secrets of success\n\n\n### Design of work\nWe also realized a good ensemble was key to success and independence is the key to successful ensemble. Therefore we always designed and ran experiments independently and discussed results later on so we don't bias each other as well. In order to facilitate identical experimental conditions and also not to do double work we created a central repository of all our pipeline components adhering to good coding principles. \n\n### Choosing the models and not overfitting\n As the CV and LB correlation was not great, we had two choices: we could either probe LB and understand more about the test data but we had almost no time/submissions left for that. Our other choice was choosing models not based on their CV scores but by looking at the validation graphs, evaluating them visually if the training and validation loss look stable, and then if the final model was close to the best model (with frequent evaluation) and so on. So this was more of the data scientist instinct that helped us. We never chose a model because the CV score was high. \n\n### Model definition\nThe attention head that were used in almost all the public notebook we came across took the pad tokens into account. Therefore we modified it to exclude the pad tokens in the attention calculation. \n\n``` python\nclass MaskAddedAttentionHead(nn.Module):\n def __init__(self, input_dim, head_hidden_dim):\n super(MaskAddedAttentionHead, self).__init__()\n self.W = nn.Linear(input_dim, head_hidden_dim)\n self.V = nn.Linear(head_hidden_dim, 1)\n \n def forward(self, x, attention_mask):\n attention_scores = self.V(torch.tanh(self.W(x)))\n attention_scores = attention_scores + attention_mask\n attention_scores = torch.softmax(attention_scores, dim=1)\n attentive_x = attention_scores * x\n attentive_x = attentive_x.sum(axis=1)\n return attentive_x\n```\n\nOnce we stabilized the training, replacing RMSE loss with MSE loss gave us a good boost. We didn't dig deeper into the reason but I guess it had something to do with gradient flow. \n\n### Leveraging massive external data\nOne of our star differences was the pretraining of our models with the external data. Thanks to @markwijkhuizen we did not have to scrape Wikipedia. We converted the problem into a text ranking problem and trained the model with the ranking loss as the objective. We understood that Bradley Terry scores were essentially computed by comparing pairs of text excerpts therefore we have to do something similar. As many of the text excerpts come from Wikipedia and Simple Wikipedia we trained a ranker as follows: ![](https://i.ibb.co/HNPN1vG/text-comparator.png)\nOur initial experiments with pretraining did not give us good results so we dropped it. Only on the last day after running out of ideas we remembered abandoning this and thought why not try it again. The reason why transfer learning did not work well in the past for us but worked well now was because our transformer became a stable bed now. \n\nNow when we look back the key to rapid progress in three days comes from the fact that we invested all our effort in preparing a very stable experimental set-up. \n\nThank you all for all your valuable discussion in the forums and notebooks. Without great work being made public, we couldn't have done this. I really cannot thank @maunish and @rhtsingh enough. This continues to inspire us to develop more and more publicly. Thank you.\n\nHere is our code: https://www.kaggle.com/vigneshbaskaran/commonlit-nn-kit", "votes": 113 }, { "id": 1441121, "postDate": "2021-08-03T19:40:12.413Z", "content": "

Great solution and great climb to 40th place. Stablizing training was indeed key!

\n

We also focused on this and plotted val loss curves. Differential learning rates and schedules, freezing backbones, heads with different losses, and adjusting dropout all helped. Here is a plot showing training with and without dropout.

", "rawMarkdown": "Great solution and great climb to 40th place. Stablizing training was indeed key! \n\nWe also focused on this and plotted val loss curves. Differential learning rates and schedules, freezing backbones, heads with different losses, and adjusting dropout all helped. [Here][1] is a plot showing training with and without dropout.\n\n[1]: https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260729", "votes": 3, "replies": [ { "id": 1445047, "postDate": "2021-08-04T07:41:37.987Z", "content": "

Thank you Chris. We also experimented with dropout and observed similar effect so we also set all the dropouts to 0. We set both the hidden and the attention dropouts to 0. We didn't find this inmtuitive but nevertheless we couldn't investigate the reason.

\n
'roberta_hidden_dropout_prob': 0.0,\n'roberta_attention_probs_dropout_prob': 0.0,\n
", "rawMarkdown": "Thank you Chris. We also experimented with dropout and observed similar effect so we also set all the dropouts to 0. We set both the hidden and the attention dropouts to 0. We didn't find this inmtuitive but nevertheless we couldn't investigate the reason. \n``` python\n'roberta_hidden_dropout_prob': 0.0,\n'roberta_attention_probs_dropout_prob': 0.0,\n```\n", "votes": 2 }, { "id": 1472444, "postDate": "2021-08-14T21:40:14.980Z", "content": "

Hi @cdeotte , what does heads with different losses and freezing backbones mean? Can you link me to resources / explanations please? Thank you so much!

", "rawMarkdown": "Hi @cdeotte , what does heads with different losses and freezing backbones mean? Can you link me to resources / explanations please? Thank you so much!" } ] }, { "id": 1429578, "postDate": "2021-08-03T10:47:39.217Z", "content": "

Great, It was a pleasure working with you @vigneshbaskaran . A lot happened during these last 30 days, learned a lot, and got one more good friend through kaggle ๐Ÿ’ฏ๐ŸŽ‰

", "rawMarkdown": "Great, It was a pleasure working with you @vigneshbaskaran . A lot happened during these last 30 days, learned a lot, and got one more good friend through kaggle ๐Ÿ’ฏ๐ŸŽ‰", "votes": 3, "replies": [ { "id": 1431532, "postDate": "2021-08-03T12:35:25.593Z", "content": "

It was the same working with you @shahules! I am extremely happy that I got a great friend through Kaggle! I learnt so much from you. Thank you Shahul!

", "rawMarkdown": "It was the same working with you @shahules! I am extremely happy that I got a great friend through Kaggle! I learnt so much from you. Thank you Shahul! ", "votes": 1 } ] }, { "id": 1456312, "postDate": "2021-08-06T21:23:39.357Z", "content": "

Great work!

", "rawMarkdown": "Great work!", "votes": 1 }, { "id": 1455592, "postDate": "2021-08-06T16:06:00.787Z", "content": "

Hi Vignesh, thanks so much for this very thoughtful and inspiring post. I have a question about the ranking loss? How exactly does it work? What is the \"comparator\", \"difficulty ranking\" and \"readability ground truth\" (I couldn't seem to find it in your code)?

", "rawMarkdown": "Hi Vignesh, thanks so much for this very thoughtful and inspiring post. I have a question about the ranking loss? How exactly does it work? What is the \"comparator\", \"difficulty ranking\" and \"readability ground truth\" (I couldn't seem to find it in your code)?", "votes": 1, "replies": [ { "id": 1464225, "postDate": "2021-08-10T13:50:30.433Z", "content": "

Hi Gary,
\nI am very sorry for the late response. I have released a small prototype of the comparator here: https://www.kaggle.com/vigneshbaskaran/commonlit-text-comparator-rmse-vs-ranking-loss

\n

Essentially comparator is a model that takes in a pair of inputs and identifies which one is more difficult to read in comparison to the other. Readability ground truth is the target of the text available from the training data. Please go through the notebook and please don't hesitate to write to me if you have follow up questions. Thank you

", "rawMarkdown": "Hi Gary,\nI am very sorry for the late response. I have released a small prototype of the comparator here: https://www.kaggle.com/vigneshbaskaran/commonlit-text-comparator-rmse-vs-ranking-loss\n\nEssentially **comparator** is a model that takes in a pair of inputs and identifies which one is more difficult to read in comparison to the other. **Readability ground truth** is the target of the text available from the training data. Please go through the notebook and please don't hesitate to write to me if you have follow up questions. Thank you", "votes": 1 } ] }, { "id": 1450331, "postDate": "2021-08-05T04:53:33.743Z", "content": "

If you could only tell about how to survive till the end of the competition ? This is my first competition ,I really want to make it till the end.

", "rawMarkdown": "If you could only tell about how to survive till the end of the competition ? This is my first competition ,I really want to make it till the end.", "votes": 1, "replies": [ { "id": 1454970, "postDate": "2021-08-06T12:01:48.550Z", "content": "

Hi @shivu11
\nIt was the most difficult part of this competition. I made it very clear to myself it is ok if I don't make progress but its not ok to drop out. Here is a blog that inspired me to keep going: https://typesense.org/blog/the-unreasonable-effectiveness-of-just-showing-up-everyday/ . Hope this helps you as well! Good luck with future competitions

", "rawMarkdown": "Hi @shivu11 \nIt was the most difficult part of this competition. I made it very clear to myself it is ok if I don't make progress but its not ok to drop out. Here is a blog that inspired me to keep going: https://typesense.org/blog/the-unreasonable-effectiveness-of-just-showing-up-everyday/ . Hope this helps you as well! Good luck with future competitions", "votes": 5 } ] }, { "id": 1448654, "postDate": "2021-08-04T18:15:27.107Z", "content": "

Great oneโ€ฆ

", "rawMarkdown": "Great one...", "votes": 1 }, { "id": 1441480, "postDate": "2021-08-03T20:26:41.513Z", "content": "

It is a great detailed explanation and thanks for the resources you mentioned .

", "rawMarkdown": "It is a great detailed explanation and thanks for the resources you mentioned . ", "votes": 1 }, { "id": 1432616, "postDate": "2021-08-03T13:26:33.930Z", "content": "

Nice Work, and congrats on your big improvement in the last few days of the competition!

\n

Quick question about differential learning rate- is it the same as layer-wise learning decay? I also read that you were training on the entire dataset. Can you elaborate on the advantages of doing this compared to k-fold? Thanks a lot!

", "rawMarkdown": "Nice Work, and congrats on your big improvement in the last few days of the competition!\n\nQuick question about differential learning rate- is it the same as layer-wise learning decay? I also read that you were training on the entire dataset. Can you elaborate on the advantages of doing this compared to k-fold? Thanks a lot!", "votes": 1, "replies": [ { "id": 1445123, "postDate": "2021-08-04T07:53:33.383Z", "content": "

Thanks @overwatcher. Yes its the same. I just noticed that my private score has improved by +0.003 just by training on full dataset. We know this is a great boost in this competition. So all credits goes to @cdeotte

", "rawMarkdown": "Thanks @overwatcher. Yes its the same. I just noticed that my private score has improved by +0.003 just by training on full dataset. We know this is a great boost in this competition. So all credits goes to @cdeotte ", "votes": 1 } ] }, { "id": 1432018, "postDate": "2021-08-03T12:58:18.543Z", "content": "

Thanks for sharing, great stuff, all really clear and logical and sensible. Read your discussion posts @vigneshbaskaran and happy to see you were able to achieve a great progression.

", "rawMarkdown": "Thanks for sharing, great stuff, all really clear and logical and sensible. Read your discussion posts @vigneshbaskaran and happy to see you were able to achieve a great progression.", "votes": 1, "replies": [ { "id": 1445125, "postDate": "2021-08-04T07:54:01.397Z", "content": "

Thank you @davidedwards1

", "rawMarkdown": "Thank you @davidedwards1 " } ] }, { "id": 1445206, "postDate": "2021-08-04T08:07:12.563Z", "content": "

Congratulations @vigneshbaskaran for your awesome work. I'm glad to have been helpful.

", "rawMarkdown": "Congratulations [@vigneshbaskaran](https://www.kaggle.com/vigneshbaskaran) for your awesome work. I'm glad to have been helpful.", "votes": 2, "replies": [ { "id": 1449049, "postDate": "2021-08-04T19:44:12.833Z", "content": "

Thank you very much @jcesquiveld

", "rawMarkdown": "Thank you very much @jcesquiveld " } ] }, { "id": 1443151, "postDate": "2021-08-04T01:55:46.403Z", "content": "

Thank you for this nice explanation. Very much informative

", "rawMarkdown": "Thank you for this nice explanation. Very much informative", "votes": 2 }, { "id": 1464738, "postDate": "2021-08-10T17:25:41.277Z", "content": "

Really Nice work!

", "rawMarkdown": "Really Nice work!" }, { "id": 1443522, "postDate": "2021-08-04T03:16:05.290Z", "content": "

Hi, Baskaran, Congrats!
\nCould you share from with notebook you have learned the differential learning rate? Thank you!

", "rawMarkdown": "Hi, Baskaran, Congrats!\nCould you share from with notebook you have learned the differential learning rate? Thank you!", "replies": [ { "id": 1443790, "postDate": "2021-08-04T04:10:15.867Z", "content": "

Hi @spy1234567 this is how I used differential learning rate
\nhttps://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch?scriptVersionId=67087445&cellId=14
\nHope this help๐Ÿ˜Š

", "rawMarkdown": "Hi @spy1234567 this is how I used differential learning rate\nhttps://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch?scriptVersionId=67087445&cellId=14\nHope this help๐Ÿ˜Š" }, { "id": 1445105, "postDate": "2021-08-04T07:51:08.760Z", "content": "

Hi @spy1234567
\nWe learnt about this technique from @jcesquiveld's discussion post and we implemented it ourselves. You may find our code here: https://www.kaggle.com/vigneshbaskaran/commonlit-nn-kit

\n

Under the function get_optimizer_parameters with group_mode=be_wd you may find our implementation.

", "rawMarkdown": "Hi @spy1234567\nWe learnt about this technique from @jcesquiveld's discussion post and we implemented it ourselves. You may find our code here: https://www.kaggle.com/vigneshbaskaran/commonlit-nn-kit\n\nUnder the function `get_optimizer_parameters` with `group_mode`=`be_wd` you may find our implementation.", "votes": 1 }, { "id": 1446457, "postDate": "2021-08-04T12:19:04.113Z", "content": "

Hi, @saiyanwarrior and @vigneshbaskaran , thanks a lot for your kind help!

", "rawMarkdown": "Hi, @saiyanwarrior and @vigneshbaskaran , thanks a lot for your kind help!", "votes": 1 } ] }, { "id": 1440356, "postDate": "2021-08-03T18:11:29.363Z", "content": "

Nice work!!!

", "rawMarkdown": "Nice work!!!" }, { "id": 1447289, "postDate": "2021-08-04T14:36:01.587Z", "content": "

Thank you for this nice explanation.

", "rawMarkdown": "Thank you for this nice explanation.", "votes": 1 }, { "id": 1455418, "postDate": "2021-08-06T15:02:20.167Z", "content": "

Thank you so much @vigneshbaskaran

", "rawMarkdown": "Thank you so much @vigneshbaskaran " } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:47:04Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6760665, "solution_url": "/competitions/commonlitreadabilityprize/writeups/southies-40th-place-solution-how-we-improved-our-r" }