First of all, I would like to say thank you very much to my wonderful teammates, @horsek, and @lftuwujie, I learned a lot from them during this competition. Next, thank you to all competitors, the competition host, and Kaggle for competing and organizing this amazing competition. Besides, congratulation to all teams in the gold medal zone.
\nAt the beginning of this competition, I told @horsek that let's treat it as our chance to learn about NLP because I didn't have knowledge of it and it has been my first NLP competition.
\nIn this write-up, I would like to share not only my models but also my journey in this competition, from starting points to forming ideas under a newbie's viewpoint like me.
\nI started my journey with this wonderful notebook (https://www.kaggle.com/rhtsingh/commonlit-readability-prize-roberta-torch-fit), please give him/her an upvote.
\nHowever, I didn't fork it but read and re-wrote my own notebook. This helped me to debug, and forming some ideas to further customize and optimize the pipeline.
\nAfter this step, my best single model is the RoBERTa-large and scored 0.468 on the public LB and 0.469 on the private LB, my inference notebook is here
\nThe setting of this model is,
\nIn this competition (and also in all of my competitions), I studied related literature a lot. This step helped me to understand the concept of the competition and find new ideas. The first idea I got is the Quadratic-Weighted-Kappa loss, the second is the custom Bradley-Terry loss.
\nThe idea of this loss came from this paper. The main idea of this loss is binning the excerpt targets into categories, and train the model to classify them into these categories. In my opinion, this step is equivalent to label-smoothing, because it gathers excerpts with similar targets into a bin. The QWK loss is not a must, you can implement it with Cross-entropy loss. This is the implementation of this QWK loss,
\nclass QuadraticWeightedKappaLoss(nn.Module):\n def __init__(self, num_cat = 7, device = 'cpu'):\n super(QuadraticWeightedKappaLoss, self).__init__()\n self.num_cat = num_cat\n cats = torch.arange(num_cat).to(device)\n self.weights = (cats.view(-1,1) - cats.view(1,-1)).pow(2) / (num_cat - 1)**2\n\n def _confusion_matrix(self, pred_cat, true_cat):\n confusion_matrix = torch.zeros((self.num_cat, self.num_cat)).to(pred_cat.device)\n for t, p in zip(true_cat.view(-1), pred_cat.view(-1)):\n confusion_matrix[t.long(), p.long()] += 1\n return confusion_matrix\n\n def forward(self, pred_cat, true_cat):\n # Confusion matrix\n O = self._confusion_matrix(pred_cat, true_cat)\n\n # Count elements in each category\n true_hist = torch.bincount(true_cat, minlength = self.num_cat)\n pred_hist = torch.bincount(pred_cat, minlength = self.num_cat)\n\n # Expected values\n E = torch.outer(true_hist, pred_hist)\n\n # Normlization\n O = O / torch.sum(O)\n E = E / torch.sum(E)\n\n # Weighted Kappa\n numerator = torch.sum(self.weights * O)\n denominator = torch.sum(self.weights * E)\n\n return numerator / denominator\nMy best model after this step is again a RoBERTa-large with the following setting,
\nIt scored 0.466/0.466 on the public/private LB.
\nThis is so far the most effective trick and my most contribution to my team. The idea came from the observation about the benchmark excerpt which its target and standard error are both 0. As I studied the information from the host, each excerpt in the corpus is compared to some others. Then, by using the Bradley-Terry method, their prior targets are estimated. Finally, the prior targets of all excerpts are subtracted to the target a chosen benchmark excerpt to obtain the data we have.
\nFrom this observation, I designed my models accordingly. The model architecture is here (sorry that I couldn't insert an image to a post, so I uploaded my model architecture in another notebook).
\nThe implementation of the custom BT loss is as follow,
\nclass BradleyTerryLoss(nn.Module):\n def __init__(self):\n super(BradleyTerryLoss, self).__init__()\n\n def forward(self, pred_mean, true_mean):\n batch_size = len(pred_mean)\n true_comparison = true_mean.view(-1,1) - true_mean.view(1,-1)\n pred_comparison = pred_mean.view(-1,1) - pred_mean.view(1,-1)\n return torch.log(1 + torch.tril(torch.exp(-true_comparison * pred_comparison))).sum() / (batch_size * (batch_size - 1) / 2)\nWith this structure, the predicted target of the benchmark excerpt is always 0, which is consistent with the truth. This loss also allows models to compare the targets to each other. We applied this custom loss with many models of my teams.
\nThis custom loss gave me the best single model so far with 0.461/0.462 on the public and private LB.
\nMy best model after this step is again a RoBERTa-large with the following setting,
Using this setting, I train my models with different types of backbone. The following summarizes my best models according to public LB with each backbone (public LB/private LB),
\nFor each backbone, I sometimes changed the setting (evaluate less frequently, omit the QWK loss, training with another fold splitting, etc.) to avoid overfitting.
\nFinally, I simply took the equal average of all of my 13 models, it gave me 0.453/0.456 on the public/private LB. Ensemble my average model with my teammates' models gave us 0.448/0.452 on the public/private LB.
\nAlthough the result is not as good as I expected, I still enjoyed the competition a lot. From nearly 0 in NLP, after 3 months, I learned a lot from my teammates and fellow competitors. I hope to meet you in future competitions!
\n@shinomoriaoshi , I was studying this competition with a more specific focus on interesting loss engg strategies. Of course your solution tops my list. QWK is very apt and BT seems to be even more apt. My question is on the BT loss that you used…which is something of the form of log(1+e to power yhat . pred) for every pair in the batch. But ideally in the formula we should have restricted the prediction and groundtruths to +1 and -1 only for win and loss respectively. But I think you are using the BT scores directly. Is that correct or I am misunderstanding something? and somehow that too seems to work…
", "votes": 1, "replies": [] }, { "id": 1433007, "author_name": "ABCbum", "author_url": "", "post_date": "2021-08-03T13:40:37.380000", "content": "Congrats @shinomoriaoshi !! Few weeks ago I considered merging with your team but my ranking was not great and you guys just boosted up to the top so I decided not to since my approach requires a bit more submissions at the time. Could have been a great opportunity to learn from you guys!
", "votes": 1, "replies": [ { "id": 1433465, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2021-08-03T14:01:45.340000", "content": "Thank you for your kind words. Haha, so now we should be the ones who regret about that. Anyway, congratulation with your solo gold. I look forward to working with you in future competitions.
", "votes": 1, "replies": [] } ] }, { "id": 1428643, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-03T09:40:46.007000", "content": "Nice…I think this is the first solution post that talks of binning and loss-strategy. I wanted to explore it but couldn't get much of a chance. I wrote some code based on my own logic but scrapped it somewhere mid-way(should have persisted). I strongly felt that outliers skewed MSE and entropy related losses would give much better results…Anyway, good to read about QWK loss. As you mentioned, I guess the trick is to read more related literature…
", "votes": 1, "replies": [ { "id": 1428886, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2021-08-03T10:12:20.187000", "content": "Thank you, I am currently working in an academic environment, so I guess reading literature as my first step is just my habit, haha.
", "votes": 1, "replies": [] }, { "id": 1429993, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-03T11:11:27.530000", "content": "I need to inculcate that habit for sure :)
\nW.r.t HAN, I was finally able to get it to a .473 score - in case you are interested in the model - https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257674
", "votes": 1, "replies": [] }, { "id": 1430534, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2021-08-03T11:45:01.883000", "content": "Thank you very much. I couldn't make HAN model break 0.50x. I will study your solution for sure.
", "votes": 1, "replies": [] } ] }, { "id": 1446336, "author_name": "hosuke", "author_url": "", "post_date": "2021-08-04T11:55:26.483000", "content": "@shinomoriaoshi Thank you so much for inviting me to this exciting competition! I had really great and meaningful days with you and @lftuwujie.
\nAt first I joined this competition as a learning experience, but as our ranking improved, I became more and more serious about it. haha
\nLet's win the gold medal together next time!
Haha, after seeing the improvement, we commit to this competition more seriously without realizing it. Let's get a gold next time!
", "votes": 1, "replies": [] } ] }, { "id": 1431232, "author_name": "Jie Wu", "author_url": "", "post_date": "2021-08-03T12:17:11.903000", "content": "@shinomoriaoshi very good summary, and it's a great journey to work with both you and @horsek. I also learnt quite a lot from you too. Looking forward to working together again.
", "votes": 2, "replies": [ { "id": 1432314, "author_name": "Minh Tri Phan", "author_url": "", "post_date": "2021-08-03T13:11:56.663000", "content": "Thank you for your kind words, Jie. Me too, I am looking forward to working with you in the future.
", "votes": 1, "replies": [] } ] } ], "comment_count": 11, "messages": [ { "id": 1427550, "postDate": "2021-08-03T08:33:16.753Z", "content": "First of all, I would like to say thank you very much to my wonderful teammates, @horsek, and @lftuwujie, I learned a lot from them during this competition. Next, thank you to all competitors, the competition host, and Kaggle for competing and organizing this amazing competition. Besides, congratulation to all teams in the gold medal zone.
\nAt the beginning of this competition, I told @horsek that let's treat it as our chance to learn about NLP because I didn't have knowledge of it and it has been my first NLP competition.
\nIn this write-up, I would like to share not only my models but also my journey in this competition, from starting points to forming ideas under a newbie's viewpoint like me.
\nI started my journey with this wonderful notebook (https://www.kaggle.com/rhtsingh/commonlit-readability-prize-roberta-torch-fit), please give him/her an upvote.
\nHowever, I didn't fork it but read and re-wrote my own notebook. This helped me to debug, and forming some ideas to further customize and optimize the pipeline.
\nAfter this step, my best single model is the RoBERTa-large and scored 0.468 on the public LB and 0.469 on the private LB, my inference notebook is here
\nThe setting of this model is,
\nIn this competition (and also in all of my competitions), I studied related literature a lot. This step helped me to understand the concept of the competition and find new ideas. The first idea I got is the Quadratic-Weighted-Kappa loss, the second is the custom Bradley-Terry loss.
\nThe idea of this loss came from this paper. The main idea of this loss is binning the excerpt targets into categories, and train the model to classify them into these categories. In my opinion, this step is equivalent to label-smoothing, because it gathers excerpts with similar targets into a bin. The QWK loss is not a must, you can implement it with Cross-entropy loss. This is the implementation of this QWK loss,
\nclass QuadraticWeightedKappaLoss(nn.Module):\n def __init__(self, num_cat = 7, device = 'cpu'):\n super(QuadraticWeightedKappaLoss, self).__init__()\n self.num_cat = num_cat\n cats = torch.arange(num_cat).to(device)\n self.weights = (cats.view(-1,1) - cats.view(1,-1)).pow(2) / (num_cat - 1)**2\n\n def _confusion_matrix(self, pred_cat, true_cat):\n confusion_matrix = torch.zeros((self.num_cat, self.num_cat)).to(pred_cat.device)\n for t, p in zip(true_cat.view(-1), pred_cat.view(-1)):\n confusion_matrix[t.long(), p.long()] += 1\n return confusion_matrix\n\n def forward(self, pred_cat, true_cat):\n # Confusion matrix\n O = self._confusion_matrix(pred_cat, true_cat)\n\n # Count elements in each category\n true_hist = torch.bincount(true_cat, minlength = self.num_cat)\n pred_hist = torch.bincount(pred_cat, minlength = self.num_cat)\n\n # Expected values\n E = torch.outer(true_hist, pred_hist)\n\n # Normlization\n O = O / torch.sum(O)\n E = E / torch.sum(E)\n\n # Weighted Kappa\n numerator = torch.sum(self.weights * O)\n denominator = torch.sum(self.weights * E)\n\n return numerator / denominator\nMy best model after this step is again a RoBERTa-large with the following setting,
\nIt scored 0.466/0.466 on the public/private LB.
\nThis is so far the most effective trick and my most contribution to my team. The idea came from the observation about the benchmark excerpt which its target and standard error are both 0. As I studied the information from the host, each excerpt in the corpus is compared to some others. Then, by using the Bradley-Terry method, their prior targets are estimated. Finally, the prior targets of all excerpts are subtracted to the target a chosen benchmark excerpt to obtain the data we have.
\nFrom this observation, I designed my models accordingly. The model architecture is here (sorry that I couldn't insert an image to a post, so I uploaded my model architecture in another notebook).
\nThe implementation of the custom BT loss is as follow,
\nclass BradleyTerryLoss(nn.Module):\n def __init__(self):\n super(BradleyTerryLoss, self).__init__()\n\n def forward(self, pred_mean, true_mean):\n batch_size = len(pred_mean)\n true_comparison = true_mean.view(-1,1) - true_mean.view(1,-1)\n pred_comparison = pred_mean.view(-1,1) - pred_mean.view(1,-1)\n return torch.log(1 + torch.tril(torch.exp(-true_comparison * pred_comparison))).sum() / (batch_size * (batch_size - 1) / 2)\nWith this structure, the predicted target of the benchmark excerpt is always 0, which is consistent with the truth. This loss also allows models to compare the targets to each other. We applied this custom loss with many models of my teams.
\nThis custom loss gave me the best single model so far with 0.461/0.462 on the public and private LB.
\nMy best model after this step is again a RoBERTa-large with the following setting,
Using this setting, I train my models with different types of backbone. The following summarizes my best models according to public LB with each backbone (public LB/private LB),
\nFor each backbone, I sometimes changed the setting (evaluate less frequently, omit the QWK loss, training with another fold splitting, etc.) to avoid overfitting.
\nFinally, I simply took the equal average of all of my 13 models, it gave me 0.453/0.456 on the public/private LB. Ensemble my average model with my teammates' models gave us 0.448/0.452 on the public/private LB.
\nAlthough the result is not as good as I expected, I still enjoyed the competition a lot. From nearly 0 in NLP, after 3 months, I learned a lot from my teammates and fellow competitors. I hope to meet you in future competitions!
\n@shinomoriaoshi , I was studying this competition with a more specific focus on interesting loss engg strategies. Of course your solution tops my list. QWK is very apt and BT seems to be even more apt. My question is on the BT loss that you used…which is something of the form of log(1+e to power yhat . pred) for every pair in the batch. But ideally in the formula we should have restricted the prediction and groundtruths to +1 and -1 only for win and loss respectively. But I think you are using the BT scores directly. Is that correct or I am misunderstanding something? and somehow that too seems to work…
", "rawMarkdown": "@shinomoriaoshi , I was studying this competition with a more specific focus on interesting loss engg strategies. Of course your solution tops my list. QWK is very apt and BT seems to be even more apt. My question is on the BT loss that you used...which is something of the form of log(1+e to power yhat . pred) for every pair in the batch. But ideally in the formula we should have restricted the prediction and groundtruths to +1 and -1 only for win and loss respectively. But I think you are using the BT scores directly. Is that correct or I am misunderstanding something? and somehow that too seems to work...", "votes": 1 }, { "id": 1433007, "postDate": "2021-08-03T13:40:37.380Z", "content": "Congrats @shinomoriaoshi !! Few weeks ago I considered merging with your team but my ranking was not great and you guys just boosted up to the top so I decided not to since my approach requires a bit more submissions at the time. Could have been a great opportunity to learn from you guys!
", "rawMarkdown": "Congrats @shinomoriaoshi !! Few weeks ago I considered merging with your team but my ranking was not great and you guys just boosted up to the top so I decided not to since my approach requires a bit more submissions at the time. Could have been a great opportunity to learn from you guys!", "votes": 1, "replies": [ { "id": 1433465, "postDate": "2021-08-03T14:01:45.340Z", "content": "Thank you for your kind words. Haha, so now we should be the ones who regret about that. Anyway, congratulation with your solo gold. I look forward to working with you in future competitions.
", "rawMarkdown": "Thank you for your kind words. Haha, so now we should be the ones who regret about that. Anyway, congratulation with your solo gold. I look forward to working with you in future competitions.", "votes": 1 } ] }, { "id": 1428643, "postDate": "2021-08-03T09:40:46.007Z", "content": "Nice…I think this is the first solution post that talks of binning and loss-strategy. I wanted to explore it but couldn't get much of a chance. I wrote some code based on my own logic but scrapped it somewhere mid-way(should have persisted). I strongly felt that outliers skewed MSE and entropy related losses would give much better results…Anyway, good to read about QWK loss. As you mentioned, I guess the trick is to read more related literature…
", "rawMarkdown": "Nice...I think this is the first solution post that talks of binning and loss-strategy. I wanted to explore it but couldn't get much of a chance. I wrote some code based on my own logic but scrapped it somewhere mid-way(should have persisted). I strongly felt that outliers skewed MSE and entropy related losses would give much better results...Anyway, good to read about QWK loss. As you mentioned, I guess the trick is to read more related literature...", "votes": 1, "replies": [ { "id": 1428886, "postDate": "2021-08-03T10:12:20.187Z", "content": "Thank you, I am currently working in an academic environment, so I guess reading literature as my first step is just my habit, haha.
", "rawMarkdown": "Thank you, I am currently working in an academic environment, so I guess reading literature as my first step is just my habit, haha.", "votes": 1 }, { "id": 1429993, "postDate": "2021-08-03T11:11:27.530Z", "content": "I need to inculcate that habit for sure :)
\nW.r.t HAN, I was finally able to get it to a .473 score - in case you are interested in the model - https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257674
", "rawMarkdown": "I need to inculcate that habit for sure :)\n\nW.r.t HAN, I was finally able to get it to a .473 score - in case you are interested in the model - https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257674", "votes": 1 }, { "id": 1430534, "postDate": "2021-08-03T11:45:01.883Z", "content": "Thank you very much. I couldn't make HAN model break 0.50x. I will study your solution for sure.
", "rawMarkdown": "Thank you very much. I couldn't make HAN model break 0.50x. I will study your solution for sure.", "votes": 1 } ] }, { "id": 1446336, "postDate": "2021-08-04T11:55:26.483Z", "content": "@shinomoriaoshi Thank you so much for inviting me to this exciting competition! I had really great and meaningful days with you and @lftuwujie.
\nAt first I joined this competition as a learning experience, but as our ranking improved, I became more and more serious about it. haha
\nLet's win the gold medal together next time!
Haha, after seeing the improvement, we commit to this competition more seriously without realizing it. Let's get a gold next time!
", "rawMarkdown": "Haha, after seeing the improvement, we commit to this competition more seriously without realizing it. Let's get a gold next time!", "votes": 1 } ] }, { "id": 1431232, "postDate": "2021-08-03T12:17:11.903Z", "content": "@shinomoriaoshi very good summary, and it's a great journey to work with both you and @horsek. I also learnt quite a lot from you too. Looking forward to working together again.
", "rawMarkdown": "@shinomoriaoshi very good summary, and it's a great journey to work with both you and @horsek. I also learnt quite a lot from you too. Looking forward to working together again.", "votes": 2, "replies": [ { "id": 1432314, "postDate": "2021-08-03T13:11:56.663Z", "content": "Thank you for your kind words, Jie. Me too, I am looking forward to working with you in the future.
", "rawMarkdown": "Thank you for your kind words, Jie. Me too, I am looking forward to working with you in the future.", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:46:53Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6747769, "solution_url": "/competitions/commonlitreadabilityprize/writeups/trust-what-you-trust-private-28th-public-14th-solu" }