{
  "id": 446654,
  "title": "61th place solution",
  "content": "<h1>61th place solution</h1>\n<p>First of all,  I'd like to thank the competition host for organizing such an intriguing contest and my team members.  <br>\nMany of the ideas for this solution were contributed by my teammates.</p>\n<h3>Summary</h3>\n<p>The pipeline of our solution is as follows  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3836857%2F03a0f6f09ed783bbdbc1cdc759ab0fca%2Fimage.png?generation=1697118683872550&amp;alt=media\" alt=\"\"><br>\nThis solution was inspired by this <a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect\" target=\"_blank\">notebook</a>. Thanks for sharing this great notebook.</p>\n<h3>Strategy</h3>\n<p>The public leaderboard was derived from a single prompt, but based on <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/443223\" target=\"_blank\">discussion</a>, the private leaderboard encompasses over 100 prompts. This observation led me to place a heightened emphasis on cross-validation (CV). However, the CV too was based on merely four prompts, making it somewhat untrusted.<br>\nGiven the uncertainty in both CV and the public leaderboard, I opted for a strategy that emphasizes robustness. Consequently, I integrated various backbones and input texts into the ensemble, optimizing to reduce the inference time. Furthermore, considering that the evaluation metric was RMSE, ensuring prediction accuracy was paramount. Therefore, even models with suboptimal individual scores were incorporated into the ensemble.</p>\n<p>I couldn't effectively utilize the prompt_text, so I shifted my strategy to shorten the input length and reduce inference time.  </p>\n<h3>BERT Model Training</h3>\n<p>Training used Nakama's excellent <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">notebook</a> on Feedback3.<br>\nCommon to all  <br>\nepoch : 3  <br>\nschduler : cosine  <br>\nNo dropout  </p>\n<table>\n<thead>\n<tr>\n<th>backbone</th>\n<th>text</th>\n<th>maxlen</th>\n<th>loss</th>\n<th>freeze</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-base</td>\n<td>title question summary</td>\n<td>512</td>\n<td>SmoothL1Loss</td>\n<td>4</td>\n<td>0.54756</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>question summary</td>\n<td>512</td>\n<td>SmoothL1Loss</td>\n<td>12</td>\n<td>0.53988</td>\n</tr>\n<tr>\n<td>deberta-xlarge</td>\n<td>summary title question</td>\n<td>512</td>\n<td>SmoothL1Loss</td>\n<td>2</td>\n<td>0.54400</td>\n</tr>\n<tr>\n<td>electra-large</td>\n<td>question summary</td>\n<td>1024</td>\n<td>RMSE</td>\n<td>2</td>\n<td>0.55925</td>\n</tr>\n<tr>\n<td>muppet-roberta-large</td>\n<td>question summary</td>\n<td>512</td>\n<td>SmoothL1Loss</td>\n<td>4</td>\n<td>0.56337</td>\n</tr>\n</tbody>\n</table>\n<h3>GBDT</h3>\n<p>LightGBM, CatBoost, and XGBoost training were all parameter optimized with optuna.  <br>\nThe amount of features is based on this <a href=\"https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail\" target=\"_blank\">notebook</a>.</p>\n<h3>did not work</h3>\n<ul>\n<li>awp</li>\n<li>fgm</li>\n<li>SVR<ul>\n<li>(feedback3 chirs notebook)[<a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a>]</li></ul></li>\n<li>pseudo (might have worked)<ul>\n<li>The CV of feedback3 and old commonlit single had a gain of more than 0.01 on any backbone, but it worsened when put in GBDT, so it was not used.</li></ul></li>\n<li>text cleaning<ul>\n<li>Could capitalization and STOPWORD have also been a feature of wording?</li></ul></li>\n<li>wording anomaly detection<ul>\n<li>By removing data with extremely poor wording, the RMSE deteriorated significantly, so an anomaly detection model was created specifically for this, but it was abandoned due to lack of progress in learning.</li></ul></li>\n<li>FlanT5 encoder<ul>\n<li>The encoder part was taken from FlanT5, an encoder-decoder model often used in actual summarization, and used in the backbone, but the CV was 0.8x, which was very poor, so it was not used.</li></ul></li>\n</ul>\n<h3>Important Citations</h3>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect\" target=\"_blank\">2stage pipeline</a></li>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">feedback3 train notebook</a></li>\n<li><a href=\"https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/260729\" target=\"_blank\">no dropout</a></li>\n<li><a href=\"https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail\" target=\"_blank\">get features</a></li>\n</ul>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2479216,
      "postDate": "2023-10-12T13:53:56.503Z",
      "content": "<h1>61th place solution</h1>\n<p>First of all,  I'd like to thank the competition host for organizing such an intriguing contest and my team members.  <br>\nMany of the ideas for this solution were contributed by my teammates.</p>\n<h3>Summary</h3>\n<p>The pipeline of our solution is as follows  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3836857%2F03a0f6f09ed783bbdbc1cdc759ab0fca%2Fimage.png?generation=1697118683872550&amp;alt=media\" alt=\"\"><br>\nThis solution was inspired by this <a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect\" target=\"_blank\">notebook</a>. Thanks for sharing this great notebook.</p>\n<h3>Strategy</h3>\n<p>The public leaderboard was derived from a single prompt, but based on <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/443223\" target=\"_blank\">discussion</a>, the private leaderboard encompasses over 100 prompts. This observation led me to place a heightened emphasis on cross-validation (CV). However, the CV too was based on merely four prompts, making it somewhat untrusted.<br>\nGiven the uncertainty in both CV and the public leaderboard, I opted for a strategy that emphasizes robustness. Consequently, I integrated various backbones and input texts into the ensemble, optimizing to reduce the inference time. Furthermore, considering that the evaluation metric was RMSE, ensuring prediction accuracy was paramount. Therefore, even models with suboptimal individual scores were incorporated into the ensemble.</p>\n<p>I couldn't effectively utilize the prompt_text, so I shifted my strategy to shorten the input length and reduce inference time.  </p>\n<h3>BERT Model Training</h3>\n<p>Training used Nakama's excellent <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">notebook</a> on Feedback3.<br>\nCommon to all  <br>\nepoch : 3  <br>\nschduler : cosine  <br>\nNo dropout  </p>\n<table>\n<thead>\n<tr>\n<th>backbone</th>\n<th>text</th>\n<th>maxlen</th>\n<th>loss</th>\n<th>freeze</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-base</td>\n<td>title question summary</td>\n<td>512</td>\n<td>SmoothL1Loss</td>\n<td>4</td>\n<td>0.54756</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>question summary</td>\n<td>512</td>\n<td>SmoothL1Loss</td>\n<td>12</td>\n<td>0.53988</td>\n</tr>\n<tr>\n<td>deberta-xlarge</td>\n<td>summary title question</td>\n<td>512</td>\n<td>SmoothL1Loss</td>\n<td>2</td>\n<td>0.54400</td>\n</tr>\n<tr>\n<td>electra-large</td>\n<td>question summary</td>\n<td>1024</td>\n<td>RMSE</td>\n<td>2</td>\n<td>0.55925</td>\n</tr>\n<tr>\n<td>muppet-roberta-large</td>\n<td>question summary</td>\n<td>512</td>\n<td>SmoothL1Loss</td>\n<td>4</td>\n<td>0.56337</td>\n</tr>\n</tbody>\n</table>\n<h3>GBDT</h3>\n<p>LightGBM, CatBoost, and XGBoost training were all parameter optimized with optuna.  <br>\nThe amount of features is based on this <a href=\"https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail\" target=\"_blank\">notebook</a>.</p>\n<h3>did not work</h3>\n<ul>\n<li>awp</li>\n<li>fgm</li>\n<li>SVR<ul>\n<li>(feedback3 chirs notebook)[<a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a>]</li></ul></li>\n<li>pseudo (might have worked)<ul>\n<li>The CV of feedback3 and old commonlit single had a gain of more than 0.01 on any backbone, but it worsened when put in GBDT, so it was not used.</li></ul></li>\n<li>text cleaning<ul>\n<li>Could capitalization and STOPWORD have also been a feature of wording?</li></ul></li>\n<li>wording anomaly detection<ul>\n<li>By removing data with extremely poor wording, the RMSE deteriorated significantly, so an anomaly detection model was created specifically for this, but it was abandoned due to lack of progress in learning.</li></ul></li>\n<li>FlanT5 encoder<ul>\n<li>The encoder part was taken from FlanT5, an encoder-decoder model often used in actual summarization, and used in the backbone, but the CV was 0.8x, which was very poor, so it was not used.</li></ul></li>\n</ul>\n<h3>Important Citations</h3>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect\" target=\"_blank\">2stage pipeline</a></li>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">feedback3 train notebook</a></li>\n<li><a href=\"https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/260729\" target=\"_blank\">no dropout</a></li>\n<li><a href=\"https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail\" target=\"_blank\">get features</a></li>\n</ul>",
      "rawMarkdown": "\n# 61th place solution\n\nFirst of all,  I'd like to thank the competition host for organizing such an intriguing contest and my team members.  \nMany of the ideas for this solution were contributed by my teammates.\n\n### Summary\nThe pipeline of our solution is as follows  \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3836857%2F03a0f6f09ed783bbdbc1cdc759ab0fca%2Fimage.png?generation=1697118683872550&alt=media)\n\nThis solution was inspired by this [notebook](https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect). Thanks for sharing this great notebook.\n\n### Strategy\nThe public leaderboard was derived from a single prompt, but based on [discussion](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/443223), the private leaderboard encompasses over 100 prompts. This observation led me to place a heightened emphasis on cross-validation (CV). However, the CV too was based on merely four prompts, making it somewhat untrusted.\n\nGiven the uncertainty in both CV and the public leaderboard, I opted for a strategy that emphasizes robustness. Consequently, I integrated various backbones and input texts into the ensemble, optimizing to reduce the inference time. Furthermore, considering that the evaluation metric was RMSE, ensuring prediction accuracy was paramount. Therefore, even models with suboptimal individual scores were incorporated into the ensemble.\n \nI couldn't effectively utilize the prompt_text, so I shifted my strategy to shorten the input length and reduce inference time.  \n\n### BERT Model Training\nTraining used Nakama's excellent [notebook](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train) on Feedback3.\n\nCommon to all  \nepoch : 3  \nschduler : cosine  \nNo dropout  \n\n\n| backbone | text | maxlen | loss | freeze | CV |\n| --- | --- | --- | --- | --- | --- |\n| deberta-v3-base | title question summary | 512 | SmoothL1Loss | 4 | 0.54756 |\n| deberta-v3-large | question summary | 512 | SmoothL1Loss | 12 | 0.53988 |\n| deberta-xlarge | summary title question | 512 | SmoothL1Loss | 2 | 0.54400 |\n| electra-large | question summary | 1024 | RMSE | 2 | 0.55925 |\n| muppet-roberta-large | question summary | 512 | SmoothL1Loss | 4 | 0.56337 |\n\n### GBDT \nLightGBM, CatBoost, and XGBoost training were all parameter optimized with optuna.  \nThe amount of features is based on this [notebook](https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail).\n\n\n### did not work\n- awp\n- fgm\n- SVR\n    - (feedback3 chirs notebook)[https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x]\n- pseudo (might have worked)\n    - The CV of feedback3 and old commonlit single had a gain of more than 0.01 on any backbone, but it worsened when put in GBDT, so it was not used.\n- text cleaning\n    - Could capitalization and STOPWORD have also been a feature of wording?\n- wording anomaly detection\n    - By removing data with extremely poor wording, the RMSE deteriorated significantly, so an anomaly detection model was created specifically for this, but it was abandoned due to lack of progress in learning.\n- FlanT5 encoder\n    - The encoder part was taken from FlanT5, an encoder-decoder model often used in actual summarization, and used in the backbone, but the CV was 0.8x, which was very poor, so it was not used.\n\n### Important Citations\n- [2stage pipeline](https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect)\n- [feedback3 train notebook](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)\n- [no dropout](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/260729)\n- [get features](https://www.kaggle.com/code/wchan757/achieving-lb-0-47-with-just-lightgbm-detail)\n",
      "votes": 8
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:04:00Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10674914,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/wanwan7123-61th-place-solution"
}