{
  "id": 448625,
  "title": "21th place solution",
  "content": "<p>First of all, I would like to thank the organizers for hosting such an interesting competition. Also thanks to kaggler for sharing so much useful information through notebooks and discussions.</p>\n<p>I share my solution.</p>\n<p><br></p>\n<h2>Overview</h2>\n<p>My final submission is an ensemble of two deberta-v3-large models and LightGBM.<br>\nThe pipeline is as follows.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6454400%2F9c4ca8159dbb9fb98f9cefa2e4e90b3d%2F(81).png?generation=1697805022893951&amp;alt=media\" alt=\"\"></p>\n<p><br></p>\n<h2>Model</h2>\n<ul>\n<li>Deberta v3 large<ul>\n<li>input_text = text+prompt_question+prompt_text</li>\n<li>maxlen = 1024</li>\n<li>CLS token</li>\n<li>loss : RMSELoss</li>\n<li>epoch = 4</li></ul></li>\n</ul>\n<p><br></p>\n<ul>\n<li>Deberta v3 base<ul>\n<li>input_text = prompt_title+len2text+prompt_question+text (※len2text : [<a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/437591\" target=\"_blank\">ref</a>])</li>\n<li>maxlen = 512</li>\n<li>Attention Pooling</li>\n<li>loss : RMSELoss</li>\n<li>epoch = 4</li></ul></li>\n</ul>\n<p><br></p>\n<ul>\n<li>LightGBM<ul>\n<li>Almost the same as <a href=\"https://www.kaggle.com/code/tsunotsuno/debertav3-lgbm-no-autocorrect\" target=\"_blank\">publish nootbook</a></li></ul></li>\n</ul>\n<p><br><br></p>\n<h2>Kye Points</h2>\n<p>The following two points contributed significantly to improving the accuracy of the deberta model.</p>\n<ul>\n<li>Grouped-LLRD (Layer Wise Learning Rate Decay)</li>\n<li>freeze layer (freeze 4 layers close to embeddeding)  [<a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/433754#2405543\" target=\"_blank\">ref</a>]</li>\n</ul>\n<p><br></p>\n<h2>CV</h2>\n<ul>\n<li>validation strategy : GroupKFold(groups=prompt_id)</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>39c16e</th>\n<th>3b9047</th>\n<th>ebad26</th>\n<th>814d6b</th>\n<th>cv</th>\n<th>public</th>\n<th>private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>model1</td>\n<td>0.4768</td>\n<td>0.5045</td>\n<td>0.4461</td>\n<td>0.5842</td>\n<td>0.4948</td>\n<td>0.466</td>\n<td>0.471</td>\n</tr>\n<tr>\n<td>model2</td>\n<td>0.4540</td>\n<td>0.5154</td>\n<td>0.4427</td>\n<td>0.5830</td>\n<td>0.4907</td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>LGBM</td>\n<td>0.4730</td>\n<td>0.5802</td>\n<td>0.4592</td>\n<td>0.5809</td>\n<td>0.5197</td>\n<td>0.449</td>\n<td>0.481</td>\n</tr>\n</tbody>\n</table>\n<p><br></p>\n<h3>Ensemble</h3>\n<ul>\n<li>weighted average<ul>\n<li>cv : 0.4777</li>\n<li>Public LB : 0.4351</li>\n<li>Private LB : 0.46163</li></ul></li>\n</ul>\n<p><br></p>\n<h2>Didn't Work</h2>\n<ul>\n<li>text cleaning</li>\n<li>awp</li>\n<li>fgm</li>\n<li>svr </li>\n<li>pseudo labels </li>\n<li>etc…</li>\n</ul>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2490158,
      "postDate": "2023-10-20T13:30:08.657Z",
      "content": "<p>First of all, I would like to thank the organizers for hosting such an interesting competition. Also thanks to kaggler for sharing so much useful information through notebooks and discussions.</p>\n<p>I share my solution.</p>\n<p><br></p>\n<h2>Overview</h2>\n<p>My final submission is an ensemble of two deberta-v3-large models and LightGBM.<br>\nThe pipeline is as follows.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6454400%2F9c4ca8159dbb9fb98f9cefa2e4e90b3d%2F(81).png?generation=1697805022893951&amp;alt=media\" alt=\"\"></p>\n<p><br></p>\n<h2>Model</h2>\n<ul>\n<li>Deberta v3 large<ul>\n<li>input_text = text+prompt_question+prompt_text</li>\n<li>maxlen = 1024</li>\n<li>CLS token</li>\n<li>loss : RMSELoss</li>\n<li>epoch = 4</li></ul></li>\n</ul>\n<p><br></p>\n<ul>\n<li>Deberta v3 base<ul>\n<li>input_text = prompt_title+len2text+prompt_question+text (※len2text : [<a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/437591\" target=\"_blank\">ref</a>])</li>\n<li>maxlen = 512</li>\n<li>Attention Pooling</li>\n<li>loss : RMSELoss</li>\n<li>epoch = 4</li></ul></li>\n</ul>\n<p><br></p>\n<ul>\n<li>LightGBM<ul>\n<li>Almost the same as <a href=\"https://www.kaggle.com/code/tsunotsuno/debertav3-lgbm-no-autocorrect\" target=\"_blank\">publish nootbook</a></li></ul></li>\n</ul>\n<p><br><br></p>\n<h2>Kye Points</h2>\n<p>The following two points contributed significantly to improving the accuracy of the deberta model.</p>\n<ul>\n<li>Grouped-LLRD (Layer Wise Learning Rate Decay)</li>\n<li>freeze layer (freeze 4 layers close to embeddeding)  [<a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/433754#2405543\" target=\"_blank\">ref</a>]</li>\n</ul>\n<p><br></p>\n<h2>CV</h2>\n<ul>\n<li>validation strategy : GroupKFold(groups=prompt_id)</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>39c16e</th>\n<th>3b9047</th>\n<th>ebad26</th>\n<th>814d6b</th>\n<th>cv</th>\n<th>public</th>\n<th>private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>model1</td>\n<td>0.4768</td>\n<td>0.5045</td>\n<td>0.4461</td>\n<td>0.5842</td>\n<td>0.4948</td>\n<td>0.466</td>\n<td>0.471</td>\n</tr>\n<tr>\n<td>model2</td>\n<td>0.4540</td>\n<td>0.5154</td>\n<td>0.4427</td>\n<td>0.5830</td>\n<td>0.4907</td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>LGBM</td>\n<td>0.4730</td>\n<td>0.5802</td>\n<td>0.4592</td>\n<td>0.5809</td>\n<td>0.5197</td>\n<td>0.449</td>\n<td>0.481</td>\n</tr>\n</tbody>\n</table>\n<p><br></p>\n<h3>Ensemble</h3>\n<ul>\n<li>weighted average<ul>\n<li>cv : 0.4777</li>\n<li>Public LB : 0.4351</li>\n<li>Private LB : 0.46163</li></ul></li>\n</ul>\n<p><br></p>\n<h2>Didn't Work</h2>\n<ul>\n<li>text cleaning</li>\n<li>awp</li>\n<li>fgm</li>\n<li>svr </li>\n<li>pseudo labels </li>\n<li>etc…</li>\n</ul>",
      "rawMarkdown": "First of all, I would like to thank the organizers for hosting such an interesting competition. Also thanks to kaggler for sharing so much useful information through notebooks and discussions.\n\nI share my solution.\n\n<br />\n## Overview\nMy final submission is an ensemble of two deberta-v3-large models and LightGBM.\nThe pipeline is as follows.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6454400%2F9c4ca8159dbb9fb98f9cefa2e4e90b3d%2F(81).png?generation=1697805022893951&alt=media)\n\n<br />\n\n\n## Model\n- Deberta v3 large\n  - input_text = text+prompt_question+prompt_text\n  - maxlen = 1024\n  - CLS token\n  - loss : RMSELoss\n  - epoch = 4\n\n<br />\n\n- Deberta v3 base\n  - input_text = prompt_title+len2text+prompt_question+text (※len2text : [[ref](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/437591)])\n  - maxlen = 512\n  - Attention Pooling\n  - loss : RMSELoss\n  - epoch = 4\n\n<br />\n\n- LightGBM\n  - Almost the same as [publish nootbook](https://www.kaggle.com/code/tsunotsuno/debertav3-lgbm-no-autocorrect)\n\n\n<br /><br />\n\n## Kye Points\nThe following two points contributed significantly to improving the accuracy of the deberta model.\n- Grouped-LLRD (Layer Wise Learning Rate Decay)\n- freeze layer (freeze 4 layers close to embeddeding)  [[ref](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/433754#2405543)]\n\n\n<br />\n\n## CV\n- validation strategy : GroupKFold(groups=prompt_id)\n\n|  | 39c16e | 3b9047 | ebad26 | 814d6b | cv | public | private |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| model1 | 0.4768 | 0.5045 | 0.4461 | 0.5842 | 0.4948 | 0.466 | 0.471 |\n| model2 | 0.4540 | 0.5154 | 0.4427 | 0.5830 | 0.4907 |   |   |\n| LGBM | 0.4730 | 0.5802 | 0.4592 | 0.5809 | 0.5197 | 0.449 |0.481 | \n\n\n<br />\n\n### Ensemble\n- weighted average\n  - cv : 0.4777\n  - Public LB : 0.4351\n  - Private LB : 0.46163\n\n<br />\n\n\n## Didn't Work\n- text cleaning\n- awp\n- fgm\n- svr \n- pseudo labels \n- etc…\n",
      "votes": 2
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:08:00Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10678481,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/hisariki-21th-place-solution"
}