{
  "id": 447822,
  "title": "42nd place solution ",
  "content": "<p>This was my first Kaggle competition and I ended in 42nd position out of over 2000 teams.</p>\n<ul>\n<li>Training code: <a href=\"https://github.com/Lukasz-Cesarski/cless/blob/main/cless/run_sweep.py\" target=\"_blank\">cless/run_sweep.py</a>  </li>\n<li>Inference <a href=\"https://www.kaggle.com/code/luki493/lgbm-ensemble-iter-e6-open\" target=\"_blank\">code</a></li>\n</ul>\n<h2>Solution</h2>\n<p>Final solution contains</p>\n<ul>\n<li>6 <a href=\"https://huggingface.co/docs/transformers/model_doc/deberta\" target=\"_blank\">Deberta</a> models (2 base and 4 large)</li>\n<li>LGBM on top of every model </li>\n<li>Late Fusion ensemble</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2200033%2Feab6107c139b5f93f4f14b440ed33a7d%2Fsolution.png?generation=1697541256075838&amp;alt=media\" alt=\"\"></p>\n<h2>Results</h2>\n<h3>Cross Validation:</h3>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>checkpoint_name</th>\n<th>model_name</th>\n<th>adds</th>\n<th>pseudo</th>\n<th>content_rmse</th>\n<th>wording_rmse</th>\n<th>mcrmse</th>\n<th>lgbm_content_rmse</th>\n<th>lgbm_wording_rmse</th>\n<th>lgbm_mcrmse</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>lrg-add-nops-202310101531</td>\n<td>large</td>\n<td>True</td>\n<td>False</td>\n<td>0.436815</td>\n<td>0.563804</td>\n<td>0.500309</td>\n<td>0.416913</td>\n<td>0.547992</td>\n<td>0.482452</td>\n</tr>\n<tr>\n<td>2</td>\n<td>lrg-add2-nops-202310071225</td>\n<td>large</td>\n<td>True</td>\n<td>False</td>\n<td>0.422034</td>\n<td>0.557535</td>\n<td>0.489785</td>\n<td>0.422094</td>\n<td>0.552657</td>\n<td>0.487376</td>\n</tr>\n<tr>\n<td>3</td>\n<td>deberta-large-pseudo-20231004</td>\n<td>large</td>\n<td>False</td>\n<td>True</td>\n<td>0.447833</td>\n<td>0.599299</td>\n<td>0.523566</td>\n<td>0.431393</td>\n<td>0.559015</td>\n<td>0.495204</td>\n</tr>\n<tr>\n<td>4</td>\n<td>cless-deberta-20230919-2131-ensamble</td>\n<td>base</td>\n<td>False</td>\n<td>False</td>\n<td>0.478477</td>\n<td>0.6168</td>\n<td>0.547638</td>\n<td>0.436959</td>\n<td>0.556711</td>\n<td>0.496835</td>\n</tr>\n<tr>\n<td>5</td>\n<td>base-noadd-pseu-202310101211</td>\n<td>base</td>\n<td>False</td>\n<td>True</td>\n<td>0.450015</td>\n<td>0.631051</td>\n<td>0.540533</td>\n<td>0.426784</td>\n<td>0.570818</td>\n<td>0.498801</td>\n</tr>\n<tr>\n<td>6</td>\n<td>lrg-add2-ps-frz8-202310061415</td>\n<td>large</td>\n<td>True</td>\n<td>True</td>\n<td>0.450208</td>\n<td>0.574412</td>\n<td>0.51231</td>\n<td>0.439451</td>\n<td>0.563622</td>\n<td>0.501536</td>\n</tr>\n</tbody>\n</table>\n<p>Columns:</p>\n<ul>\n<li><strong>checkpoint_name</strong> - name of model checkpoint (4 folds inside) </li>\n<li><strong>model_name</strong> - which model was used: <code>microsoft/deberta-v3-base</code> or <code>microsoft/deberta-v3-large</code></li>\n<li><strong>adds</strong> - if <code>True</code> then <code>prompt_text</code> and <code>prompt_question</code> were both added to model as input</li>\n<li><strong>pseudo</strong> - pretraining on <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/data\" target=\"_blank\">Feedback Prize - English Language Learning</a> data</li>\n<li><strong>content_rmse</strong>, <strong>wording_rmse</strong>, <strong>mcrmse</strong> - metrics of raw Deberta Regressor model </li>\n<li><strong>lgbm_content_rmse</strong>, <strong>lgbm_wording_rmse</strong>, <strong>lgbm_mcrmse</strong> - metrics of LGBM Regressor on top of every Deberta</li>\n</ul>\n<p><strong>Note</strong>: Models without <code>prompt_text</code> and <code>prompt_question</code> in input (adds=False) has much worse CV <code>mcrmse</code>. <br>\nHowever this is mitigated by LGBM which utilizes <code>prompt_text</code> handcrafted text mining techniques and <code>lgbm_mcrmse</code> <br>\nis significantly improved for these models</p>\n<h3>Ensemble metrics</h3>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>metric</th>\n<th>value</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>content_rmse</td>\n<td>0.40780695722821314</td>\n</tr>\n<tr>\n<td>2</td>\n<td>wording_rmse</td>\n<td>0.5389858871892348</td>\n</tr>\n<tr>\n<td>3</td>\n<td>mcrmse</td>\n<td>0.47339642220872397</td>\n</tr>\n</tbody>\n</table>\n<h3>Leaderboard</h3>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>metric</th>\n<th>value</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>mcrmse (public)</td>\n<td>0.432</td>\n</tr>\n<tr>\n<td>2</td>\n<td>mcrmse (private)</td>\n<td>0.468</td>\n</tr>\n</tbody>\n</table>\n<p>Kudos for the author of <a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect\" target=\"_blank\">this notebook</a> it helped me a lot.</p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2485674,
      "postDate": "2023-10-17T11:14:43.007Z",
      "content": "<p>This was my first Kaggle competition and I ended in 42nd position out of over 2000 teams.</p>\n<ul>\n<li>Training code: <a href=\"https://github.com/Lukasz-Cesarski/cless/blob/main/cless/run_sweep.py\" target=\"_blank\">cless/run_sweep.py</a>  </li>\n<li>Inference <a href=\"https://www.kaggle.com/code/luki493/lgbm-ensemble-iter-e6-open\" target=\"_blank\">code</a></li>\n</ul>\n<h2>Solution</h2>\n<p>Final solution contains</p>\n<ul>\n<li>6 <a href=\"https://huggingface.co/docs/transformers/model_doc/deberta\" target=\"_blank\">Deberta</a> models (2 base and 4 large)</li>\n<li>LGBM on top of every model </li>\n<li>Late Fusion ensemble</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2200033%2Feab6107c139b5f93f4f14b440ed33a7d%2Fsolution.png?generation=1697541256075838&amp;alt=media\" alt=\"\"></p>\n<h2>Results</h2>\n<h3>Cross Validation:</h3>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>checkpoint_name</th>\n<th>model_name</th>\n<th>adds</th>\n<th>pseudo</th>\n<th>content_rmse</th>\n<th>wording_rmse</th>\n<th>mcrmse</th>\n<th>lgbm_content_rmse</th>\n<th>lgbm_wording_rmse</th>\n<th>lgbm_mcrmse</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>lrg-add-nops-202310101531</td>\n<td>large</td>\n<td>True</td>\n<td>False</td>\n<td>0.436815</td>\n<td>0.563804</td>\n<td>0.500309</td>\n<td>0.416913</td>\n<td>0.547992</td>\n<td>0.482452</td>\n</tr>\n<tr>\n<td>2</td>\n<td>lrg-add2-nops-202310071225</td>\n<td>large</td>\n<td>True</td>\n<td>False</td>\n<td>0.422034</td>\n<td>0.557535</td>\n<td>0.489785</td>\n<td>0.422094</td>\n<td>0.552657</td>\n<td>0.487376</td>\n</tr>\n<tr>\n<td>3</td>\n<td>deberta-large-pseudo-20231004</td>\n<td>large</td>\n<td>False</td>\n<td>True</td>\n<td>0.447833</td>\n<td>0.599299</td>\n<td>0.523566</td>\n<td>0.431393</td>\n<td>0.559015</td>\n<td>0.495204</td>\n</tr>\n<tr>\n<td>4</td>\n<td>cless-deberta-20230919-2131-ensamble</td>\n<td>base</td>\n<td>False</td>\n<td>False</td>\n<td>0.478477</td>\n<td>0.6168</td>\n<td>0.547638</td>\n<td>0.436959</td>\n<td>0.556711</td>\n<td>0.496835</td>\n</tr>\n<tr>\n<td>5</td>\n<td>base-noadd-pseu-202310101211</td>\n<td>base</td>\n<td>False</td>\n<td>True</td>\n<td>0.450015</td>\n<td>0.631051</td>\n<td>0.540533</td>\n<td>0.426784</td>\n<td>0.570818</td>\n<td>0.498801</td>\n</tr>\n<tr>\n<td>6</td>\n<td>lrg-add2-ps-frz8-202310061415</td>\n<td>large</td>\n<td>True</td>\n<td>True</td>\n<td>0.450208</td>\n<td>0.574412</td>\n<td>0.51231</td>\n<td>0.439451</td>\n<td>0.563622</td>\n<td>0.501536</td>\n</tr>\n</tbody>\n</table>\n<p>Columns:</p>\n<ul>\n<li><strong>checkpoint_name</strong> - name of model checkpoint (4 folds inside) </li>\n<li><strong>model_name</strong> - which model was used: <code>microsoft/deberta-v3-base</code> or <code>microsoft/deberta-v3-large</code></li>\n<li><strong>adds</strong> - if <code>True</code> then <code>prompt_text</code> and <code>prompt_question</code> were both added to model as input</li>\n<li><strong>pseudo</strong> - pretraining on <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/data\" target=\"_blank\">Feedback Prize - English Language Learning</a> data</li>\n<li><strong>content_rmse</strong>, <strong>wording_rmse</strong>, <strong>mcrmse</strong> - metrics of raw Deberta Regressor model </li>\n<li><strong>lgbm_content_rmse</strong>, <strong>lgbm_wording_rmse</strong>, <strong>lgbm_mcrmse</strong> - metrics of LGBM Regressor on top of every Deberta</li>\n</ul>\n<p><strong>Note</strong>: Models without <code>prompt_text</code> and <code>prompt_question</code> in input (adds=False) has much worse CV <code>mcrmse</code>. <br>\nHowever this is mitigated by LGBM which utilizes <code>prompt_text</code> handcrafted text mining techniques and <code>lgbm_mcrmse</code> <br>\nis significantly improved for these models</p>\n<h3>Ensemble metrics</h3>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>metric</th>\n<th>value</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>content_rmse</td>\n<td>0.40780695722821314</td>\n</tr>\n<tr>\n<td>2</td>\n<td>wording_rmse</td>\n<td>0.5389858871892348</td>\n</tr>\n<tr>\n<td>3</td>\n<td>mcrmse</td>\n<td>0.47339642220872397</td>\n</tr>\n</tbody>\n</table>\n<h3>Leaderboard</h3>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>metric</th>\n<th>value</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>mcrmse (public)</td>\n<td>0.432</td>\n</tr>\n<tr>\n<td>2</td>\n<td>mcrmse (private)</td>\n<td>0.468</td>\n</tr>\n</tbody>\n</table>\n<p>Kudos for the author of <a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect\" target=\"_blank\">this notebook</a> it helped me a lot.</p>",
      "rawMarkdown": "This was my first Kaggle competition and I ended in 42nd position out of over 2000 teams.\n\n* Training code: [cless/run_sweep.py](https://github.com/Lukasz-Cesarski/cless/blob/main/cless/run_sweep.py)  \n* Inference [code](https://www.kaggle.com/code/luki493/lgbm-ensemble-iter-e6-open)\n\n## Solution\n\nFinal solution contains\n* 6 [Deberta](https://huggingface.co/docs/transformers/model_doc/deberta) models (2 base and 4 large)\n* LGBM on top of every model \n* Late Fusion ensemble\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2200033%2Feab6107c139b5f93f4f14b440ed33a7d%2Fsolution.png?generation=1697541256075838&alt=media)\n\n\n## Results\n\n### Cross Validation:\n\n|    | checkpoint_name                      | model_name   | adds   | pseudo |   content_rmse |   wording_rmse |   mcrmse |   lgbm_content_rmse |   lgbm_wording_rmse |   lgbm_mcrmse |\n|---:|:-------------------------------------|:-------------|:-------|:-------|---------------:|---------------:|---------:|--------------------:|--------------------:|--------------:|\n|  1 | lrg-add-nops-202310101531            | large        | True   | False  |       0.436815 |       0.563804 | 0.500309 |            0.416913 |            0.547992 |      0.482452 |\n|  2 | lrg-add2-nops-202310071225           | large        | True   | False  |       0.422034 |       0.557535 | 0.489785 |            0.422094 |            0.552657 |      0.487376 |\n|  3 | deberta-large-pseudo-20231004        | large        | False  | True   |       0.447833 |       0.599299 | 0.523566 |            0.431393 |            0.559015 |      0.495204 |\n|  4 | cless-deberta-20230919-2131-ensamble | base         | False  | False  |       0.478477 |       0.6168   | 0.547638 |            0.436959 |            0.556711 |      0.496835 |\n|  5 | base-noadd-pseu-202310101211         | base         | False  | True   |       0.450015 |       0.631051 | 0.540533 |            0.426784 |            0.570818 |      0.498801 |\n|  6 | lrg-add2-ps-frz8-202310061415        | large        | True   | True   |       0.450208 |       0.574412 | 0.51231  |            0.439451 |            0.563622 |      0.501536 |\n\nColumns:\n* **checkpoint_name** - name of model checkpoint (4 folds inside) \n* **model_name** - which model was used: `microsoft/deberta-v3-base` or `microsoft/deberta-v3-large`\n* **adds** - if `True` then `prompt_text` and `prompt_question` were both added to model as input\n* **pseudo** - pretraining on [Feedback Prize - English Language Learning](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/data) data\n* **content_rmse**, **wording_rmse**, **mcrmse** - metrics of raw Deberta Regressor model \n* **lgbm_content_rmse**, **lgbm_wording_rmse**, **lgbm_mcrmse** - metrics of LGBM Regressor on top of every Deberta\n\n\n**Note**: Models without `prompt_text` and `prompt_question` in input (adds=False) has much worse CV `mcrmse`. \nHowever this is mitigated by LGBM which utilizes `prompt_text` handcrafted text mining techniques and `lgbm_mcrmse` \nis significantly improved for these models\n\n### Ensemble metrics\n\n|   | metric                    | value |\n|--:|:--------------------------|:------|\n| 1 | content_rmse | 0.40780695722821314 | \n| 2 | wording_rmse | 0.5389858871892348 | \n| 3 | mcrmse | 0.47339642220872397 | \n\n### Leaderboard\n\n|   | metric           | value |\n|--:|:-----------------|:------|\n| 1 | mcrmse (public)  | 0.432 | \n| 2 | mcrmse (private) | 0.468 |\n\nKudos for the author of [this notebook](https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-spell-autocorrect) it helped me a lot.\n",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:07:12Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10743093,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/lawliet-42nd-place-solution"
}