{
  "id": 257795,
  "title": "#24 Solution",
  "content": "<p>Firstly, thanks to the hosts and Kaggle for hosting this interesting competition. Also, great thanks to torch( <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> ) for share his/her experience, which helps me break 0.485 and could get better score.</p>\n<h1>Summery</h1>\n<p>My final model consist of 2 kind of model and totally 17 models.</p>\n<ol>\n<li>single model: <strong>Input</strong>: encoded row text. <strong>Model</strong>: roberta/deberta/bart last_hidden_states with mean pooling and attention head.</li>\n<li>pair model: <strong>Input</strong>: baseline encoded text and encoded row text. <strong>Model</strong>: same architecture for baseline text and row text, then concatenate (or minus).</li>\n</ol>\n<p><em>As hosts say, every score is a compare of one text to baseline text.</em> <a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/236403\" target=\"_blank\">here</a></p>\n<h1>Fine-tune</h1>\n<p>My final models are fine-tuned with almost same fine-tune strategy. That is:</p>\n<ul>\n<li>Epochs: 5 for base mode and 3 for large model</li>\n<li>Batch Size: almost maximize the GPU memory. (4 - 24)</li>\n<li>Gradient Accumulate: 1-2</li>\n<li>Optimizer: AdamW + Lookahead</li>\n<li>LR Scheduler: None for base model and cosine_schedule_with_warmup for large model (warmup steps &gt; 0)</li>\n<li>LLRD: use torch's shared settings and set base_lr=5e-5</li>\n<li>Random initial top layers: 1-2 for base model and 4-6 for large model</li>\n<li>Evaluating for every n steps: 10 to 60 steps</li>\n</ul>\n<h1>Scoring</h1>\n<p>Just report my best score of the same base transformers.</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Pair(1/0)</th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Roberta-large</td>\n<td>0</td>\n<td>0.47835</td>\n<td>0.466</td>\n<td>0.470</td>\n</tr>\n<tr>\n<td>Roberta-large</td>\n<td>1</td>\n<td>0.46694</td>\n<td>0.460</td>\n<td>0.459</td>\n</tr>\n<tr>\n<td>Roberta-base</td>\n<td>1</td>\n<td>0.47325</td>\n<td>0.475</td>\n<td>0.477</td>\n</tr>\n<tr>\n<td>Deberta-large</td>\n<td>0</td>\n<td>0.46440</td>\n<td>0.460</td>\n<td>0.461</td>\n</tr>\n<tr>\n<td>Deberta-large</td>\n<td>1</td>\n<td>0.46673</td>\n<td>0.462</td>\n<td>0.464</td>\n</tr>\n<tr>\n<td>Bart-large</td>\n<td>0</td>\n<td>0.46184</td>\n<td>0.462</td>\n<td>0.464</td>\n</tr>\n<tr>\n<td>Bart-large</td>\n<td>1</td>\n<td>0.46401</td>\n<td>0.463</td>\n<td>0.463</td>\n</tr>\n<tr>\n<td>Bart-base</td>\n<td>1</td>\n<td>0.47103</td>\n<td>0.470</td>\n<td>0.472</td>\n</tr>\n<tr>\n<td>Xlnet-large-cased</td>\n<td>1</td>\n<td>0.47427</td>\n<td>0.475</td>\n<td>0.465</td>\n</tr>\n<tr>\n<td>Electra-large</td>\n<td>1</td>\n<td>0.47100</td>\n<td>0.467</td>\n<td>0.472</td>\n</tr>\n<tr>\n<td>xlm-roberta-large</td>\n<td>1</td>\n<td>0.48622</td>\n<td>0.477</td>\n<td>0.478</td>\n</tr>\n<tr>\n<td>luke-large</td>\n<td>1</td>\n<td>0.47148</td>\n<td>0.471</td>\n<td>0.465</td>\n</tr>\n<tr>\n<td>funnel-large-base</td>\n<td>1</td>\n<td>0.47117</td>\n<td>0.469</td>\n<td>0.468</td>\n</tr>\n</tbody>\n</table>\n<p>Could not make other models work, and could not get a score below 0.5 even for fold 0.</p>\n<h1>Ensemble</h1>\n<p>Simple <code>BayesianRidge</code> regression. Best score: CV is 0.447 and Public LB 0.451 Private LB 0.452.</p>\n<p>Thanks!</p>",
  "comments": [
    {
      "id": 1443933,
      "author_name": "torch",
      "author_url": "",
      "post_date": "2021-08-04T04:33:49.410000",
      "content": "<p>Many congratulations on your solo silver and thanks <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> for sharing your solution. </p>\n<p>I learned a lot from our discussions during the competition and it was really amazing watching you consistently climbing the leaderboard. Great effort and well deserved! 😊</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1450621,
          "author_name": "yao",
          "author_url": "",
          "post_date": "2021-08-05T06:29:04.790000",
          "content": "<p>Without your help, I could not get these improvement. Really thanks! Hope you to get your notebook GM someday!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1430092,
      "author_name": "Ahmed Dawoud",
      "author_url": "",
      "post_date": "2021-08-03T11:17:25.647000",
      "content": "<p>Great Job. Thanks for sharing!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1450622,
          "author_name": "yao",
          "author_url": "",
          "post_date": "2021-08-05T06:29:28.230000",
          "content": "<p>You are welcome！</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1425325,
      "author_name": "huiqin",
      "author_url": "",
      "post_date": "2021-08-03T06:45:30.940000",
      "content": "<p>Thanks for sharing! You did the good job!   We also train Bart-large, but it was only 0.48x lb. </p>",
      "votes": 0,
      "replies": [
        {
          "id": 1426361,
          "author_name": "yao",
          "author_url": "",
          "post_date": "2021-08-03T07:30:14.150000",
          "content": "<p>Thank you! Congratulations on winning another gold medal!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 6,
  "messages": [
    {
      "id": 1423408,
      "postDate": "2021-08-03T05:38:59.153Z",
      "content": "<p>Firstly, thanks to the hosts and Kaggle for hosting this interesting competition. Also, great thanks to torch( <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> ) for share his/her experience, which helps me break 0.485 and could get better score.</p>\n<h1>Summery</h1>\n<p>My final model consist of 2 kind of model and totally 17 models.</p>\n<ol>\n<li>single model: <strong>Input</strong>: encoded row text. <strong>Model</strong>: roberta/deberta/bart last_hidden_states with mean pooling and attention head.</li>\n<li>pair model: <strong>Input</strong>: baseline encoded text and encoded row text. <strong>Model</strong>: same architecture for baseline text and row text, then concatenate (or minus).</li>\n</ol>\n<p><em>As hosts say, every score is a compare of one text to baseline text.</em> <a href=\"https://www.kaggle.com/c/commonlitreadabilityprize/discussion/236403\" target=\"_blank\">here</a></p>\n<h1>Fine-tune</h1>\n<p>My final models are fine-tuned with almost same fine-tune strategy. That is:</p>\n<ul>\n<li>Epochs: 5 for base mode and 3 for large model</li>\n<li>Batch Size: almost maximize the GPU memory. (4 - 24)</li>\n<li>Gradient Accumulate: 1-2</li>\n<li>Optimizer: AdamW + Lookahead</li>\n<li>LR Scheduler: None for base model and cosine_schedule_with_warmup for large model (warmup steps &gt; 0)</li>\n<li>LLRD: use torch's shared settings and set base_lr=5e-5</li>\n<li>Random initial top layers: 1-2 for base model and 4-6 for large model</li>\n<li>Evaluating for every n steps: 10 to 60 steps</li>\n</ul>\n<h1>Scoring</h1>\n<p>Just report my best score of the same base transformers.</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Pair(1/0)</th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Roberta-large</td>\n<td>0</td>\n<td>0.47835</td>\n<td>0.466</td>\n<td>0.470</td>\n</tr>\n<tr>\n<td>Roberta-large</td>\n<td>1</td>\n<td>0.46694</td>\n<td>0.460</td>\n<td>0.459</td>\n</tr>\n<tr>\n<td>Roberta-base</td>\n<td>1</td>\n<td>0.47325</td>\n<td>0.475</td>\n<td>0.477</td>\n</tr>\n<tr>\n<td>Deberta-large</td>\n<td>0</td>\n<td>0.46440</td>\n<td>0.460</td>\n<td>0.461</td>\n</tr>\n<tr>\n<td>Deberta-large</td>\n<td>1</td>\n<td>0.46673</td>\n<td>0.462</td>\n<td>0.464</td>\n</tr>\n<tr>\n<td>Bart-large</td>\n<td>0</td>\n<td>0.46184</td>\n<td>0.462</td>\n<td>0.464</td>\n</tr>\n<tr>\n<td>Bart-large</td>\n<td>1</td>\n<td>0.46401</td>\n<td>0.463</td>\n<td>0.463</td>\n</tr>\n<tr>\n<td>Bart-base</td>\n<td>1</td>\n<td>0.47103</td>\n<td>0.470</td>\n<td>0.472</td>\n</tr>\n<tr>\n<td>Xlnet-large-cased</td>\n<td>1</td>\n<td>0.47427</td>\n<td>0.475</td>\n<td>0.465</td>\n</tr>\n<tr>\n<td>Electra-large</td>\n<td>1</td>\n<td>0.47100</td>\n<td>0.467</td>\n<td>0.472</td>\n</tr>\n<tr>\n<td>xlm-roberta-large</td>\n<td>1</td>\n<td>0.48622</td>\n<td>0.477</td>\n<td>0.478</td>\n</tr>\n<tr>\n<td>luke-large</td>\n<td>1</td>\n<td>0.47148</td>\n<td>0.471</td>\n<td>0.465</td>\n</tr>\n<tr>\n<td>funnel-large-base</td>\n<td>1</td>\n<td>0.47117</td>\n<td>0.469</td>\n<td>0.468</td>\n</tr>\n</tbody>\n</table>\n<p>Could not make other models work, and could not get a score below 0.5 even for fold 0.</p>\n<h1>Ensemble</h1>\n<p>Simple <code>BayesianRidge</code> regression. Best score: CV is 0.447 and Public LB 0.451 Private LB 0.452.</p>\n<p>Thanks!</p>",
      "rawMarkdown": "Firstly, thanks to the hosts and Kaggle for hosting this interesting competition. Also, great thanks to torch( @rhtsingh ) for share his/her experience, which helps me break 0.485 and could get better score.\n\n# Summery\nMy final model consist of 2 kind of model and totally 17 models.\n1. single model: **Input**: encoded row text. **Model**: roberta/deberta/bart last_hidden_states with mean pooling and attention head.\n2. pair model: **Input**: baseline encoded text and encoded row text. **Model**: same architecture for baseline text and row text, then concatenate (or minus).\n\n*As hosts say, every score is a compare of one text to baseline text.* [here](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/236403)\n\n# Fine-tune\nMy final models are fine-tuned with almost same fine-tune strategy. That is:\n\n- Epochs: 5 for base mode and 3 for large model\n- Batch Size: almost maximize the GPU memory. (4 - 24)\n- Gradient Accumulate: 1-2\n- Optimizer: AdamW + Lookahead\n- LR Scheduler: None for base model and cosine_schedule_with_warmup for large model (warmup steps > 0)\n- LLRD: use torch's shared settings and set base_lr=5e-5\n- Random initial top layers: 1-2 for base model and 4-6 for large model\n- Evaluating for every n steps: 10 to 60 steps\n\n# Scoring\nJust report my best score of the same base transformers.\n| Model | Pair(1/0) |  CV  | Public | Private |\n| --- | --- | --- | --- | --- | \n| Roberta-large | 0 |  0.47835 |   0.466 | 0.470 |\n| Roberta-large | 1 | 0.46694  |   0.460 | 0.459 |\n| Roberta-base | 1 |  0.47325 | 0.475  | 0.477 |\n| Deberta-large | 0 | 0.46440  |  0.460 | 0.461 |\n| Deberta-large | 1 |  0.46673 |  0.462 | 0.464 |\n| Bart-large | 0 | 0.46184  | 0.462  | 0.464 |\n| Bart-large | 1 |  0.46401 | 0.463  | 0.463 |\n| Bart-base| 1 | 0.47103  | 0.470  | 0.472 |\n| Xlnet-large-cased | 1 | 0.47427  |  0.475 | 0.465 |\n| Electra-large | 1 | 0.47100  | 0.467  | 0.472 |\n| xlm-roberta-large | 1 | 0.48622  |  0.477 | 0.478 |\n| luke-large | 1 | 0.47148  | 0.471| 0.465 |\n| funnel-large-base | 1 | 0.47117  |  0.469 | 0.468 |\n\nCould not make other models work, and could not get a score below 0.5 even for fold 0.\n\n# Ensemble\nSimple `BayesianRidge` regression. Best score: CV is 0.447 and Public LB 0.451 Private LB 0.452.\n\nThanks!",
      "votes": 18
    },
    {
      "id": 1443933,
      "postDate": "2021-08-04T04:33:49.410Z",
      "content": "<p>Many congratulations on your solo silver and thanks <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a> for sharing your solution. </p>\n<p>I learned a lot from our discussions during the competition and it was really amazing watching you consistently climbing the leaderboard. Great effort and well deserved! 😊</p>",
      "rawMarkdown": "Many congratulations on your solo silver and thanks @syhens for sharing your solution. \n\nI learned a lot from our discussions during the competition and it was really amazing watching you consistently climbing the leaderboard. Great effort and well deserved! 😊",
      "votes": 1,
      "replies": [
        {
          "id": 1450621,
          "postDate": "2021-08-05T06:29:04.790Z",
          "content": "<p>Without your help, I could not get these improvement. Really thanks! Hope you to get your notebook GM someday!</p>",
          "rawMarkdown": "Without your help, I could not get these improvement. Really thanks! Hope you to get your notebook GM someday!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1430092,
      "postDate": "2021-08-03T11:17:25.647Z",
      "content": "<p>Great Job. Thanks for sharing!</p>",
      "rawMarkdown": "Great Job. Thanks for sharing!",
      "replies": [
        {
          "id": 1450622,
          "postDate": "2021-08-05T06:29:28.230Z",
          "content": "<p>You are welcome！</p>",
          "rawMarkdown": "You are welcome！"
        }
      ]
    },
    {
      "id": 1425325,
      "postDate": "2021-08-03T06:45:30.940Z",
      "content": "<p>Thanks for sharing! You did the good job!   We also train Bart-large, but it was only 0.48x lb. </p>",
      "rawMarkdown": "Thanks for sharing! You did the good job!   We also train Bart-large, but it was only 0.48x lb. ",
      "replies": [
        {
          "id": 1426361,
          "postDate": "2021-08-03T07:30:14.150Z",
          "content": "<p>Thank you! Congratulations on winning another gold medal!</p>",
          "rawMarkdown": "Thank you! Congratulations on winning another gold medal!"
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:45:28Z",
  "competition_id": 25914,
  "competition_slug": "commonlitreadabilityprize",
  "team_id": 6734669,
  "solution_url": "/competitions/commonlitreadabilityprize/writeups/yao-24-solution"
}