{
  "id": 446534,
  "title": "7th Solution",
  "content": "<p>First of all, thanks to competition organizers for hosting this interesting competition and my teamate <a href=\"https://www.kaggle.com/emiria\" target=\"_blank\">@emiria</a>. I learn a lot from emiria's ideas and code. Congrats for emiria's fourth gold medal, a new GM is on the way. And this is my first gold medal of nlp competition.</p>\n<p>The <strong>key points</strong> of our strategy:</p>\n<ul>\n<li>Add \"prompt_text\" for inputs (0.03 boost on cv)</li>\n<li>Freezing layers (0.01 boost on cv)</li>\n<li>Different inputs for blending(0.01 boost on cv)</li>\n<li>Blending with result of lightgbm(0.005 boost on cv)</li>\n<li>Focus on local cv (LB only has 13% of data)</li>\n</ul>\n<p><strong>Did not work</strong> for us:</p>\n<ul>\n<li>AWP</li>\n<li>SWA</li>\n<li>Text preprocess</li>\n<li>MLM</li>\n</ul>\n<h3>Models</h3>\n<p>Here's the discription of our models we used for finall submissions. We use groupkfold with \"prompt_id\" for local validation, and used all prompts for training when inference.</p>\n<table>\n<thead>\n<tr>\n<th>id</th>\n<th>backbone</th>\n<th>inputs</th>\n<th>maxlen</th>\n<th>loss</th>\n<th>cv</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>model1</td>\n<td>deberta-v3-large</td>\n<td>text+sep+prompt_text+sep+prompt_question</td>\n<td>1280</td>\n<td>mseloss</td>\n<td>0.500</td>\n</tr>\n<tr>\n<td>model2</td>\n<td>deberta-v3-large</td>\n<td>text+sep+prompt_title+sep+prompt_question+sep+prompt_text</td>\n<td>1280</td>\n<td>mseloss</td>\n<td>0.489</td>\n</tr>\n<tr>\n<td>model3</td>\n<td>deberta-v3-large</td>\n<td>prompt_title+sep+prompt_question+sep+text+sep+prompt_text</td>\n<td>1280</td>\n<td>mseloss</td>\n<td>0.506</td>\n</tr>\n<tr>\n<td>model4</td>\n<td>deberta-v3-large+lgb</td>\n<td>prompt_question+sep+text</td>\n<td>512</td>\n<td>mseloss</td>\n<td>0.520</td>\n</tr>\n<tr>\n<td>model5</td>\n<td>deberta-v3-large</td>\n<td>text+sep+prompt_title+sep+prompt_question+sep+prompt_text</td>\n<td>768</td>\n<td>mseloss</td>\n<td>-</td>\n</tr>\n<tr>\n<td>model6</td>\n<td>deberta-v3-large</td>\n<td>text+sep+prompt_title+sep+prompt_question+sep+prompt_text</td>\n<td>768</td>\n<td>logloss</td>\n<td>-</td>\n</tr>\n<tr>\n<td>model7</td>\n<td>deberta-large</td>\n<td>text+sep+prompt_title+sep+prompt_question+sep+prompt_text</td>\n<td>1024</td>\n<td>mseloss</td>\n<td>-</td>\n</tr>\n</tbody>\n</table>\n<h3>Results</h3>\n<p>And here's our models with best scores:<br>\nEach model is average of 2 seeds, except the \"model4\"(with lightgbm).</p>\n<table>\n<thead>\n<tr>\n<th>PB</th>\n<th>LB</th>\n<th>Picked</th>\n<th>models</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.456</td>\n<td>0.427</td>\n<td>Yes</td>\n<td>0.32*model1+0.32*model2+0.16*model3+0.2*model7</td>\n</tr>\n<tr>\n<td>0.453</td>\n<td>0.428</td>\n<td>No</td>\n<td>0.32*model1+0.32*model2+0.16*model4+0.1*model5+0.1*model6</td>\n</tr>\n</tbody>\n</table>",
  "comments": [
    {
      "id": 2478790,
      "author_name": "anji",
      "author_url": "",
      "post_date": "2023-10-12T08:02:52.577000",
      "content": "<p>Congrats on the gold medal, I notice that you ensemble four models at the end, 3 of which has a max len of 1200, I'd like to ask how you managed to compress the inference time to under 9 hours,May I inquire about the acceleration strategies you have employed during the inference? thanks</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478925,
          "author_name": "Ethan",
          "author_url": "",
          "post_date": "2023-10-12T09:29:21.517000",
          "content": "<p><a href=\"https://www.kaggle.com/zhangyaji\" target=\"_blank\">@zhangyaji</a> <br>\nMaybe three points:</p>\n<ul>\n<li><p>sort the data by \"token_length\"<br>\n<code>test_df['full_text'] = test_df['text'] + SEP + test_df['prompt_text'] + SEP + test_df['prompt_question']</code><br>\n<code>test_df['tokenize_length'] = [len(config.tokenizer(text)['input_ids']) for text in test_df['full_text'].values]</code><br>\n<code>test_df = test_df.sort_values('tokenize_length', ascending=True).reset_index(drop=True)</code></p></li>\n<li><p>use amp<br>\n<code>with torch.cuda.amp.autocast(enabled=True)</code></p></li>\n<li><p>DataCollatorWithPadding</p></li>\n</ul>\n<p>Actually, we use higher maxlen when inference(model1-3 used maxlen=1400). And each model is average of 2 seeds, so we got 8 models for blending.</p>",
          "votes": 2,
          "replies": [
            {
              "id": 2478953,
              "author_name": "anji",
              "author_url": "",
              "post_date": "2023-10-12T09:54:30.423000",
              "content": "<p>Thank you so much for your valuable response～</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2481315,
      "author_name": "DL",
      "author_url": "",
      "post_date": "2023-10-14T03:19:06.157000",
      "content": "<p>thanks for sharing ,in the end you use 0.32<em>model1+0.32</em>model2+0.16<em>model3+0.2</em>model7,can you explain how to select the The weight of the model.looking forward to your answer</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2479148,
      "author_name": "Haru",
      "author_url": "",
      "post_date": "2023-10-12T12:56:50.037000",
      "content": "<p>Congratulations on the gold Medal, and Thank you for your solution!<br>\nCan I ask PL of each model ? (model 1~7) <br>\nThank you first.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 5,
  "messages": [
    {
      "id": 2478603,
      "postDate": "2023-10-12T05:16:21.043Z",
      "content": "<p>First of all, thanks to competition organizers for hosting this interesting competition and my teamate <a href=\"https://www.kaggle.com/emiria\" target=\"_blank\">@emiria</a>. I learn a lot from emiria's ideas and code. Congrats for emiria's fourth gold medal, a new GM is on the way. And this is my first gold medal of nlp competition.</p>\n<p>The <strong>key points</strong> of our strategy:</p>\n<ul>\n<li>Add \"prompt_text\" for inputs (0.03 boost on cv)</li>\n<li>Freezing layers (0.01 boost on cv)</li>\n<li>Different inputs for blending(0.01 boost on cv)</li>\n<li>Blending with result of lightgbm(0.005 boost on cv)</li>\n<li>Focus on local cv (LB only has 13% of data)</li>\n</ul>\n<p><strong>Did not work</strong> for us:</p>\n<ul>\n<li>AWP</li>\n<li>SWA</li>\n<li>Text preprocess</li>\n<li>MLM</li>\n</ul>\n<h3>Models</h3>\n<p>Here's the discription of our models we used for finall submissions. We use groupkfold with \"prompt_id\" for local validation, and used all prompts for training when inference.</p>\n<table>\n<thead>\n<tr>\n<th>id</th>\n<th>backbone</th>\n<th>inputs</th>\n<th>maxlen</th>\n<th>loss</th>\n<th>cv</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>model1</td>\n<td>deberta-v3-large</td>\n<td>text+sep+prompt_text+sep+prompt_question</td>\n<td>1280</td>\n<td>mseloss</td>\n<td>0.500</td>\n</tr>\n<tr>\n<td>model2</td>\n<td>deberta-v3-large</td>\n<td>text+sep+prompt_title+sep+prompt_question+sep+prompt_text</td>\n<td>1280</td>\n<td>mseloss</td>\n<td>0.489</td>\n</tr>\n<tr>\n<td>model3</td>\n<td>deberta-v3-large</td>\n<td>prompt_title+sep+prompt_question+sep+text+sep+prompt_text</td>\n<td>1280</td>\n<td>mseloss</td>\n<td>0.506</td>\n</tr>\n<tr>\n<td>model4</td>\n<td>deberta-v3-large+lgb</td>\n<td>prompt_question+sep+text</td>\n<td>512</td>\n<td>mseloss</td>\n<td>0.520</td>\n</tr>\n<tr>\n<td>model5</td>\n<td>deberta-v3-large</td>\n<td>text+sep+prompt_title+sep+prompt_question+sep+prompt_text</td>\n<td>768</td>\n<td>mseloss</td>\n<td>-</td>\n</tr>\n<tr>\n<td>model6</td>\n<td>deberta-v3-large</td>\n<td>text+sep+prompt_title+sep+prompt_question+sep+prompt_text</td>\n<td>768</td>\n<td>logloss</td>\n<td>-</td>\n</tr>\n<tr>\n<td>model7</td>\n<td>deberta-large</td>\n<td>text+sep+prompt_title+sep+prompt_question+sep+prompt_text</td>\n<td>1024</td>\n<td>mseloss</td>\n<td>-</td>\n</tr>\n</tbody>\n</table>\n<h3>Results</h3>\n<p>And here's our models with best scores:<br>\nEach model is average of 2 seeds, except the \"model4\"(with lightgbm).</p>\n<table>\n<thead>\n<tr>\n<th>PB</th>\n<th>LB</th>\n<th>Picked</th>\n<th>models</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.456</td>\n<td>0.427</td>\n<td>Yes</td>\n<td>0.32*model1+0.32*model2+0.16*model3+0.2*model7</td>\n</tr>\n<tr>\n<td>0.453</td>\n<td>0.428</td>\n<td>No</td>\n<td>0.32*model1+0.32*model2+0.16*model4+0.1*model5+0.1*model6</td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "First of all, thanks to competition organizers for hosting this interesting competition and my teamate @emiria. I learn a lot from emiria's ideas and code. Congrats for emiria's fourth gold medal, a new GM is on the way. And this is my first gold medal of nlp competition.\n\nThe **key points** of our strategy:\n- Add \"prompt_text\" for inputs (0.03 boost on cv)\n- Freezing layers (0.01 boost on cv)\n- Different inputs for blending(0.01 boost on cv)\n- Blending with result of lightgbm(0.005 boost on cv)\n- Focus on local cv (LB only has 13% of data)\n\n**Did not work** for us:\n- AWP\n- SWA\n- Text preprocess\n- MLM\n\n### Models \nHere's the discription of our models we used for finall submissions. We use groupkfold with \"prompt_id\" for local validation, and used all prompts for training when inference.\n| id| backbone | inputs | maxlen| loss | cv |\n| --- | --- |\n| model1 | deberta-v3-large | text+sep+prompt_text+sep+prompt_question| 1280 | mseloss | 0.500|\n| model2 | deberta-v3-large | text+sep+prompt_title+sep+prompt_question+sep+prompt_text| 1280 | mseloss | 0.489|\n| model3 | deberta-v3-large | prompt_title+sep+prompt_question+sep+text+sep+prompt_text| 1280 | mseloss | 0.506|\n| model4 | deberta-v3-large+lgb | prompt_question+sep+text| 512 | mseloss | 0.520|\n| model5 | deberta-v3-large | text+sep+prompt_title+sep+prompt_question+sep+prompt_text| 768 | mseloss | -|\n| model6 | deberta-v3-large | text+sep+prompt_title+sep+prompt_question+sep+prompt_text| 768 | logloss | -|\n| model7 | deberta-large | text+sep+prompt_title+sep+prompt_question+sep+prompt_text| 1024 | mseloss | -|\n\n\n### Results\nAnd here's our models with best scores:\nEach model is average of 2 seeds, except the \"model4\"(with lightgbm).\n| PB | LB | Picked | models |\n| --- | --- | ---\n| 0.456 | 0.427 | Yes | 0.32\\*model1+0.32\\*model2+0.16\\*model3+0.2\\*model7 |\n| 0.453 | 0.428 | No | 0.32\\*model1+0.32\\*model2+0.16\\*model4+0.1\\*model5+0.1\\*model6|\n\n\n\n",
      "votes": 38
    },
    {
      "id": 2478790,
      "postDate": "2023-10-12T08:02:52.577Z",
      "content": "<p>Congrats on the gold medal, I notice that you ensemble four models at the end, 3 of which has a max len of 1200, I'd like to ask how you managed to compress the inference time to under 9 hours,May I inquire about the acceleration strategies you have employed during the inference? thanks</p>",
      "rawMarkdown": "Congrats on the gold medal, I notice that you ensemble four models at the end, 3 of which has a max len of 1200, I'd like to ask how you managed to compress the inference time to under 9 hours,May I inquire about the acceleration strategies you have employed during the inference? thanks",
      "votes": 1,
      "replies": [
        {
          "id": 2478925,
          "postDate": "2023-10-12T09:29:21.517Z",
          "content": "<p><a href=\"https://www.kaggle.com/zhangyaji\" target=\"_blank\">@zhangyaji</a> <br>\nMaybe three points:</p>\n<ul>\n<li><p>sort the data by \"token_length\"<br>\n<code>test_df['full_text'] = test_df['text'] + SEP + test_df['prompt_text'] + SEP + test_df['prompt_question']</code><br>\n<code>test_df['tokenize_length'] = [len(config.tokenizer(text)['input_ids']) for text in test_df['full_text'].values]</code><br>\n<code>test_df = test_df.sort_values('tokenize_length', ascending=True).reset_index(drop=True)</code></p></li>\n<li><p>use amp<br>\n<code>with torch.cuda.amp.autocast(enabled=True)</code></p></li>\n<li><p>DataCollatorWithPadding</p></li>\n</ul>\n<p>Actually, we use higher maxlen when inference(model1-3 used maxlen=1400). And each model is average of 2 seeds, so we got 8 models for blending.</p>",
          "rawMarkdown": "@zhangyaji \nMaybe three points:\n- sort the data by \"token_length\"\n`test_df['full_text'] = test_df['text'] + SEP + test_df['prompt_text'] + SEP + test_df['prompt_question']`\n`test_df['tokenize_length'] = [len(config.tokenizer(text)['input_ids']) for text in test_df['full_text'].values]`\n`test_df = test_df.sort_values('tokenize_length', ascending=True).reset_index(drop=True)`\n- use amp\n`with torch.cuda.amp.autocast(enabled=True)`\n\n- DataCollatorWithPadding\n\nActually, we use higher maxlen when inference(model1-3 used maxlen=1400). And each model is average of 2 seeds, so we got 8 models for blending.",
          "votes": 2,
          "replies": [
            {
              "id": 2478953,
              "postDate": "2023-10-12T09:54:30.423Z",
              "content": "<p>Thank you so much for your valuable response～</p>",
              "rawMarkdown": "Thank you so much for your valuable response～"
            }
          ]
        }
      ]
    },
    {
      "id": 2481315,
      "postDate": "2023-10-14T03:19:06.157Z",
      "content": "<p>thanks for sharing ,in the end you use 0.32<em>model1+0.32</em>model2+0.16<em>model3+0.2</em>model7,can you explain how to select the The weight of the model.looking forward to your answer</p>",
      "rawMarkdown": "thanks for sharing ,in the end you use 0.32*model1+0.32*model2+0.16*model3+0.2*model7,can you explain how to select the The weight of the model.looking forward to your answer"
    },
    {
      "id": 2479148,
      "postDate": "2023-10-12T12:56:50.037Z",
      "content": "<p>Congratulations on the gold Medal, and Thank you for your solution!<br>\nCan I ask PL of each model ? (model 1~7) <br>\nThank you first.</p>",
      "rawMarkdown": "Congratulations on the gold Medal, and Thank you for your solution!\nCan I ask PL of each model ? (model 1~7) \nThank you first."
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:01:41Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10708302,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/ethan-ria-7th-solution"
}