{
  "id": 446644,
  "title": "43th place solution (711 shake up)",
  "content": "<p>Thank you for organizers hosting the competition. I also appreciate everyone at Kaggle sharing insights through codes and discussions.</p>\n<p>The reason why I was at very low rank (754 th) is that I didn't have enough time to improve public leaderboard score due to severe fever from 10/3 to the competition end.<br>\n(I have planed to reach similar CV score reported in <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/424330\" target=\"_blank\">Single Model CV-LB discussion</a> first, then try to improve CV and public leaderboard score. But there wasn't enough time to do later.)</p>\n<h1>Overview of the approach</h1>\n<p>My solution is mean ensemble of two deberta-v3-large with different settings. Each model has 4 weights (4 folds), so the total number of weights for inference is 8. I have used best CV checkpoints.<br>\nCV setup is Group K fold by prompt_id and CV scores of each model are 0.482 and 0.492.</p>\n<h1>Details of submission</h1>\n<p>Most influential part for CV score improvement is pooling inside student summary. Other things bring small improvements compared with that. Details are written in the following sections.</p>\n<h2>Model input</h2>\n<p>Each model has different model input.<br>\nOne is <code>[START] text [END] prompt_question [SEP] prompt_text</code>.<br>\nAnother is <code>[START_QUESTION] prompt_question [END_QUESTION] [START_SUMMARY] text [END_SUMMARY] [SEP] [START_TITLE] prompt_title [END_TITLE] [START_PROMPT] prompt_text [END_PROMPT]</code>.</p>\n<h2>Model architecture</h2>\n<p>Each model has different pooling.<br>\nOne is mean pooling inside <code>[START]</code> and <code>[END]</code> tokens which means pool inside student summary.<br>\nAnother is GeM pooling inside <code>[CLS]</code> <code>[SEP]</code> tokens which means pool inside prompt question and student summary.</p>\n<p>Second model has separate heads to predict content and wording scores.</p>\n<h2>Other settings</h2>\n<h3>Same settings across models</h3>\n<ul>\n<li>Gradient clipping = 10.</li>\n<li>Set <code>hidden_dropout_prob</code> and <code>attention_probs_dropout_prob</code> are 0.</li>\n<li>Tokenizer max length is 1024.</li>\n</ul>\n<h3>Different settings across models</h3>\n<ul>\n<li>The frequency of CV computation to save best checkpoints during training is 100 and 300.</li>\n<li>Second model has different learning rate for heads.</li>\n</ul>\n<h1>Things not worked</h1>\n<p>I have guessed most important thing of this competition is combining student summary and prompt text effectively. Most of time I have focused this direction but none of them bring significant improvements (They bring similar CV score of final models but even ensemble of them didn't give huge improvements).</p>\n<ul>\n<li>After passing summary and prompt_text to backbone, pool summary and prompt_text then take bert cross attention of them<ul>\n<li>I have also tried to stack bert cross attention and bert layer multiple times like <a href=\"https://arxiv.org/abs/2112.03857\" target=\"_blank\">this paper</a>, but not worked</li></ul></li>\n<li>Pass summary and prompt text separately to the backbone, then concatenate those hidden states and pass to the head</li>\n<li>re initialization of layers</li>\n<li>freezing layers</li>\n<li>ensemble with other backbones (debrta-large, longformer)</li>\n<li>pretty long max length (4096)</li>\n</ul>",
  "comments": [
    {
      "id": 2480015,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-10-13T04:46:40.300000",
      "content": "<p><a href=\"https://www.kaggle.com/shutotakahashi\" target=\"_blank\">@shutotakahashi</a> great work. Hope you are feeling better now.</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 1,
  "messages": [
    {
      "id": 2479189,
      "postDate": "2023-10-12T13:39:38.860Z",
      "content": "<p>Thank you for organizers hosting the competition. I also appreciate everyone at Kaggle sharing insights through codes and discussions.</p>\n<p>The reason why I was at very low rank (754 th) is that I didn't have enough time to improve public leaderboard score due to severe fever from 10/3 to the competition end.<br>\n(I have planed to reach similar CV score reported in <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/424330\" target=\"_blank\">Single Model CV-LB discussion</a> first, then try to improve CV and public leaderboard score. But there wasn't enough time to do later.)</p>\n<h1>Overview of the approach</h1>\n<p>My solution is mean ensemble of two deberta-v3-large with different settings. Each model has 4 weights (4 folds), so the total number of weights for inference is 8. I have used best CV checkpoints.<br>\nCV setup is Group K fold by prompt_id and CV scores of each model are 0.482 and 0.492.</p>\n<h1>Details of submission</h1>\n<p>Most influential part for CV score improvement is pooling inside student summary. Other things bring small improvements compared with that. Details are written in the following sections.</p>\n<h2>Model input</h2>\n<p>Each model has different model input.<br>\nOne is <code>[START] text [END] prompt_question [SEP] prompt_text</code>.<br>\nAnother is <code>[START_QUESTION] prompt_question [END_QUESTION] [START_SUMMARY] text [END_SUMMARY] [SEP] [START_TITLE] prompt_title [END_TITLE] [START_PROMPT] prompt_text [END_PROMPT]</code>.</p>\n<h2>Model architecture</h2>\n<p>Each model has different pooling.<br>\nOne is mean pooling inside <code>[START]</code> and <code>[END]</code> tokens which means pool inside student summary.<br>\nAnother is GeM pooling inside <code>[CLS]</code> <code>[SEP]</code> tokens which means pool inside prompt question and student summary.</p>\n<p>Second model has separate heads to predict content and wording scores.</p>\n<h2>Other settings</h2>\n<h3>Same settings across models</h3>\n<ul>\n<li>Gradient clipping = 10.</li>\n<li>Set <code>hidden_dropout_prob</code> and <code>attention_probs_dropout_prob</code> are 0.</li>\n<li>Tokenizer max length is 1024.</li>\n</ul>\n<h3>Different settings across models</h3>\n<ul>\n<li>The frequency of CV computation to save best checkpoints during training is 100 and 300.</li>\n<li>Second model has different learning rate for heads.</li>\n</ul>\n<h1>Things not worked</h1>\n<p>I have guessed most important thing of this competition is combining student summary and prompt text effectively. Most of time I have focused this direction but none of them bring significant improvements (They bring similar CV score of final models but even ensemble of them didn't give huge improvements).</p>\n<ul>\n<li>After passing summary and prompt_text to backbone, pool summary and prompt_text then take bert cross attention of them<ul>\n<li>I have also tried to stack bert cross attention and bert layer multiple times like <a href=\"https://arxiv.org/abs/2112.03857\" target=\"_blank\">this paper</a>, but not worked</li></ul></li>\n<li>Pass summary and prompt text separately to the backbone, then concatenate those hidden states and pass to the head</li>\n<li>re initialization of layers</li>\n<li>freezing layers</li>\n<li>ensemble with other backbones (debrta-large, longformer)</li>\n<li>pretty long max length (4096)</li>\n</ul>",
      "rawMarkdown": "Thank you for organizers hosting the competition. I also appreciate everyone at Kaggle sharing insights through codes and discussions.\n\nThe reason why I was at very low rank (754 th) is that I didn't have enough time to improve public leaderboard score due to severe fever from 10/3 to the competition end.\n(I have planed to reach similar CV score reported in [Single Model CV-LB discussion](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/424330) first, then try to improve CV and public leaderboard score. But there wasn't enough time to do later.)\n\n# Overview of the approach\n\nMy solution is mean ensemble of two deberta-v3-large with different settings. Each model has 4 weights (4 folds), so the total number of weights for inference is 8. I have used best CV checkpoints.\nCV setup is Group K fold by prompt_id and CV scores of each model are 0.482 and 0.492.\n\n# Details of submission\n\nMost influential part for CV score improvement is pooling inside student summary. Other things bring small improvements compared with that. Details are written in the following sections.\n\n## Model input\n\nEach model has different model input.\nOne is `[START] text [END] prompt_question [SEP] prompt_text`.\nAnother is `[START_QUESTION] prompt_question [END_QUESTION] [START_SUMMARY] text [END_SUMMARY] [SEP] [START_TITLE] prompt_title [END_TITLE] [START_PROMPT] prompt_text [END_PROMPT]`.\n\n\n## Model architecture\n\nEach model has different pooling.\nOne is mean pooling inside `[START]` and `[END]` tokens which means pool inside student summary.\nAnother is GeM pooling inside `[CLS]` `[SEP]` tokens which means pool inside prompt question and student summary.\n\nSecond model has separate heads to predict content and wording scores.\n\n## Other settings\n\n### Same settings across models\n\n- Gradient clipping = 10.\n- Set `hidden_dropout_prob` and `attention_probs_dropout_prob` are 0.\n- Tokenizer max length is 1024.\n\n### Different settings across models\n\n- The frequency of CV computation to save best checkpoints during training is 100 and 300.\n- Second model has different learning rate for heads.\n\n# Things not worked\n\nI have guessed most important thing of this competition is combining student summary and prompt text effectively. Most of time I have focused this direction but none of them bring significant improvements (They bring similar CV score of final models but even ensemble of them didn't give huge improvements).\n\n- After passing summary and prompt_text to backbone, pool summary and prompt_text then take bert cross attention of them\n    - I have also tried to stack bert cross attention and bert layer multiple times like [this paper](https://arxiv.org/abs/2112.03857), but not worked\n- Pass summary and prompt text separately to the backbone, then concatenate those hidden states and pass to the head\n- re initialization of layers\n- freezing layers\n- ensemble with other backbones (debrta-large, longformer)\n- pretty long max length (4096)",
      "votes": 7
    },
    {
      "id": 2480015,
      "postDate": "2023-10-13T04:46:40.300Z",
      "content": "<p><a href=\"https://www.kaggle.com/shutotakahashi\" target=\"_blank\">@shutotakahashi</a> great work. Hope you are feeling better now.</p>",
      "rawMarkdown": "@shutotakahashi great work. Hope you are feeling better now.\n",
      "votes": 1,
      "isDeleted": true
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:03:31Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10798374,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/stakahashi-43th-place-solution-711-shake-up"
}