{
  "id": 446584,
  "title": "5th place solution",
  "content": "<p>First of all, I would like to thank the host for hosting such a wonderful competition.Also thanks to kagglers for sharing valuable information in notebooks and discussions.</p>\n<p>I'll briefly share my approach that led to the 5th place finish. Honestly, I didn't use any special techniques, so I believe the shake-up was largely due to luck. However, I'm extremely pleased with my first solo gold.</p>\n<h3>Overview</h3>\n<p>The key points of my approach are mainly threefold.</p>\n<ul>\n<li>Use all of summary, question, title, and prompt_text.</li>\n<li>Use a single deberta v3 large model, improved with longer max_len and freeze layer, etc.</li>\n<li>An ensemble of deberta v3 large and LightGBM.</li>\n</ul>\n<h3>Approach</h3>\n<ul>\n<li><p>deberta v3 large</p>\n<ul>\n<li>Input all of summary + question + title + prompt_text.</li>\n<li>No preprocessing.</li>\n<li>max_len: 1536.</li>\n<li>Freeze embedding and the first 18 layers. </li>\n<li>No dropout.</li>\n<li>Train content and wording simultaneously.</li></ul></li>\n<li><p>LightGBM</p>\n<ul>\n<li>Uses mostly the same features as public notebooks.</li></ul></li>\n<li><p>ensemble</p>\n<ul>\n<li>Weighted average of deberta and LGB.</li>\n<li>Weights optimized using nelder mead.</li></ul></li>\n</ul>\n<h3>validation</h3>\n<ul>\n<li>strategy<ul>\n<li>groupkfold (group = prompt_id)</li></ul></li>\n<li>cv score<ul>\n<li>deberta v3 large :0.4816</li>\n<li>LGBM :  0.5513</li>\n<li>ensemble : 0.4748</li></ul></li>\n</ul>\n<h3>What Didn't Work</h3>\n<ul>\n<li><p>Models other than deberta v3 large.</p>\n<ul>\n<li>When including prompt_text as input, other models (including deberta-v3-base) had much poorer performance.</li></ul></li>\n<li><p>Additional training data with LLM</p>\n<ul>\n<li>Considered data augmentation by creating summaries from prompt_texts using LLM.</li>\n<li>I used the scraped prompt_text from commonlit.</li>\n<li>Created ground truth using pseudo labels and added to training, but it did not improve.</li></ul></li>\n<li><p>LGB-based stacking</p>\n<ul>\n<li>Tried stacking using deberta's oof, similar to public notebooks, but it did not improve in my model.</li>\n<li>Hence, changed to the weighted average of each model as mentioned above.</li></ul></li>\n<li><p>Text preprocessing.</p></li>\n<li><p>Inputting other summaries from the same prompt.</p>\n<ul>\n<li>Inspired by the PPPM competition's magic, considered concatenating other summaries from the same prompt, but it did not improve.</li></ul></li>\n<li><p>etc…</p></li>\n</ul>",
  "comments": [
    {
      "id": 2479448,
      "author_name": "Mingxian Gao",
      "author_url": "",
      "post_date": "2023-10-12T16:12:36.357000",
      "content": "<p>I see that when i try to implement the deberta large model, there is a error say that out of memory, could you please give an indication of how to fix this problem. Thank you!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2479876,
          "author_name": "monnu",
          "author_url": "",
          "post_date": "2023-10-13T01:04:30.287000",
          "content": "<p>I reduced memory usage by applying the following three things. </p>\n<ul>\n<li>Freeze embedding and the first 18 layers</li>\n<li>fp16</li>\n<li>gradient_checkpointing</li>\n</ul>\n<p>This allows deberta-v3-large(max_len :1536, batch_size 8) to train with 16Gib of GPU memory.</p>",
          "votes": 6,
          "replies": [
            {
              "id": 2481560,
              "author_name": "XinYiLea",
              "author_url": "",
              "post_date": "2023-10-14T08:17:59.803000",
              "content": "<p>This is also the problem that troubles me ,and thank you for your answer.</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2480725,
      "author_name": "Takashi Someya",
      "author_url": "",
      "post_date": "2023-10-13T14:09:40.030000",
      "content": "<p>Congrats on your solo gold!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2480954,
          "author_name": "monnu",
          "author_url": "",
          "post_date": "2023-10-13T16:53:53.297000",
          "content": "<p>Thank you! And congratulations on winning the gold medal in the intense LLM competition!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2480434,
      "author_name": "Xiang Huang",
      "author_url": "",
      "post_date": "2023-10-13T10:06:51.650000",
      "content": "<p>Very good solution,  Congrats and thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2478883,
      "author_name": "chumajin",
      "author_url": "",
      "post_date": "2023-10-12T08:51:41.533000",
      "content": "<p>Congratulations on your powerful solo gold!<br>\nThe single model in deberta-v3-large is stronger and better than us.<br>\nThis was impressive.</p>\n<blockquote>\n  <p>Inspired by the PPPM competition's magic, considered concatenating other summaries from the same prompt, but it did not improve.</p>\n</blockquote>\n<p>I did this too and it was not good at all. lol</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478929,
          "author_name": "monnu",
          "author_url": "",
          "post_date": "2023-10-12T09:33:11.997000",
          "content": "<p>Thank you for your comment.<br>\nAnd congrats to you on 4th place and Grand Master !</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2481385,
      "author_name": "Bien Nguyen",
      "author_url": "",
      "post_date": "2023-10-14T04:51:18.867000",
      "content": "<p>Congrats on your impressive solution, I wonder how can you choose the max_len is 1536.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2479937,
      "author_name": "Cotmd",
      "author_url": "",
      "post_date": "2023-10-13T02:37:26.070000",
      "content": "<p>Congrats!! Amazing solo gold<br>\nI have a question here!</p>\n<p><code>Freeze embedding and the first 18 layers</code><br>\n-&gt;How many epochs trained?</p>\n<p>Thanks for providing this impressive and cool solution</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2480081,
          "author_name": "monnu",
          "author_url": "",
          "post_date": "2023-10-13T05:40:41.400000",
          "content": "<p>Thanks you! I trained 4 epochs. </p>",
          "votes": 2,
          "replies": [
            {
              "id": 2481007,
              "author_name": "Cotmd",
              "author_url": "",
              "post_date": "2023-10-13T17:42:01.880000",
              "content": "<p>That was a good idea. I've never thought about freezing more than half of the layers😅<br>\nThx!</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "comment_count": 12,
  "messages": [
    {
      "id": 2478855,
      "postDate": "2023-10-12T08:33:31.773Z",
      "content": "<p>First of all, I would like to thank the host for hosting such a wonderful competition.Also thanks to kagglers for sharing valuable information in notebooks and discussions.</p>\n<p>I'll briefly share my approach that led to the 5th place finish. Honestly, I didn't use any special techniques, so I believe the shake-up was largely due to luck. However, I'm extremely pleased with my first solo gold.</p>\n<h3>Overview</h3>\n<p>The key points of my approach are mainly threefold.</p>\n<ul>\n<li>Use all of summary, question, title, and prompt_text.</li>\n<li>Use a single deberta v3 large model, improved with longer max_len and freeze layer, etc.</li>\n<li>An ensemble of deberta v3 large and LightGBM.</li>\n</ul>\n<h3>Approach</h3>\n<ul>\n<li><p>deberta v3 large</p>\n<ul>\n<li>Input all of summary + question + title + prompt_text.</li>\n<li>No preprocessing.</li>\n<li>max_len: 1536.</li>\n<li>Freeze embedding and the first 18 layers. </li>\n<li>No dropout.</li>\n<li>Train content and wording simultaneously.</li></ul></li>\n<li><p>LightGBM</p>\n<ul>\n<li>Uses mostly the same features as public notebooks.</li></ul></li>\n<li><p>ensemble</p>\n<ul>\n<li>Weighted average of deberta and LGB.</li>\n<li>Weights optimized using nelder mead.</li></ul></li>\n</ul>\n<h3>validation</h3>\n<ul>\n<li>strategy<ul>\n<li>groupkfold (group = prompt_id)</li></ul></li>\n<li>cv score<ul>\n<li>deberta v3 large :0.4816</li>\n<li>LGBM :  0.5513</li>\n<li>ensemble : 0.4748</li></ul></li>\n</ul>\n<h3>What Didn't Work</h3>\n<ul>\n<li><p>Models other than deberta v3 large.</p>\n<ul>\n<li>When including prompt_text as input, other models (including deberta-v3-base) had much poorer performance.</li></ul></li>\n<li><p>Additional training data with LLM</p>\n<ul>\n<li>Considered data augmentation by creating summaries from prompt_texts using LLM.</li>\n<li>I used the scraped prompt_text from commonlit.</li>\n<li>Created ground truth using pseudo labels and added to training, but it did not improve.</li></ul></li>\n<li><p>LGB-based stacking</p>\n<ul>\n<li>Tried stacking using deberta's oof, similar to public notebooks, but it did not improve in my model.</li>\n<li>Hence, changed to the weighted average of each model as mentioned above.</li></ul></li>\n<li><p>Text preprocessing.</p></li>\n<li><p>Inputting other summaries from the same prompt.</p>\n<ul>\n<li>Inspired by the PPPM competition's magic, considered concatenating other summaries from the same prompt, but it did not improve.</li></ul></li>\n<li><p>etc…</p></li>\n</ul>",
      "rawMarkdown": "First of all, I would like to thank the host for hosting such a wonderful competition.Also thanks to kagglers for sharing valuable information in notebooks and discussions.\n\nI'll briefly share my approach that led to the 5th place finish. Honestly, I didn't use any special techniques, so I believe the shake-up was largely due to luck. However, I'm extremely pleased with my first solo gold.\n\n### Overview\nThe key points of my approach are mainly threefold.\n* Use all of summary, question, title, and prompt_text.\n* Use a single deberta v3 large model, improved with longer max_len and freeze layer, etc.\n* An ensemble of deberta v3 large and LightGBM.\n\n### Approach\n* deberta v3 large\n  * Input all of summary + question + title + prompt_text.\n  * No preprocessing.\n  * max_len: 1536.\n  * Freeze embedding and the first 18 layers. \n  * No dropout.\n  * Train content and wording simultaneously.\n\n* LightGBM\n  * Uses mostly the same features as public notebooks.\n\n* ensemble\n  * Weighted average of deberta and LGB.\n  * Weights optimized using nelder mead.\n\n### validation\n* strategy\n  * groupkfold (group = prompt_id)\n* cv score\n  * deberta v3 large :0.4816\n  * LGBM :  0.5513\n  * ensemble : 0.4748\n\n\n### What Didn't Work\n\n* Models other than deberta v3 large.\n    * When including prompt_text as input, other models (including deberta-v3-base) had much poorer performance.\n\n* Additional training data with LLM\n  * Considered data augmentation by creating summaries from prompt_texts using LLM.\n  * I used the scraped prompt_text from commonlit.\n  * Created ground truth using pseudo labels and added to training, but it did not improve.\n\n* LGB-based stacking\n    * Tried stacking using deberta's oof, similar to public notebooks, but it did not improve in my model.\n    * Hence, changed to the weighted average of each model as mentioned above.\n\n* Text preprocessing.\n* Inputting other summaries from the same prompt.\n  * Inspired by the PPPM competition's magic, considered concatenating other summaries from the same prompt, but it did not improve.\n\n* etc...",
      "votes": 46
    },
    {
      "id": 2479448,
      "postDate": "2023-10-12T16:12:36.357Z",
      "content": "<p>I see that when i try to implement the deberta large model, there is a error say that out of memory, could you please give an indication of how to fix this problem. Thank you!</p>",
      "rawMarkdown": "I see that when i try to implement the deberta large model, there is a error say that out of memory, could you please give an indication of how to fix this problem. Thank you!",
      "votes": 2,
      "replies": [
        {
          "id": 2479876,
          "postDate": "2023-10-13T01:04:30.287Z",
          "content": "<p>I reduced memory usage by applying the following three things. </p>\n<ul>\n<li>Freeze embedding and the first 18 layers</li>\n<li>fp16</li>\n<li>gradient_checkpointing</li>\n</ul>\n<p>This allows deberta-v3-large(max_len :1536, batch_size 8) to train with 16Gib of GPU memory.</p>",
          "rawMarkdown": "I reduced memory usage by applying the following three things. \n* Freeze embedding and the first 18 layers\n* fp16\n* gradient_checkpointing\n\nThis allows deberta-v3-large(max_len :1536, batch_size 8) to train with 16Gib of GPU memory.",
          "votes": 6,
          "replies": [
            {
              "id": 2481560,
              "postDate": "2023-10-14T08:17:59.803Z",
              "content": "<p>This is also the problem that troubles me ,and thank you for your answer.</p>",
              "rawMarkdown": "This is also the problem that troubles me ,and thank you for your answer."
            }
          ]
        }
      ]
    },
    {
      "id": 2480725,
      "postDate": "2023-10-13T14:09:40.030Z",
      "content": "<p>Congrats on your solo gold!</p>",
      "rawMarkdown": "Congrats on your solo gold!",
      "votes": 1,
      "replies": [
        {
          "id": 2480954,
          "postDate": "2023-10-13T16:53:53.297Z",
          "content": "<p>Thank you! And congratulations on winning the gold medal in the intense LLM competition!</p>",
          "rawMarkdown": "Thank you! And congratulations on winning the gold medal in the intense LLM competition!",
          "votes": 1
        }
      ]
    },
    {
      "id": 2480434,
      "postDate": "2023-10-13T10:06:51.650Z",
      "content": "<p>Very good solution,  Congrats and thanks for sharing!</p>",
      "rawMarkdown": "Very good solution,  Congrats and thanks for sharing!",
      "votes": 1
    },
    {
      "id": 2478883,
      "postDate": "2023-10-12T08:51:41.533Z",
      "content": "<p>Congratulations on your powerful solo gold!<br>\nThe single model in deberta-v3-large is stronger and better than us.<br>\nThis was impressive.</p>\n<blockquote>\n  <p>Inspired by the PPPM competition's magic, considered concatenating other summaries from the same prompt, but it did not improve.</p>\n</blockquote>\n<p>I did this too and it was not good at all. lol</p>",
      "rawMarkdown": "Congratulations on your powerful solo gold!\nThe single model in deberta-v3-large is stronger and better than us.\nThis was impressive.\n\n>  Inspired by the PPPM competition's magic, considered concatenating other summaries from the same prompt, but it did not improve.\n\n\nI did this too and it was not good at all. lol",
      "votes": 1,
      "replies": [
        {
          "id": 2478929,
          "postDate": "2023-10-12T09:33:11.997Z",
          "content": "<p>Thank you for your comment.<br>\nAnd congrats to you on 4th place and Grand Master !</p>",
          "rawMarkdown": "Thank you for your comment.\nAnd congrats to you on 4th place and Grand Master !",
          "votes": 1
        }
      ]
    },
    {
      "id": 2481385,
      "postDate": "2023-10-14T04:51:18.867Z",
      "content": "<p>Congrats on your impressive solution, I wonder how can you choose the max_len is 1536.</p>",
      "rawMarkdown": "Congrats on your impressive solution, I wonder how can you choose the max_len is 1536."
    },
    {
      "id": 2479937,
      "postDate": "2023-10-13T02:37:26.070Z",
      "content": "<p>Congrats!! Amazing solo gold<br>\nI have a question here!</p>\n<p><code>Freeze embedding and the first 18 layers</code><br>\n-&gt;How many epochs trained?</p>\n<p>Thanks for providing this impressive and cool solution</p>",
      "rawMarkdown": "Congrats!! Amazing solo gold\nI have a question here!\n\n`Freeze embedding and the first 18 layers`\n->How many epochs trained?\n\nThanks for providing this impressive and cool solution",
      "replies": [
        {
          "id": 2480081,
          "postDate": "2023-10-13T05:40:41.400Z",
          "content": "<p>Thanks you! I trained 4 epochs. </p>",
          "rawMarkdown": "Thanks you! I trained 4 epochs. ",
          "votes": 2,
          "replies": [
            {
              "id": 2481007,
              "postDate": "2023-10-13T17:42:01.880Z",
              "content": "<p>That was a good idea. I've never thought about freezing more than half of the layers😅<br>\nThx!</p>",
              "rawMarkdown": "That was a good idea. I've never thought about freezing more than half of the layers😅\nThx!"
            }
          ]
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-23T15:39:57Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10675525,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/monnu-5th-place-solution"
}