{
  "id": 446676,
  "title": "20th place solution",
  "content": "<p>Thank you to the organizers for the fun competition and everyone who participated. <br>\nI share my solution.</p>\n<h1>Summary</h1>\n<ul>\n<li>When I added prompt_text, the local score clearly improved. However, as a result of the experiment, I found that prompt_text is very long (more than 4000) for test dataset. So  I set the max_length to 5000. This is the important point for me.</li>\n<li>weighted ensemble: deberta-v3-large + LightGBM = 7:3</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F164129%2F5c1484cc862d2bcea8e3c359b287799b%2Fimage2.png?generation=1697126808205146&amp;alt=media\" alt=\"solution-image\"></p>\n<h1>1. Validation Strategy</h1>\n<ul>\n<li>GroupKFold : 4-fold (groups=prompt_id)</li>\n</ul>\n<h1>2. deberta-v3-large model</h1>\n<ul>\n<li>input<ul>\n<li>input_text = answer: summary_text [SEP] title: prompt_title | question: prompt_question | text: prompt_text</li>\n<li>tokenizer: max_length=5000</li>\n<li>not text cleansing</li></ul></li>\n<li>model:<ul>\n<li>pretrained-model: microsoft/deberta-v3-large</li>\n<li>freeze: 18 layers</li>\n<li>head: cls-token</li>\n<li>max_position_embeddings=5000</li></ul></li>\n<li>train: <ul>\n<li>loss: torch.nn.SmoothL1Loss()</li>\n<li>optimizer: Adam</li>\n<li>batch_size=2</li>\n<li>epoch=10</li>\n<li>scheduler: cosine annealing</li></ul></li>\n<li>predict:<ul>\n<li>max_token=5000 </li>\n<li>batch_size=1</li>\n<li>only 2-fold (When it was 4-fold, it timed out.)</li></ul></li>\n</ul>\n<h1>3. LightGBM</h1>\n<ul>\n<li>features: 141 (count_word, count_paragraph, count_sentence, count_space, etc.)</li>\n<li>train: 2 model (content, wording)</li>\n</ul>\n<h1>4. Ensemble</h1>\n<ul>\n<li>weighted average (tune weight by train_oof)</li>\n<li>deberta-v3-large : LightGBM = 7:3</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>id</th>\n<th>model</th>\n<th>local-cv</th>\n<th>public</th>\n<th>private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>Deberta-v3-large (4fold)</td>\n<td>0.4964</td>\n<td>Timeout</td>\n<td>Timeout</td>\n</tr>\n<tr>\n<td>2</td>\n<td>Deberta-v3-large (2fold)</td>\n<td>0.5007</td>\n<td>0.461</td>\n<td>0.476</td>\n</tr>\n<tr>\n<td>3</td>\n<td>LightGBM</td>\n<td>0.5532</td>\n<td>0.495</td>\n<td>0.516</td>\n</tr>\n<tr>\n<td>4</td>\n<td>ensemble(2+3)</td>\n<td>0.4822</td>\n<td>0.455</td>\n<td>0.461</td>\n</tr>\n</tbody>\n</table>\n<h1>Others</h1>\n<ul>\n<li>pooling-layer: almost same</li>\n<li>N-hidden-layer: almost same</li>\n<li>LSTM/1dcnn-layer: not work</li>\n<li>Set lr for each layer: not work</li>\n<li>Longformer: not work</li>\n<li>summarizing prompt_text using Llama2: not work</li>\n</ul>\n<p>When I added prompt_text, the local score clearly improved. I also considered it important to have a model that can handle the long token length of test data. For this reason, although it was a submitted file that was outside the medal range, I believed in the local score and selected this submit file. I'm glad I chose it.<br>\nHowever, I realized the length of the tokens two days before the end of the competition, so I regret not being able to try 4-fold model. .</p>\n<p>Thank you for reading.</p>",
  "comments": [
    {
      "id": 2479464,
      "author_name": "takai380",
      "author_url": "",
      "post_date": "2023-10-12T16:24:26.437000",
      "content": "<p>Thanks for sharing your solution!</p>\n<p>You mentioned that you set max_len to 5000, did you not get memory out? Did you do anything creative at that time?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2479500,
          "author_name": "Moro",
          "author_url": "",
          "post_date": "2023-10-12T16:50:40.520000",
          "content": "<p>Thank you for your comment.<br>\nI did the following to avoid CUDA out of memory.</p>\n<h1>(1) Automatic Mixed Precision (AMP)</h1>\n<ul>\n<li>Reduce memory usage with AMP</li>\n<li><a href=\"https://pytorch.org/docs/stable/notes/amp_examples.html\" target=\"_blank\">https://pytorch.org/docs/stable/notes/amp_examples.html</a></li>\n</ul>\n<pre><code>scaler = GradScaler()\n autocast(device_type=, dtype=torch.float16):\n    preds = model(input_data)\n</code></pre>\n<h1>(2) use DataCollatorWithPadding</h1>\n<ul>\n<li>Truncate at maximum length in mini-batch</li>\n<li>DataCollatorWithPadding(tokenizer=tokenizer, padding=\"longest\")</li>\n<li><a href=\"https://huggingface.co/docs/transformers/main_classes/data_collator\" target=\"_blank\">https://huggingface.co/docs/transformers/main_classes/data_collator</a></li>\n</ul>\n<pre><code>dataset_train = BertDataset(input_x, input_y, MAX_LEN=MAX_LEN, tokenizer=tokenizer, isTrain=)\ndataloader_train = DataLoader(\n    dataset_train, batch_size=batch_size, shuffle=, num_workers=num_workers, pin_memory=,\n    collate_fn=DataCollatorWithPadding(tokenizer=tokenizer, padding=),\n    )\n</code></pre>",
          "votes": 2,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 2479424,
      "postDate": "2023-10-12T15:58:23.147Z",
      "content": "<p>Thank you to the organizers for the fun competition and everyone who participated. <br>\nI share my solution.</p>\n<h1>Summary</h1>\n<ul>\n<li>When I added prompt_text, the local score clearly improved. However, as a result of the experiment, I found that prompt_text is very long (more than 4000) for test dataset. So  I set the max_length to 5000. This is the important point for me.</li>\n<li>weighted ensemble: deberta-v3-large + LightGBM = 7:3</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F164129%2F5c1484cc862d2bcea8e3c359b287799b%2Fimage2.png?generation=1697126808205146&amp;alt=media\" alt=\"solution-image\"></p>\n<h1>1. Validation Strategy</h1>\n<ul>\n<li>GroupKFold : 4-fold (groups=prompt_id)</li>\n</ul>\n<h1>2. deberta-v3-large model</h1>\n<ul>\n<li>input<ul>\n<li>input_text = answer: summary_text [SEP] title: prompt_title | question: prompt_question | text: prompt_text</li>\n<li>tokenizer: max_length=5000</li>\n<li>not text cleansing</li></ul></li>\n<li>model:<ul>\n<li>pretrained-model: microsoft/deberta-v3-large</li>\n<li>freeze: 18 layers</li>\n<li>head: cls-token</li>\n<li>max_position_embeddings=5000</li></ul></li>\n<li>train: <ul>\n<li>loss: torch.nn.SmoothL1Loss()</li>\n<li>optimizer: Adam</li>\n<li>batch_size=2</li>\n<li>epoch=10</li>\n<li>scheduler: cosine annealing</li></ul></li>\n<li>predict:<ul>\n<li>max_token=5000 </li>\n<li>batch_size=1</li>\n<li>only 2-fold (When it was 4-fold, it timed out.)</li></ul></li>\n</ul>\n<h1>3. LightGBM</h1>\n<ul>\n<li>features: 141 (count_word, count_paragraph, count_sentence, count_space, etc.)</li>\n<li>train: 2 model (content, wording)</li>\n</ul>\n<h1>4. Ensemble</h1>\n<ul>\n<li>weighted average (tune weight by train_oof)</li>\n<li>deberta-v3-large : LightGBM = 7:3</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>id</th>\n<th>model</th>\n<th>local-cv</th>\n<th>public</th>\n<th>private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>Deberta-v3-large (4fold)</td>\n<td>0.4964</td>\n<td>Timeout</td>\n<td>Timeout</td>\n</tr>\n<tr>\n<td>2</td>\n<td>Deberta-v3-large (2fold)</td>\n<td>0.5007</td>\n<td>0.461</td>\n<td>0.476</td>\n</tr>\n<tr>\n<td>3</td>\n<td>LightGBM</td>\n<td>0.5532</td>\n<td>0.495</td>\n<td>0.516</td>\n</tr>\n<tr>\n<td>4</td>\n<td>ensemble(2+3)</td>\n<td>0.4822</td>\n<td>0.455</td>\n<td>0.461</td>\n</tr>\n</tbody>\n</table>\n<h1>Others</h1>\n<ul>\n<li>pooling-layer: almost same</li>\n<li>N-hidden-layer: almost same</li>\n<li>LSTM/1dcnn-layer: not work</li>\n<li>Set lr for each layer: not work</li>\n<li>Longformer: not work</li>\n<li>summarizing prompt_text using Llama2: not work</li>\n</ul>\n<p>When I added prompt_text, the local score clearly improved. I also considered it important to have a model that can handle the long token length of test data. For this reason, although it was a submitted file that was outside the medal range, I believed in the local score and selected this submit file. I'm glad I chose it.<br>\nHowever, I realized the length of the tokens two days before the end of the competition, so I regret not being able to try 4-fold model. .</p>\n<p>Thank you for reading.</p>",
      "rawMarkdown": "Thank you to the organizers for the fun competition and everyone who participated. \nI share my solution.\n\n# Summary\n- When I added prompt_text, the local score clearly improved. However, as a result of the experiment, I found that prompt_text is very long (more than 4000) for test dataset. So  I set the max_length to 5000. This is the important point for me.\n- weighted ensemble: deberta-v3-large + LightGBM = 7:3\n\n![solution-image](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F164129%2F5c1484cc862d2bcea8e3c359b287799b%2Fimage2.png?generation=1697126808205146&alt=media)\n\n# 1. Validation Strategy\n- GroupKFold : 4-fold (groups=prompt_id)\n\n# 2. deberta-v3-large model\n- input\n    - input_text = answer: summary_text [SEP] title: prompt_title | question: prompt_question | text: prompt_text\n    - tokenizer: max_length=5000\n    - not text cleansing\n- model:\n    - pretrained-model: microsoft/deberta-v3-large\n    - freeze: 18 layers\n    - head: cls-token\n    - max_position_embeddings=5000\n- train: \n    - loss: torch.nn.SmoothL1Loss()\n    - optimizer: Adam\n    - batch_size=2\n    - epoch=10\n    - scheduler: cosine annealing\n- predict:\n    - max_token=5000 \n    - batch_size=1\n    - only 2-fold (When it was 4-fold, it timed out.)\n\n# 3. LightGBM\n- features: 141 (count_word, count_paragraph, count_sentence, count_space, etc.)\n- train: 2 model (content, wording)\n\n# 4. Ensemble\n- weighted average (tune weight by train_oof)\n- deberta-v3-large : LightGBM = 7:3\n\n| id | model | local-cv | public | private |\n|---|---|---:|---:|---:|\n| 1 | Deberta-v3-large (4fold) | 0.4964 | Timeout | Timeout |\n| 2 | Deberta-v3-large (2fold) | 0.5007 | 0.461 | 0.476 |\n| 3 | LightGBM | 0.5532 | 0.495 | 0.516 |\n| 4 | ensemble(2+3) | 0.4822 | 0.455 | 0.461 |\n\n# Others\n- pooling-layer: almost same\n- N-hidden-layer: almost same\n- LSTM/1dcnn-layer: not work\n- Set lr for each layer: not work\n- Longformer: not work\n- summarizing prompt_text using Llama2: not work\n\nWhen I added prompt_text, the local score clearly improved. I also considered it important to have a model that can handle the long token length of test data. For this reason, although it was a submitted file that was outside the medal range, I believed in the local score and selected this submit file. I'm glad I chose it.\nHowever, I realized the length of the tokens two days before the end of the competition, so I regret not being able to try 4-fold model. .\n\nThank you for reading.",
      "votes": 17
    },
    {
      "id": 2479464,
      "postDate": "2023-10-12T16:24:26.437Z",
      "content": "<p>Thanks for sharing your solution!</p>\n<p>You mentioned that you set max_len to 5000, did you not get memory out? Did you do anything creative at that time?</p>",
      "rawMarkdown": "Thanks for sharing your solution!\n\nYou mentioned that you set max_len to 5000, did you not get memory out? Did you do anything creative at that time?",
      "votes": 1,
      "replies": [
        {
          "id": 2479500,
          "postDate": "2023-10-12T16:50:40.520Z",
          "content": "<p>Thank you for your comment.<br>\nI did the following to avoid CUDA out of memory.</p>\n<h1>(1) Automatic Mixed Precision (AMP)</h1>\n<ul>\n<li>Reduce memory usage with AMP</li>\n<li><a href=\"https://pytorch.org/docs/stable/notes/amp_examples.html\" target=\"_blank\">https://pytorch.org/docs/stable/notes/amp_examples.html</a></li>\n</ul>\n<pre><code>scaler = GradScaler()\n autocast(device_type=, dtype=torch.float16):\n    preds = model(input_data)\n</code></pre>\n<h1>(2) use DataCollatorWithPadding</h1>\n<ul>\n<li>Truncate at maximum length in mini-batch</li>\n<li>DataCollatorWithPadding(tokenizer=tokenizer, padding=\"longest\")</li>\n<li><a href=\"https://huggingface.co/docs/transformers/main_classes/data_collator\" target=\"_blank\">https://huggingface.co/docs/transformers/main_classes/data_collator</a></li>\n</ul>\n<pre><code>dataset_train = BertDataset(input_x, input_y, MAX_LEN=MAX_LEN, tokenizer=tokenizer, isTrain=)\ndataloader_train = DataLoader(\n    dataset_train, batch_size=batch_size, shuffle=, num_workers=num_workers, pin_memory=,\n    collate_fn=DataCollatorWithPadding(tokenizer=tokenizer, padding=),\n    )\n</code></pre>",
          "rawMarkdown": "Thank you for your comment.\nI did the following to avoid CUDA out of memory.\n\n# (1) Automatic Mixed Precision (AMP)\n- Reduce memory usage with AMP\n- https://pytorch.org/docs/stable/notes/amp_examples.html\n\n```python\nscaler = GradScaler()\nwith autocast(device_type=\"cuda\", dtype=torch.float16):\n    preds = model(input_data)\n```\n\n# (2) use DataCollatorWithPadding\n- Truncate at maximum length in mini-batch\n- DataCollatorWithPadding(tokenizer=tokenizer, padding=\"longest\")\n- https://huggingface.co/docs/transformers/main_classes/data_collator\n\n```python\ndataset_train = BertDataset(input_x, input_y, MAX_LEN=MAX_LEN, tokenizer=tokenizer, isTrain=True)\ndataloader_train = DataLoader(\n    dataset_train, batch_size=batch_size, shuffle=True, num_workers=num_workers, pin_memory=True,\n    collate_fn=DataCollatorWithPadding(tokenizer=tokenizer, padding=\"longest\"),\n    )\n```",
          "votes": 2
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:04:12Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10808120,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/moro-20th-place-solution"
}