{
  "id": 446516,
  "title": "49th Place Solution [Single Model]",
  "content": "<p>Thanks a lot to the hosts and Kaggle for hosting this interesting competition. I personally had a great time working on this competition.</p>\n<h2>Summary:</h2>\n<p>My solution primarily relies on a single DeBERTa-v3-large model with a custom MeanPooling layer. Used all the text columns <code>text</code>, <code>prompt_text</code>, <code>prompt_question</code>and <code>prompt_title</code> for training. Trainied with 1500 max_len and infer with 2048.  </p>\n<h2>Input Processing:</h2>\n<p>To prepare the input data, I added two special tokens, '[SUMMARY_START]' and '[SUMMARY_END]', before and after the summary text, respectively. Then, I appended all the other prompt data with a '[SEP]' token. The final input text structure looked like this:<br>\n'[SUMMARY_START]text[SUMMARY_END][SEP]prompt_text[SEP]prompt_question[SEP]prompt_title[SEP]'.</p>\n<h2>Training Details:</h2>\n<p>The DeBERTa-v3-large model was trained with a maximum sequence length of 1500 using the input structure mentioned above. I also modified the MeanPooling layer to calculate the mean only between the '[SUMMARY_START]' and '[SUMMARY_END]' tokens.</p>\n<h2>CV - LB Details</h2>\n<ul>\n<li>CV - 0.474 Public LB - 0.43 Private LB 0.469</li>\n</ul>\n<h2>Inference Time:</h2>\n<p>It's important to note that this single model has a significant inference time of approximately 8.5 hours.</p>\n<h2>What Worked:</h2>\n<ul>\n<li>Inferencing on larger Max Len</li>\n<li>Averaging Multiple checkpoints per epoch.</li>\n</ul>\n<h2>What Didn't Work:</h2>\n<ul>\n<li>Implementing a second-stage model (LGBM/XGBOOST).</li>\n<li>Text cleaning.</li>\n<li>Post-processing techniques, such as target smoothing.</li>\n</ul>\n<h2>Training Code</h2>\n<p><a href=\"https://github.com/rohitsingh02/CommonLit-ESS\" target=\"_blank\">https://github.com/rohitsingh02/CommonLit-ESS</a></p>\n<h2>Inference Kernel</h2>\n<p><a href=\"https://www.kaggle.com/code/rohitsingh9990/commonlit-ensemble-new-v2?scriptVersionId=145818122\" target=\"_blank\">https://www.kaggle.com/code/rohitsingh9990/commonlit-ensemble-new-v2?scriptVersionId=145818122</a></p>",
  "comments": [
    {
      "id": 2478696,
      "author_name": "anji",
      "author_url": "",
      "post_date": "2023-10-12T06:48:16.623000",
      "content": "<p>It helps me a lot. Thanks. I'd like to ask if the reason why we are able to use a <strong>larger Max Len</strong> in the inference is due to deberta's relative positional encoding structure?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478710,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2023-10-12T07:04:09.143000",
          "content": "<p>Based on the problem statement, task was to rate summary text based on given prompt texts. So summary is written by students based on the prompt_text and prompt_text has large tokens [more than 3k tokens in private for some prompts]. Also in my experiments adding full prompt_text leads to better CV, so i decided to go with larger max_len. Perhaps i should have compromised on max_len and added more variability through multi-model ensemble, that could have given me better results on private LB. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2478652,
      "author_name": "xavier fernando",
      "author_url": "",
      "post_date": "2023-10-12T06:17:26.793000",
      "content": "<p>How did you extract the value , after sending the inputs to debertaV3. then extract(mean pooling) the tokens between the '[SUMMARY_START]' and '[SUMMARY_END]'. could you please explain that idea. </p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478673,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2023-10-12T06:29:56.163000",
          "content": "<p>First i save the token ids like     <br>\n<code>config.text_start_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_START]\")\n config.text_end_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_END]\")</code></p>\n<p>and while processing outputs i used </p>\n<pre><code> = model(inputs)\n = inputs[]\n\n = (input_ids == config.text_start_token).nonzero()[:, ]\n = (input_ids == config.text_end_token).nonzero()[:, ]\n = torch.arange(input_ids.size()).expand(input_ids.size(), -).to(input_ids.device)\n = (mask &gt;= start_tokens.unsqueeze()) &amp; (mask &lt;= end_tokens.unsqueeze())\n = y_preds * mask.unsqueeze().float()\n</code></pre>",
          "votes": 4,
          "replies": [
            {
              "id": 2478689,
              "author_name": "xavier fernando",
              "author_url": "",
              "post_date": "2023-10-12T06:42:00.693000",
              "content": "<p>Thanks a lot <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2484081,
              "author_name": "anji",
              "author_url": "",
              "post_date": "2023-10-16T08:02:34.523000",
              "content": "<blockquote>\n  <p><code>config.text_start_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_START]\")\n config.text_end_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_END]\")</code><br>\n  hi，I tried to reproduce your code, and I found that text_start_token and text_end_token are both 3, which is the tokens corresponding to '[UNK]', I don't know what I did wrong. I think it is because the tokens [SUMMARY_START] and [SUMMARY_END] do not exist in the deberta pretrained vocab.<br>\n  <img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10348695%2F11ae26779a6f2fdde0a55485e6dcb7a2%2FWechatIMG1284.jpg?generation=1697443348671003&amp;alt=media\" alt=\"\"></p>\n</blockquote>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2484088,
              "author_name": "Dracarys",
              "author_url": "",
              "post_date": "2023-10-16T08:11:57.970000",
              "content": "<p>You need to add these special token to the tokenizer first<br>\n`   all_special_tokens =[\"[SUMMARY_START]\", \"[SUMMARY_END]\"]</p>\n<pre><code>tokenizer = AutoTokenizer.from_pretrained(\n    config.architecture.model_name,\n    =,\n    =all_special_tokens\n)`\n</code></pre>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2484102,
              "author_name": "anji",
              "author_url": "",
              "post_date": "2023-10-16T08:18:11.210000",
              "content": "<p>Thank you so much, I get it!!! </p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2478507,
      "author_name": "AK",
      "author_url": "",
      "post_date": "2023-10-12T04:00:44.077000",
      "content": "<p>Nice job <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> </p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478620,
          "author_name": "Dracarys",
          "author_url": "",
          "post_date": "2023-10-12T05:34:07.867000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/aman1391\" target=\"_blank\">@aman1391</a> and congratulations to you and your team for gold finish.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2481265,
      "author_name": "DL",
      "author_url": "",
      "post_date": "2023-10-14T01:30:08.707000",
      "content": "<p>thanks for sharing，really nice work</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 11,
  "messages": [
    {
      "id": 2478498,
      "postDate": "2023-10-12T03:52:29.263Z",
      "content": "<p>Thanks a lot to the hosts and Kaggle for hosting this interesting competition. I personally had a great time working on this competition.</p>\n<h2>Summary:</h2>\n<p>My solution primarily relies on a single DeBERTa-v3-large model with a custom MeanPooling layer. Used all the text columns <code>text</code>, <code>prompt_text</code>, <code>prompt_question</code>and <code>prompt_title</code> for training. Trainied with 1500 max_len and infer with 2048.  </p>\n<h2>Input Processing:</h2>\n<p>To prepare the input data, I added two special tokens, '[SUMMARY_START]' and '[SUMMARY_END]', before and after the summary text, respectively. Then, I appended all the other prompt data with a '[SEP]' token. The final input text structure looked like this:<br>\n'[SUMMARY_START]text[SUMMARY_END][SEP]prompt_text[SEP]prompt_question[SEP]prompt_title[SEP]'.</p>\n<h2>Training Details:</h2>\n<p>The DeBERTa-v3-large model was trained with a maximum sequence length of 1500 using the input structure mentioned above. I also modified the MeanPooling layer to calculate the mean only between the '[SUMMARY_START]' and '[SUMMARY_END]' tokens.</p>\n<h2>CV - LB Details</h2>\n<ul>\n<li>CV - 0.474 Public LB - 0.43 Private LB 0.469</li>\n</ul>\n<h2>Inference Time:</h2>\n<p>It's important to note that this single model has a significant inference time of approximately 8.5 hours.</p>\n<h2>What Worked:</h2>\n<ul>\n<li>Inferencing on larger Max Len</li>\n<li>Averaging Multiple checkpoints per epoch.</li>\n</ul>\n<h2>What Didn't Work:</h2>\n<ul>\n<li>Implementing a second-stage model (LGBM/XGBOOST).</li>\n<li>Text cleaning.</li>\n<li>Post-processing techniques, such as target smoothing.</li>\n</ul>\n<h2>Training Code</h2>\n<p><a href=\"https://github.com/rohitsingh02/CommonLit-ESS\" target=\"_blank\">https://github.com/rohitsingh02/CommonLit-ESS</a></p>\n<h2>Inference Kernel</h2>\n<p><a href=\"https://www.kaggle.com/code/rohitsingh9990/commonlit-ensemble-new-v2?scriptVersionId=145818122\" target=\"_blank\">https://www.kaggle.com/code/rohitsingh9990/commonlit-ensemble-new-v2?scriptVersionId=145818122</a></p>",
      "rawMarkdown": "Thanks a lot to the hosts and Kaggle for hosting this interesting competition. I personally had a great time working on this competition.\n\n## Summary:\nMy solution primarily relies on a single DeBERTa-v3-large model with a custom MeanPooling layer. Used all the text columns `text`, `prompt_text`, `prompt_question`and `prompt_title` for training. Trainied with 1500 max_len and infer with 2048.  \n\n## Input Processing:\nTo prepare the input data, I added two special tokens, '[SUMMARY_START]' and '[SUMMARY_END]', before and after the summary text, respectively. Then, I appended all the other prompt data with a '[SEP]' token. The final input text structure looked like this:\n'[SUMMARY_START]text[SUMMARY_END][SEP]prompt_text[SEP]prompt_question[SEP]prompt_title[SEP]'.\n\n## Training Details:\nThe DeBERTa-v3-large model was trained with a maximum sequence length of 1500 using the input structure mentioned above. I also modified the MeanPooling layer to calculate the mean only between the '[SUMMARY_START]' and '[SUMMARY_END]' tokens.\n\n## CV - LB Details\n* CV - 0.474 Public LB - 0.43 Private LB 0.469\n\n## Inference Time:\nIt's important to note that this single model has a significant inference time of approximately 8.5 hours.\n\n## What Worked: \n* Inferencing on larger Max Len\n* Averaging Multiple checkpoints per epoch.\n\n## What Didn't Work:\n\n* Implementing a second-stage model (LGBM/XGBOOST).\n* Text cleaning.\n* Post-processing techniques, such as target smoothing.\n\n\n## Training Code\nhttps://github.com/rohitsingh02/CommonLit-ESS\n\n## Inference Kernel \nhttps://www.kaggle.com/code/rohitsingh9990/commonlit-ensemble-new-v2?scriptVersionId=145818122\n\n",
      "votes": 23
    },
    {
      "id": 2478696,
      "postDate": "2023-10-12T06:48:16.623Z",
      "content": "<p>It helps me a lot. Thanks. I'd like to ask if the reason why we are able to use a <strong>larger Max Len</strong> in the inference is due to deberta's relative positional encoding structure?</p>",
      "rawMarkdown": "It helps me a lot. Thanks. I'd like to ask if the reason why we are able to use a **larger Max Len** in the inference is due to deberta's relative positional encoding structure?",
      "votes": 1,
      "replies": [
        {
          "id": 2478710,
          "postDate": "2023-10-12T07:04:09.143Z",
          "content": "<p>Based on the problem statement, task was to rate summary text based on given prompt texts. So summary is written by students based on the prompt_text and prompt_text has large tokens [more than 3k tokens in private for some prompts]. Also in my experiments adding full prompt_text leads to better CV, so i decided to go with larger max_len. Perhaps i should have compromised on max_len and added more variability through multi-model ensemble, that could have given me better results on private LB. </p>",
          "rawMarkdown": "Based on the problem statement, task was to rate summary text based on given prompt texts. So summary is written by students based on the prompt_text and prompt_text has large tokens [more than 3k tokens in private for some prompts]. Also in my experiments adding full prompt_text leads to better CV, so i decided to go with larger max_len. Perhaps i should have compromised on max_len and added more variability through multi-model ensemble, that could have given me better results on private LB. ",
          "votes": 1
        }
      ]
    },
    {
      "id": 2478652,
      "postDate": "2023-10-12T06:17:26.793Z",
      "content": "<p>How did you extract the value , after sending the inputs to debertaV3. then extract(mean pooling) the tokens between the '[SUMMARY_START]' and '[SUMMARY_END]'. could you please explain that idea. </p>",
      "rawMarkdown": "How did you extract the value , after sending the inputs to debertaV3. then extract(mean pooling) the tokens between the '[SUMMARY_START]' and '[SUMMARY_END]'. could you please explain that idea. ",
      "votes": 1,
      "replies": [
        {
          "id": 2478673,
          "postDate": "2023-10-12T06:29:56.163Z",
          "content": "<p>First i save the token ids like     <br>\n<code>config.text_start_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_START]\")\n config.text_end_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_END]\")</code></p>\n<p>and while processing outputs i used </p>\n<pre><code> = model(inputs)\n = inputs[]\n\n = (input_ids == config.text_start_token).nonzero()[:, ]\n = (input_ids == config.text_end_token).nonzero()[:, ]\n = torch.arange(input_ids.size()).expand(input_ids.size(), -).to(input_ids.device)\n = (mask &gt;= start_tokens.unsqueeze()) &amp; (mask &lt;= end_tokens.unsqueeze())\n = y_preds * mask.unsqueeze().float()\n</code></pre>",
          "rawMarkdown": "First i save the token ids like     \n`config.text_start_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_START]\")\n config.text_end_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_END]\")`\n\nand while processing outputs i used \n```\ny_preds = model(inputs)\ninput_ids = inputs['input_ids']\n# Find the positions of start and end tokens\nstart_tokens = (input_ids == config.text_start_token).nonzero()[:, 1]\nend_tokens = (input_ids == config.text_end_token).nonzero()[:, 1]\nmask = torch.arange(input_ids.size(1)).expand(input_ids.size(0), -1).to(input_ids.device)\nmask = (mask >= start_tokens.unsqueeze(1)) & (mask <= end_tokens.unsqueeze(1))\ny_preds = y_preds * mask.unsqueeze(2).float()\n```\n",
          "votes": 4,
          "replies": [
            {
              "id": 2478689,
              "postDate": "2023-10-12T06:42:00.693Z",
              "content": "<p>Thanks a lot <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> </p>",
              "rawMarkdown": "Thanks a lot @rohitsingh9990 "
            },
            {
              "id": 2484081,
              "postDate": "2023-10-16T08:02:34.523Z",
              "content": "<blockquote>\n  <p><code>config.text_start_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_START]\")\n config.text_end_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_END]\")</code><br>\n  hi，I tried to reproduce your code, and I found that text_start_token and text_end_token are both 3, which is the tokens corresponding to '[UNK]', I don't know what I did wrong. I think it is because the tokens [SUMMARY_START] and [SUMMARY_END] do not exist in the deberta pretrained vocab.<br>\n  <img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10348695%2F11ae26779a6f2fdde0a55485e6dcb7a2%2FWechatIMG1284.jpg?generation=1697443348671003&amp;alt=media\" alt=\"\"></p>\n</blockquote>",
              "rawMarkdown": " > `config.text_start_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_START]\")\n>  config.text_end_token = tokenizer.convert_tokens_to_ids(\"[SUMMARY_END]\")`\n\nhi，I tried to reproduce your code, and I found that text_start_token and text_end_token are both 3, which is the tokens corresponding to '[UNK]', I don't know what I did wrong. I think it is because the tokens [SUMMARY_START] and [SUMMARY_END] do not exist in the deberta pretrained vocab.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10348695%2F11ae26779a6f2fdde0a55485e6dcb7a2%2FWechatIMG1284.jpg?generation=1697443348671003&alt=media)"
            },
            {
              "id": 2484088,
              "postDate": "2023-10-16T08:11:57.970Z",
              "content": "<p>You need to add these special token to the tokenizer first<br>\n`   all_special_tokens =[\"[SUMMARY_START]\", \"[SUMMARY_END]\"]</p>\n<pre><code>tokenizer = AutoTokenizer.from_pretrained(\n    config.architecture.model_name,\n    =,\n    =all_special_tokens\n)`\n</code></pre>",
              "rawMarkdown": "You need to add these special token to the tokenizer first\n`   all_special_tokens =[\"[SUMMARY_START]\", \"[SUMMARY_END]\"]\n    \n    tokenizer = AutoTokenizer.from_pretrained(\n        config.architecture.model_name,\n        use_fast=True,\n        additional_special_tokens=all_special_tokens\n    )`",
              "votes": 1
            },
            {
              "id": 2484102,
              "postDate": "2023-10-16T08:18:11.210Z",
              "content": "<p>Thank you so much, I get it!!! </p>",
              "rawMarkdown": "Thank you so much, I get it!!! "
            }
          ]
        }
      ]
    },
    {
      "id": 2478507,
      "postDate": "2023-10-12T04:00:44.077Z",
      "content": "<p>Nice job <a href=\"https://www.kaggle.com/rohitsingh9990\" target=\"_blank\">@rohitsingh9990</a> </p>",
      "rawMarkdown": "Nice job @rohitsingh9990 ",
      "votes": 1,
      "replies": [
        {
          "id": 2478620,
          "postDate": "2023-10-12T05:34:07.867Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/aman1391\" target=\"_blank\">@aman1391</a> and congratulations to you and your team for gold finish.</p>",
          "rawMarkdown": "Thanks @aman1391 and congratulations to you and your team for gold finish.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2481265,
      "postDate": "2023-10-14T01:30:08.707Z",
      "content": "<p>thanks for sharing，really nice work</p>",
      "rawMarkdown": "thanks for sharing，really nice work"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:01:21Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10742337,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/dracarys-49th-place-solution-single-model"
}