{
  "id": 447944,
  "title": "1st Place Efficiency Solution",
  "content": "<p>This is a classic distill into deberta-v3-xsmall. Runs in ~15mins. </p>\n<p>Inference on prompt title, prompt question, summary text.</p>\n<p>The magic: tokenize prompt text, lookup token/2-token/3-token matches—and include as token_type_ids (0-6, for all windows, is better than 0-3, max pooled by 1/2/3-token)</p>\n<p>The model knows exactly which text is quoted/copied, and which words and phrases come from the prompt, which is a lot of contextual information at 1/10th the sequence length. </p>\n<p>—-</p>\n<p>Base models are deberta-v3-base with attn/cls pooling, and 4/7 token type ids. These four plus lgb are stacked into out of fold predictions, for distillation. </p>\n<p>—</p>\n<p>Content is adjusted by ~80% and wording is adjusted by 20-40% by prompt mean. </p>\n<p>Final ensemble is 68/42 nn/lgb for content; 75/30 for wording. </p>\n<p>—</p>\n<p>Distillation includes cosine annealing from pseudo labels to half pseudo half label across the course of three epochs. </p>\n<p>Many other ideas were tried (cross-attention to prompt; embedding approaches; massive library of generative text; etc); these were the ones that worked. </p>\n<p>Credit to Psi, Team Turing, and Kaggle—also the gzip paper. May release code after packaging. </p>",
  "comments": [
    {
      "id": 2486393,
      "author_name": "Stochoshi G",
      "author_url": "",
      "post_date": "2023-10-17T21:42:47.360000",
      "content": "<p>Update: </p>\n<p>Code—<a href=\"https://www.kaggle.com/code/stochoshi/commonlit-d1-cpu-code-4\" target=\"_blank\">https://www.kaggle.com/code/stochoshi/commonlit-d1-cpu-code-4</a></p>\n<p>Real Code—<a href=\"https://www.kaggle.com/datasets/stochoshi/commonlit-d1-cpu-2\" target=\"_blank\">https://www.kaggle.com/datasets/stochoshi/commonlit-d1-cpu-2</a></p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 2486439,
      "author_name": "Xiang Huang",
      "author_url": "",
      "post_date": "2023-10-17T23:33:53.250000",
      "content": "<p>Congrats on 1st place! </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2486408,
      "author_name": "jagstang",
      "author_url": "",
      "post_date": "2023-10-17T22:07:50.643000",
      "content": "<p>Good grief, what a solution. Congrats on 1st place! Well-deserved to say the least.</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 3,
  "messages": [
    {
      "id": 2486390,
      "postDate": "2023-10-17T21:33:19.270Z",
      "content": "<p>This is a classic distill into deberta-v3-xsmall. Runs in ~15mins. </p>\n<p>Inference on prompt title, prompt question, summary text.</p>\n<p>The magic: tokenize prompt text, lookup token/2-token/3-token matches—and include as token_type_ids (0-6, for all windows, is better than 0-3, max pooled by 1/2/3-token)</p>\n<p>The model knows exactly which text is quoted/copied, and which words and phrases come from the prompt, which is a lot of contextual information at 1/10th the sequence length. </p>\n<p>—-</p>\n<p>Base models are deberta-v3-base with attn/cls pooling, and 4/7 token type ids. These four plus lgb are stacked into out of fold predictions, for distillation. </p>\n<p>—</p>\n<p>Content is adjusted by ~80% and wording is adjusted by 20-40% by prompt mean. </p>\n<p>Final ensemble is 68/42 nn/lgb for content; 75/30 for wording. </p>\n<p>—</p>\n<p>Distillation includes cosine annealing from pseudo labels to half pseudo half label across the course of three epochs. </p>\n<p>Many other ideas were tried (cross-attention to prompt; embedding approaches; massive library of generative text; etc); these were the ones that worked. </p>\n<p>Credit to Psi, Team Turing, and Kaggle—also the gzip paper. May release code after packaging. </p>",
      "rawMarkdown": "This is a classic distill into deberta-v3-xsmall. Runs in ~15mins. \n\nInference on prompt title, prompt question, summary text.\n\nThe magic: tokenize prompt text, lookup token/2-token/3-token matches—and include as token_type_ids (0-6, for all windows, is better than 0-3, max pooled by 1/2/3-token)\n\nThe model knows exactly which text is quoted/copied, and which words and phrases come from the prompt, which is a lot of contextual information at 1/10th the sequence length. \n\n—-\n\nBase models are deberta-v3-base with attn/cls pooling, and 4/7 token type ids. These four plus lgb are stacked into out of fold predictions, for distillation. \n\n—\n\nContent is adjusted by ~80% and wording is adjusted by 20-40% by prompt mean. \n\nFinal ensemble is 68/42 nn/lgb for content; 75/30 for wording. \n\n—\n\nDistillation includes cosine annealing from pseudo labels to half pseudo half label across the course of three epochs. \n\nMany other ideas were tried (cross-attention to prompt; embedding approaches; massive library of generative text; etc); these were the ones that worked. \n\nCredit to Psi, Team Turing, and Kaggle—also the gzip paper. May release code after packaging. \n\n",
      "votes": 19
    },
    {
      "id": 2486393,
      "postDate": "2023-10-17T21:42:47.360Z",
      "content": "<p>Update: </p>\n<p>Code—<a href=\"https://www.kaggle.com/code/stochoshi/commonlit-d1-cpu-code-4\" target=\"_blank\">https://www.kaggle.com/code/stochoshi/commonlit-d1-cpu-code-4</a></p>\n<p>Real Code—<a href=\"https://www.kaggle.com/datasets/stochoshi/commonlit-d1-cpu-2\" target=\"_blank\">https://www.kaggle.com/datasets/stochoshi/commonlit-d1-cpu-2</a></p>",
      "rawMarkdown": "Update: \n\nCode—https://www.kaggle.com/code/stochoshi/commonlit-d1-cpu-code-4\n\nReal Code—https://www.kaggle.com/datasets/stochoshi/commonlit-d1-cpu-2\n\n",
      "votes": 3
    },
    {
      "id": 2486439,
      "postDate": "2023-10-17T23:33:53.250Z",
      "content": "<p>Congrats on 1st place! </p>",
      "rawMarkdown": "Congrats on 1st place! ",
      "votes": 1
    },
    {
      "id": 2486408,
      "postDate": "2023-10-17T22:07:50.643Z",
      "content": "<p>Good grief, what a solution. Congrats on 1st place! Well-deserved to say the least.</p>",
      "rawMarkdown": "Good grief, what a solution. Congrats on 1st place! Well-deserved to say the least.",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:07:22Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10679520,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/stochoshi-1st-place-efficiency-solution"
}