{
  "id": 446846,
  "title": "108th solution🥉",
  "content": "<h1>Solution</h1>\n<p>My best submission was almost same as this public code.<br>\n(<a href=\"https://www.kaggle.com/code/tsunotsuno/debertav3-lgbm-no-autocorrect\" target=\"_blank\">https://www.kaggle.com/code/tsunotsuno/debertav3-lgbm-no-autocorrect</a>)<br>\nI appreciate <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> to share the sophisticated notebooks!</p>\n<p>I changed the deberta-v3-base to deberta-v3-large.<br>\nTraining was done on google colab notebook.</p>\n<h1>What I tried but didn't work</h1>\n<ul>\n<li>MLM training for summary texts</li>\n<li>summary_text + prompt_question +prompt_text training.</li>\n<li>feed manual features made in the above public notebook to the nlp header.</li>\n</ul>\n<p>The fact that training set and test set have no common prompt_id and the training set have only 4 prompt_ids (it seems not enough for the model to learn the interaction pattern between prompts and summaries) seemed to be the most important factor in this competition to me.</p>\n<p>So I thought it might be efficient to teach the model the interaction of the prompt and summary using the test set, because we have only 4 prompts in our training set but many prompts in test set. So I tried deberta-v3-base MLM training on test set before regression training, but it didn't work.</p>\n<p>Thinking the way to tackle this point wasted me the most of the time I consumed in this competition…<br>\nI eventually selected the deberta-v3-large simple solution but there were potential to be improved…</p>\n<h1>What I should have done</h1>\n<ul>\n<li>ensemble nlp models using different combination of input order</li>\n<li>use longer token length</li>\n<li>ensemble GBDT stacking</li>\n<li>pooling techniques (mean/max pooling, convolution, LSTM)</li>\n</ul>\n<p>and so on.</p>\n<p>I didn't noticed changing the order or combination of the input to nlp model make the enough difference to the nlp model to be used for ensemble!</p>\n<h1>Other</h1>\n<p>I noticed that deberta sequence classification model implementation uses 'type_ids' argument instead of 'token_type_ids' argument.<br>\nThis seems to be one of the cause that including the prompt_text to the deberta input harms the model performance when you use huggingface Trainer and AutoTokenizer combination.<br>\nMaybe the SEP token is not enough for the model to distinguish where to score the content and wording.<br>\nYou can solve this issue by changing the returned key name by tokenizer from token_type_ids to type_ids.</p>\n<p><a href=\"https://github.com/microsoft/DeBERTa/blob/master/DeBERTa/apps/models/sequence_classification.py\" target=\"_blank\">https://github.com/microsoft/DeBERTa/blob/master/DeBERTa/apps/models/sequence_classification.py</a></p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2480442,
      "postDate": "2023-10-13T10:09:23.387Z",
      "content": "<h1>Solution</h1>\n<p>My best submission was almost same as this public code.<br>\n(<a href=\"https://www.kaggle.com/code/tsunotsuno/debertav3-lgbm-no-autocorrect\" target=\"_blank\">https://www.kaggle.com/code/tsunotsuno/debertav3-lgbm-no-autocorrect</a>)<br>\nI appreciate <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> to share the sophisticated notebooks!</p>\n<p>I changed the deberta-v3-base to deberta-v3-large.<br>\nTraining was done on google colab notebook.</p>\n<h1>What I tried but didn't work</h1>\n<ul>\n<li>MLM training for summary texts</li>\n<li>summary_text + prompt_question +prompt_text training.</li>\n<li>feed manual features made in the above public notebook to the nlp header.</li>\n</ul>\n<p>The fact that training set and test set have no common prompt_id and the training set have only 4 prompt_ids (it seems not enough for the model to learn the interaction pattern between prompts and summaries) seemed to be the most important factor in this competition to me.</p>\n<p>So I thought it might be efficient to teach the model the interaction of the prompt and summary using the test set, because we have only 4 prompts in our training set but many prompts in test set. So I tried deberta-v3-base MLM training on test set before regression training, but it didn't work.</p>\n<p>Thinking the way to tackle this point wasted me the most of the time I consumed in this competition…<br>\nI eventually selected the deberta-v3-large simple solution but there were potential to be improved…</p>\n<h1>What I should have done</h1>\n<ul>\n<li>ensemble nlp models using different combination of input order</li>\n<li>use longer token length</li>\n<li>ensemble GBDT stacking</li>\n<li>pooling techniques (mean/max pooling, convolution, LSTM)</li>\n</ul>\n<p>and so on.</p>\n<p>I didn't noticed changing the order or combination of the input to nlp model make the enough difference to the nlp model to be used for ensemble!</p>\n<h1>Other</h1>\n<p>I noticed that deberta sequence classification model implementation uses 'type_ids' argument instead of 'token_type_ids' argument.<br>\nThis seems to be one of the cause that including the prompt_text to the deberta input harms the model performance when you use huggingface Trainer and AutoTokenizer combination.<br>\nMaybe the SEP token is not enough for the model to distinguish where to score the content and wording.<br>\nYou can solve this issue by changing the returned key name by tokenizer from token_type_ids to type_ids.</p>\n<p><a href=\"https://github.com/microsoft/DeBERTa/blob/master/DeBERTa/apps/models/sequence_classification.py\" target=\"_blank\">https://github.com/microsoft/DeBERTa/blob/master/DeBERTa/apps/models/sequence_classification.py</a></p>",
      "rawMarkdown": "# Solution\nMy best submission was almost same as this public code.\n(https://www.kaggle.com/code/tsunotsuno/debertav3-lgbm-no-autocorrect)\nI appreciate @tsunotsuno to share the sophisticated notebooks!\n\nI changed the deberta-v3-base to deberta-v3-large.\nTraining was done on google colab notebook.\n\n# What I tried but didn't work\n- MLM training for summary texts\n- summary_text + prompt_question +prompt_text training.\n- feed manual features made in the above public notebook to the nlp header.\n\nThe fact that training set and test set have no common prompt_id and the training set have only 4 prompt_ids (it seems not enough for the model to learn the interaction pattern between prompts and summaries) seemed to be the most important factor in this competition to me.\n\nSo I thought it might be efficient to teach the model the interaction of the prompt and summary using the test set, because we have only 4 prompts in our training set but many prompts in test set. So I tried deberta-v3-base MLM training on test set before regression training, but it didn't work.\n\nThinking the way to tackle this point wasted me the most of the time I consumed in this competition...\nI eventually selected the deberta-v3-large simple solution but there were potential to be improved...\n\n# What I should have done\n- ensemble nlp models using different combination of input order\n- use longer token length\n- ensemble GBDT stacking\n- pooling techniques (mean/max pooling, convolution, LSTM)\n\nand so on.\n\nI didn't noticed changing the order or combination of the input to nlp model make the enough difference to the nlp model to be used for ensemble!\n\n# Other\nI noticed that deberta sequence classification model implementation uses 'type_ids' argument instead of 'token_type_ids' argument.\nThis seems to be one of the cause that including the prompt_text to the deberta input harms the model performance when you use huggingface Trainer and AutoTokenizer combination.\nMaybe the SEP token is not enough for the model to distinguish where to score the content and wording.\nYou can solve this issue by changing the returned key name by tokenizer from token_type_ids to type_ids.\n\nhttps://github.com/microsoft/DeBERTa/blob/master/DeBERTa/apps/models/sequence_classification.py",
      "votes": 3
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:05:12Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10795699,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/nynyny67-108th-solution"
}