{
  "id": 447093,
  "title": "29th Place Solution for the CommonLit - Evaluate Student Summaries Competition",
  "content": "<p>I would like to thank the host for organizing this competition and  all the participants who generously shared valuable information. I have learned a lot.</p>\n<h2>Context</h2>\n<ul>\n<li>Business context: <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/overview\" target=\"_blank\">https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/overview</a> </li>\n<li>Data context: <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/data\" target=\"_blank\">https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/data</a></li>\n</ul>\n<h2>Overview of the approach</h2>\n<p>My approach is averaging the following models.  </p>\n<ul>\n<li>LightGBM stacking with meta-features of 9 deberta and bart models.  </li>\n<li>A deberta-v3-large model for the concatenation of summary text and prompt text. (Exp ID: 10)  <br>\nThe key aspect of the latter model is an attention mask that focuses only on the summary text, as explained by other top solutions.</li>\n</ul>\n<h4>Model Training Settings</h4>\n<p>I trained the deberta models shown in this table.  <br>\nOut of these, 8 models are deberta-v3-large, and 2 model are bart-large.  <br>\nThe bart-large models take prompt text as the encoder input and summary text as the decoder input. (However, I don't think they effectively utilize prompt text information…)  <br>\nCV strategy is Group k fold (group id = prompt_id)  </p>\n<table>\n<thead>\n<tr>\n<th>exp id</th>\n<th>model</th>\n<th>input</th>\n<th>pooling method</th>\n<th>token max length</th>\n<th>epoch</th>\n<th>freezing layer</th>\n<th>loss</th>\n<th>awp</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>deberta-v3-large</td>\n<td>text</td>\n<td>attention</td>\n<td>822</td>\n<td>10</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.542</td>\n<td>0.494</td>\n<td>0.541</td>\n</tr>\n<tr>\n<td>2</td>\n<td>facebook/bart-large</td>\n<td>text, question+prompt_text</td>\n<td>mean</td>\n<td>822</td>\n<td>4</td>\n<td>-</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.555</td>\n<td>0.514</td>\n<td>0.556</td>\n</tr>\n<tr>\n<td>3</td>\n<td>deberta-v3-large-squad2</td>\n<td>text</td>\n<td>attention</td>\n<td>822</td>\n<td>4</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.581</td>\n<td>0.496</td>\n<td>0.549</td>\n</tr>\n<tr>\n<td>4</td>\n<td>deberta-v3-large</td>\n<td>text</td>\n<td>attention</td>\n<td>822</td>\n<td>10</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>true</td>\n<td>0.535</td>\n<td>0.488</td>\n<td>0.532</td>\n</tr>\n<tr>\n<td>5</td>\n<td>bart-large</td>\n<td>text+question, prompt_text</td>\n<td>attention</td>\n<td>822</td>\n<td>10</td>\n<td>-</td>\n<td>smooth l1</td>\n<td>true</td>\n<td>0.533</td>\n<td>0.486</td>\n<td>0.510</td>\n</tr>\n<tr>\n<td>6</td>\n<td>deberta-v3-large</td>\n<td>question+title+text</td>\n<td>attention</td>\n<td>822</td>\n<td>10</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.538</td>\n<td>0.497</td>\n<td>0.563</td>\n</tr>\n<tr>\n<td>7</td>\n<td>deberta-v3-large</td>\n<td>question+title+text</td>\n<td>attention</td>\n<td>512</td>\n<td>4</td>\n<td>-</td>\n<td>smooth l1</td>\n<td>true</td>\n<td>0.628</td>\n<td>0.519</td>\n<td>0.535</td>\n</tr>\n<tr>\n<td>8</td>\n<td>deberta-v3-large</td>\n<td>title+question+text</td>\n<td>multi-scale</td>\n<td>822</td>\n<td>10</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>true</td>\n<td>0.535</td>\n<td>0.477</td>\n<td>0.538</td>\n</tr>\n<tr>\n<td>9</td>\n<td>deberta-v3-large</td>\n<td>text</td>\n<td>attention</td>\n<td>822</td>\n<td>4</td>\n<td>18</td>\n<td>mse + rank</td>\n<td>false</td>\n<td>0.551</td>\n<td>0.487</td>\n<td>0.544</td>\n</tr>\n<tr>\n<td>10</td>\n<td>deberta-v3-large</td>\n<td>text+title+question+prompt_text</td>\n<td><a href=\"#model-for-summary-text-and-prompt-text-input\">See this section</a></td>\n<td>training : 1800, prediction : 1024</td>\n<td>4</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.504</td>\n<td>0.461</td>\n<td>0.47</td>\n</tr>\n<tr>\n<td>Every training utilized layer-wise learning rate decay between 0.8 and 0.9.</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>*Sum of losses calculated in each of the last 3 hidden states.</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>**Two attention pooling layers for content and wording separately.</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>***Pooling at different levels: words, sentences, and paragraphs, then the results are pooled.</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h4>LightGBM stacking with meta features</h4>\n<p>I stacked 9 models (Exp ID: 1 to 9) using LightGBM with the following meta-features:</p>\n<ul>\n<li>Number of characters in the summary text</li>\n<li>Number of characters in the prompt text</li>\n<li>Number of unique words overlapping between summary and prompt text (excluding punctuation)</li>\n<li>Number of overlapping words between summary and prompt text</li>\n<li>Ratio of overlapping bigrams</li>\n<li>Ratio of overlapping 4-grams</li>\n<li>Cosine similarity between TF-IDF values of summary and prompt text</li>\n<li>Mean of edit distances between each summary sentence and prompt sentence</li>\n<li>Number of words in the summary text (excluding punctuation, correcting misspelling)</li>\n<li>Number of unique words overlapping between summary and prompt text (excluding punctuation, correcting misspelling)</li>\n<li>Number of overlapping bigrams between summary and prompt text (excluding punctuation, correcting misspelling)</li>\n<li>Ratio of overlapping bigrams (correcting misspelling)</li>\n<li>Number of similar summary texts (similarity calculated by cosine similarity using Universal Sentence Encoder embedding)</li>\n<li>Readability score of the prompt text<br>\nThis stacking model achieved CV: 0.478.</li>\n</ul>\n<h4>Averaging the Two Models</h4>\n<p>For the final submission, I simply averaged the stacking model and the long-token model (Exp ID: 10), and submitted it. <br>\nThe CV is 0.471, Public LB: 0.433. <br>\nI also tried nelder-mead weight optimization, but CV was not well as the averaged score.</p>\n<h2>Details of the submission</h2>\n<h3>What Worked</h3>\n<h4>Model for Summary Text and Prompt Text Input</h4>\n<p>I initially concatenated the summary text and prompt text and fed them into deberta-v3-large model. However, this didn't lead to an improvement in CV performance.  <br>\nI realized it was necessary to inform the model about the positions of the summary text tokens. Therefore, I prepared attention masks to indicate the positions of the summary text, title, question, and prompt text within the input tokens.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2419980%2F8d48845c845333584ec6044c09f7d33e%2Fattention.drawio.png?generation=1697274325973927&amp;alt=media\" alt=\"\"><br>\nThese masks were used in conjunction with deberta's output. The architecture of this model is described below.<br>\n(I apologize if it seems overly complex. According to other solutions, it appears that using an attention mask solely for the summary text may be sufficient.)<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2419980%2Fcb53a1430da088ad8e964119ff05acbe%2Fdeberta.drawio.png?generation=1697274351083740&amp;alt=media\" alt=\"\"><br>\nThis model was trained with a token length of 1800. However, during prediction, I had to reduce the token length to 1024 due to a 9-hour time limit.  <br>\nThis model (Exp ID: 10) imporoved CV scores, especially challenging prompts, resulting in CV: 0.504, Public LB: 0.461, inference time exceeded 4h~.</p>\n<h3>What didn't Work</h3>\n<h4>CNN Model to the Sentence Similarity Matrix between Summary Text and Prompt Text</h4>\n<p>I noticed that significant errors were caused by summary texts that simply copied and pasted many sentences from their prompt texts. These summaries were given the same low content and wording score. I believe that when scorers come across such a copy-and-paste summary, they tend to stop reading it and automatically assign low scores. Therefore, I wanted to address this issue by calculating the edit distances between each of the N sentences in the summary text and each of the M sentences in the prompt text. I then created an N x M similarity matrix, with each element containing edit distance ratio.  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2419980%2F300454278eb2e87f36d4e263c64cc282%2Fheatmap.png?generation=1697274594179322&amp;alt=media\" alt=\"\">  <br>\nThis matrix was fed into a CNN model, and target values were generated. However, this CNN model was not as effective as anticipated… (this idea was inspired by a method described in <a href=\"https://www.cl.cam.ac.uk/~ek358/Summarization_NAACL.pdf\" target=\"_blank\">this</a> paper).</p>\n<h3>Souces</h3>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369956\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369956</a><ul>\n<li>For information on the multi-scale approach.</li></ul></li>\n<li><a href=\"https://www.cl.cam.ac.uk/~ek358/Summarization_NAACL.pdf\" target=\"_blank\">https://www.cl.cam.ac.uk/~ek358/Summarization_NAACL.pdf</a><ul>\n<li>Reference material for tackling the same task.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering\" target=\"_blank\">https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering</a><ul>\n<li>I borrowed many features from this source. Thank you very much for your contribution!</li></ul></li>\n</ul>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2481618,
      "postDate": "2023-10-14T09:21:24.313Z",
      "content": "<p>I would like to thank the host for organizing this competition and  all the participants who generously shared valuable information. I have learned a lot.</p>\n<h2>Context</h2>\n<ul>\n<li>Business context: <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/overview\" target=\"_blank\">https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/overview</a> </li>\n<li>Data context: <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/data\" target=\"_blank\">https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/data</a></li>\n</ul>\n<h2>Overview of the approach</h2>\n<p>My approach is averaging the following models.  </p>\n<ul>\n<li>LightGBM stacking with meta-features of 9 deberta and bart models.  </li>\n<li>A deberta-v3-large model for the concatenation of summary text and prompt text. (Exp ID: 10)  <br>\nThe key aspect of the latter model is an attention mask that focuses only on the summary text, as explained by other top solutions.</li>\n</ul>\n<h4>Model Training Settings</h4>\n<p>I trained the deberta models shown in this table.  <br>\nOut of these, 8 models are deberta-v3-large, and 2 model are bart-large.  <br>\nThe bart-large models take prompt text as the encoder input and summary text as the decoder input. (However, I don't think they effectively utilize prompt text information…)  <br>\nCV strategy is Group k fold (group id = prompt_id)  </p>\n<table>\n<thead>\n<tr>\n<th>exp id</th>\n<th>model</th>\n<th>input</th>\n<th>pooling method</th>\n<th>token max length</th>\n<th>epoch</th>\n<th>freezing layer</th>\n<th>loss</th>\n<th>awp</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>deberta-v3-large</td>\n<td>text</td>\n<td>attention</td>\n<td>822</td>\n<td>10</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.542</td>\n<td>0.494</td>\n<td>0.541</td>\n</tr>\n<tr>\n<td>2</td>\n<td>facebook/bart-large</td>\n<td>text, question+prompt_text</td>\n<td>mean</td>\n<td>822</td>\n<td>4</td>\n<td>-</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.555</td>\n<td>0.514</td>\n<td>0.556</td>\n</tr>\n<tr>\n<td>3</td>\n<td>deberta-v3-large-squad2</td>\n<td>text</td>\n<td>attention</td>\n<td>822</td>\n<td>4</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.581</td>\n<td>0.496</td>\n<td>0.549</td>\n</tr>\n<tr>\n<td>4</td>\n<td>deberta-v3-large</td>\n<td>text</td>\n<td>attention</td>\n<td>822</td>\n<td>10</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>true</td>\n<td>0.535</td>\n<td>0.488</td>\n<td>0.532</td>\n</tr>\n<tr>\n<td>5</td>\n<td>bart-large</td>\n<td>text+question, prompt_text</td>\n<td>attention</td>\n<td>822</td>\n<td>10</td>\n<td>-</td>\n<td>smooth l1</td>\n<td>true</td>\n<td>0.533</td>\n<td>0.486</td>\n<td>0.510</td>\n</tr>\n<tr>\n<td>6</td>\n<td>deberta-v3-large</td>\n<td>question+title+text</td>\n<td>attention</td>\n<td>822</td>\n<td>10</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.538</td>\n<td>0.497</td>\n<td>0.563</td>\n</tr>\n<tr>\n<td>7</td>\n<td>deberta-v3-large</td>\n<td>question+title+text</td>\n<td>attention</td>\n<td>512</td>\n<td>4</td>\n<td>-</td>\n<td>smooth l1</td>\n<td>true</td>\n<td>0.628</td>\n<td>0.519</td>\n<td>0.535</td>\n</tr>\n<tr>\n<td>8</td>\n<td>deberta-v3-large</td>\n<td>title+question+text</td>\n<td>multi-scale</td>\n<td>822</td>\n<td>10</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>true</td>\n<td>0.535</td>\n<td>0.477</td>\n<td>0.538</td>\n</tr>\n<tr>\n<td>9</td>\n<td>deberta-v3-large</td>\n<td>text</td>\n<td>attention</td>\n<td>822</td>\n<td>4</td>\n<td>18</td>\n<td>mse + rank</td>\n<td>false</td>\n<td>0.551</td>\n<td>0.487</td>\n<td>0.544</td>\n</tr>\n<tr>\n<td>10</td>\n<td>deberta-v3-large</td>\n<td>text+title+question+prompt_text</td>\n<td><a href=\"#model-for-summary-text-and-prompt-text-input\">See this section</a></td>\n<td>training : 1800, prediction : 1024</td>\n<td>4</td>\n<td>18</td>\n<td>smooth l1</td>\n<td>false</td>\n<td>0.504</td>\n<td>0.461</td>\n<td>0.47</td>\n</tr>\n<tr>\n<td>Every training utilized layer-wise learning rate decay between 0.8 and 0.9.</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>*Sum of losses calculated in each of the last 3 hidden states.</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>**Two attention pooling layers for content and wording separately.</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>***Pooling at different levels: words, sentences, and paragraphs, then the results are pooled.</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h4>LightGBM stacking with meta features</h4>\n<p>I stacked 9 models (Exp ID: 1 to 9) using LightGBM with the following meta-features:</p>\n<ul>\n<li>Number of characters in the summary text</li>\n<li>Number of characters in the prompt text</li>\n<li>Number of unique words overlapping between summary and prompt text (excluding punctuation)</li>\n<li>Number of overlapping words between summary and prompt text</li>\n<li>Ratio of overlapping bigrams</li>\n<li>Ratio of overlapping 4-grams</li>\n<li>Cosine similarity between TF-IDF values of summary and prompt text</li>\n<li>Mean of edit distances between each summary sentence and prompt sentence</li>\n<li>Number of words in the summary text (excluding punctuation, correcting misspelling)</li>\n<li>Number of unique words overlapping between summary and prompt text (excluding punctuation, correcting misspelling)</li>\n<li>Number of overlapping bigrams between summary and prompt text (excluding punctuation, correcting misspelling)</li>\n<li>Ratio of overlapping bigrams (correcting misspelling)</li>\n<li>Number of similar summary texts (similarity calculated by cosine similarity using Universal Sentence Encoder embedding)</li>\n<li>Readability score of the prompt text<br>\nThis stacking model achieved CV: 0.478.</li>\n</ul>\n<h4>Averaging the Two Models</h4>\n<p>For the final submission, I simply averaged the stacking model and the long-token model (Exp ID: 10), and submitted it. <br>\nThe CV is 0.471, Public LB: 0.433. <br>\nI also tried nelder-mead weight optimization, but CV was not well as the averaged score.</p>\n<h2>Details of the submission</h2>\n<h3>What Worked</h3>\n<h4>Model for Summary Text and Prompt Text Input</h4>\n<p>I initially concatenated the summary text and prompt text and fed them into deberta-v3-large model. However, this didn't lead to an improvement in CV performance.  <br>\nI realized it was necessary to inform the model about the positions of the summary text tokens. Therefore, I prepared attention masks to indicate the positions of the summary text, title, question, and prompt text within the input tokens.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2419980%2F8d48845c845333584ec6044c09f7d33e%2Fattention.drawio.png?generation=1697274325973927&amp;alt=media\" alt=\"\"><br>\nThese masks were used in conjunction with deberta's output. The architecture of this model is described below.<br>\n(I apologize if it seems overly complex. According to other solutions, it appears that using an attention mask solely for the summary text may be sufficient.)<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2419980%2Fcb53a1430da088ad8e964119ff05acbe%2Fdeberta.drawio.png?generation=1697274351083740&amp;alt=media\" alt=\"\"><br>\nThis model was trained with a token length of 1800. However, during prediction, I had to reduce the token length to 1024 due to a 9-hour time limit.  <br>\nThis model (Exp ID: 10) imporoved CV scores, especially challenging prompts, resulting in CV: 0.504, Public LB: 0.461, inference time exceeded 4h~.</p>\n<h3>What didn't Work</h3>\n<h4>CNN Model to the Sentence Similarity Matrix between Summary Text and Prompt Text</h4>\n<p>I noticed that significant errors were caused by summary texts that simply copied and pasted many sentences from their prompt texts. These summaries were given the same low content and wording score. I believe that when scorers come across such a copy-and-paste summary, they tend to stop reading it and automatically assign low scores. Therefore, I wanted to address this issue by calculating the edit distances between each of the N sentences in the summary text and each of the M sentences in the prompt text. I then created an N x M similarity matrix, with each element containing edit distance ratio.  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2419980%2F300454278eb2e87f36d4e263c64cc282%2Fheatmap.png?generation=1697274594179322&amp;alt=media\" alt=\"\">  <br>\nThis matrix was fed into a CNN model, and target values were generated. However, this CNN model was not as effective as anticipated… (this idea was inspired by a method described in <a href=\"https://www.cl.cam.ac.uk/~ek358/Summarization_NAACL.pdf\" target=\"_blank\">this</a> paper).</p>\n<h3>Souces</h3>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369956\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369956</a><ul>\n<li>For information on the multi-scale approach.</li></ul></li>\n<li><a href=\"https://www.cl.cam.ac.uk/~ek358/Summarization_NAACL.pdf\" target=\"_blank\">https://www.cl.cam.ac.uk/~ek358/Summarization_NAACL.pdf</a><ul>\n<li>Reference material for tackling the same task.</li></ul></li>\n<li><a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering\" target=\"_blank\">https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering</a><ul>\n<li>I borrowed many features from this source. Thank you very much for your contribution!</li></ul></li>\n</ul>",
      "rawMarkdown": " I would like to thank the host for organizing this competition and  all the participants who generously shared valuable information. I have learned a lot.\n\n\n## Context\n- Business context: https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/overview \n- Data context: https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/data\n\n## Overview of the approach\nMy approach is averaging the following models.  \n- LightGBM stacking with meta-features of 9 deberta and bart models.  \n- A deberta-v3-large model for the concatenation of summary text and prompt text. (Exp ID: 10)  \n\nThe key aspect of the latter model is an attention mask that focuses only on the summary text, as explained by other top solutions.\n\n#### Model Training Settings\n\nI trained the deberta models shown in this table.  \nOut of these, 8 models are deberta-v3-large, and 2 model are bart-large.  \nThe bart-large models take prompt text as the encoder input and summary text as the decoder input. (However, I don't think they effectively utilize prompt text information...)  \nCV strategy is Group k fold (group id = prompt_id)  \n\n| exp id | model | input | pooling method | token max length | epoch | freezing layer | loss | awp | CV | Public LB | Private LB |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| 1 | deberta-v3-large | text | attention | 822 | 10 | 18 | smooth l1<sup>*</sup> | false | 0.542 | 0.494 | 0.541 |\n| 2 | facebook/bart-large | text, question+prompt_text  | mean | 822 | 4 | - | smooth l1 | false | 0.555 | 0.514 | 0.556 |\n| 3 | deberta-v3-large-squad2 | text | attention | 822 | 4 | 18 | smooth l1<sup>*</sup> | false | 0.581 | 0.496 | 0.549 | \n| 4 | deberta-v3-large | text | attention | 822 | 10 | 18 | smooth l1<sup>*</sup> | true | 0.535 | 0.488 | 0.532 | \n| 5 | bart-large | text+question, prompt_text | attention | 822 | 10 | - | smooth l1 | true | 0.533 | 0.486 | 0.510 |\n| 6 | deberta-v3-large | question+title+text | attention | 822 | 10 | 18 | smooth l1 | false | 0.538 | 0.497 | 0.563 | \n| 7 | deberta-v3-large | question+title+text | attention<sup>**</sup> | 512 | 4 | - | smooth l1 | true | 0.628 | 0.519 | 0.535 | \n| 8 | deberta-v3-large | title+question+text | multi-scale<sup>***</sup> | 822 | 10 | 18 | smooth l1 | true | 0.535 | 0.477 | 0.538 | \n| 9 | deberta-v3-large | text | attention | 822 | 4 | 18 | mse + rank | false | 0.551 | 0.487 | 0.544 |\n| 10 | deberta-v3-large | text+title+question+prompt_text | [See this section](#model-for-summary-text-and-prompt-text-input) | training : 1800, prediction : 1024 | 4 | 18 | smooth l1 | false | 0.504 | 0.461 | 0.47 |\n\nEvery training utilized layer-wise learning rate decay between 0.8 and 0.9.  \n\\*Sum of losses calculated in each of the last 3 hidden states.  \n\\*\\*Two attention pooling layers for content and wording separately.  \n\\*\\*\\*Pooling at different levels: words, sentences, and paragraphs, then the results are pooled.  \n\n\n\n#### LightGBM stacking with meta features\n\nI stacked 9 models (Exp ID: 1 to 9) using LightGBM with the following meta-features:\n- Number of characters in the summary text\n- Number of characters in the prompt text\n- Number of unique words overlapping between summary and prompt text (excluding punctuation)\n- Number of overlapping words between summary and prompt text\n- Ratio of overlapping bigrams\n- Ratio of overlapping 4-grams\n- Cosine similarity between TF-IDF values of summary and prompt text\n- Mean of edit distances between each summary sentence and prompt sentence\n- Number of words in the summary text (excluding punctuation, correcting misspelling)\n- Number of unique words overlapping between summary and prompt text (excluding punctuation, correcting misspelling)\n- Number of overlapping bigrams between summary and prompt text (excluding punctuation, correcting misspelling)\n- Ratio of overlapping bigrams (correcting misspelling)\n- Number of similar summary texts (similarity calculated by cosine similarity using Universal Sentence Encoder embedding)\n- Readability score of the prompt text\n\n\nThis stacking model achieved CV: 0.478.\n\n\n#### Averaging the Two Models\nFor the final submission, I simply averaged the stacking model and the long-token model (Exp ID: 10), and submitted it. \nThe CV is 0.471, Public LB: 0.433. \nI also tried nelder-mead weight optimization, but CV was not well as the averaged score.\n\n\n## Details of the submission\n\n### What Worked\n\n#### Model for Summary Text and Prompt Text Input\n\nI initially concatenated the summary text and prompt text and fed them into deberta-v3-large model. However, this didn't lead to an improvement in CV performance.  \nI realized it was necessary to inform the model about the positions of the summary text tokens. Therefore, I prepared attention masks to indicate the positions of the summary text, title, question, and prompt text within the input tokens.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2419980%2F8d48845c845333584ec6044c09f7d33e%2Fattention.drawio.png?generation=1697274325973927&alt=media)\n\nThese masks were used in conjunction with deberta's output. The architecture of this model is described below.\n(I apologize if it seems overly complex. According to other solutions, it appears that using an attention mask solely for the summary text may be sufficient.)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2419980%2Fcb53a1430da088ad8e964119ff05acbe%2Fdeberta.drawio.png?generation=1697274351083740&alt=media)\n\nThis model was trained with a token length of 1800. However, during prediction, I had to reduce the token length to 1024 due to a 9-hour time limit.  \nThis model (Exp ID: 10) imporoved CV scores, especially challenging prompts, resulting in CV: 0.504, Public LB: 0.461, inference time exceeded 4h~.\n\n### What didn't Work\n\n#### CNN Model to the Sentence Similarity Matrix between Summary Text and Prompt Text\nI noticed that significant errors were caused by summary texts that simply copied and pasted many sentences from their prompt texts. These summaries were given the same low content and wording score. I believe that when scorers come across such a copy-and-paste summary, they tend to stop reading it and automatically assign low scores. Therefore, I wanted to address this issue by calculating the edit distances between each of the N sentences in the summary text and each of the M sentences in the prompt text. I then created an N x M similarity matrix, with each element containing edit distance ratio.  \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2419980%2F300454278eb2e87f36d4e263c64cc282%2Fheatmap.png?generation=1697274594179322&alt=media \"edit distance ratio similarity matrix of low score summary sentences to prompt text sentences\")  \nThis matrix was fed into a CNN model, and target values were generated. However, this CNN model was not as effective as anticipated... (this idea was inspired by a method described in [this](https://www.cl.cam.ac.uk/~ek358/Summarization_NAACL.pdf) paper).\n\n\n\n### Souces\n- https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369956\n    - For information on the multi-scale approach.\n- https://www.cl.cam.ac.uk/~ek358/Summarization_NAACL.pdf\n    - Reference material for tackling the same task.\n- https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering\n    - I borrowed many features from this source. Thank you very much for your contribution!\n\n\n",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:06:04Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10675265,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/yusuke-nakayama-29th-place-solution-for-the-common"
}