{
  "id": 446512,
  "title": "19th place solution",
  "content": "<p>I'd like to start by expressing my gratitude to the organizers for hosting this competition and inspiring engaging discussions.<br>\nA special congratulations to the top participants for their outstanding achievements.<br>\nPersonally, I was on the cusp of becoming a GM with a gold medal, and it's a bit disappointing to have come so close yet fall short. Nevertheless, your hard work and insights have truly made this a memorable event for everyone. <br>\nThank you all for making it such a rewarding experience.</p>\n<p>And also thank you for my teammate takai380. </p>\n<h2>Overview</h2>\n<ul>\n<li>A two-stage structure with the 1st stage being a Transformer and the 2nd stage being a GBDT (LightGBM).</li>\n<li>For the 2nd stage GBDT, a model is created for each out-of-fold (oof) of the Transformer, and finally, a Weighted Average is taken.<ul>\n<li>Some raw prediction values from the Transformers are also included in the final Weighted Average.</li></ul></li>\n<li>Adjust the weight according to the total token count of summary_text and prompt_text.</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1305588%2F9c362ac762756babd02f63e96e36e971%2Foverview.png?generation=1697077612844422&amp;alt=media\" alt=\"overview\"></p>\n<h2>Final Model</h2>\n<p>Three models with different features and weights were selected for the final submission:</p>\n<ul>\n<li>Sub1 (TrustLB)  LB:0.422 / CV:0.463-0.465 (Different CV calculation method)</li>\n<li>Sub2 (TrustCV)  LB:0.448 / CV:0.458</li>\n<li>Sub3 (Balance)  LB:0.441 / CV:0.460</li>\n</ul>\n<h2>Transformer</h2>\n<ul>\n<li>pao<ul>\n<li>model-p1: deberta-v3-large (question + summary_text)</li>\n<li>model-p2: deberta-v2-xlarge (question + summary_text)</li></ul></li>\n<li>takai<ul>\n<li>model-t1: deberta-v3-large (question + summary_text)</li>\n<li>model-t2: deberta-v3-large (question + summary_text)</li>\n<li>model-t3: deberta-v3-large (summary_text + prompt_text)</li>\n<li>model-t4: deberta-v3-large (summary_text + question + prompt_text)</li></ul></li>\n</ul>\n<p>Predictions from model-t3 and t4 are also used in the final ensemble.<br>\nModel with prompt_text is strong, but it takes long time to inference, so just 2 models.</p>\n<ul>\n<li>For pao model:<ul>\n<li>Freezing layer</li>\n<li>Concatenate the embedding of deberta-v3-large before training to the final layer (only for model-p1)</li>\n<li>Prediction using CLS token</li>\n<li>Linear scheduler with warmup, 3 epochs</li></ul></li>\n<li>For takai model'<ul>\n<li>Freezing layer</li>\n<li>concat pooling (concat multiple layer cls tokens)</li></ul></li>\n</ul>\n<h2>LightGBM</h2>\n<ul>\n<li>Ultimately, about 80 features.</li>\n<li>10 seeds x 4 folds of LightGBM Average for each Transformer model.</li>\n<li>To add features, check if the CV is improved with a score from a 50 seed average.</li>\n</ul>\n<h3>Features</h3>\n<ul>\n<li>From public notebooks: (great thanks to <a href=\"https://www.kaggle.com/nogawanogawa\" target=\"_blank\">@nogawanogawa</a>)<ul>\n<li><a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering\" target=\"_blank\">https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering</a></li></ul></li>\n<li>Jaccard coefficient between question/prompt_text and summary_text.</li>\n<li>Calculate tfidf for summary_text per prompt, and take the average tfidf per record.<ul>\n<li>For all of summary_text and only words not in prompt_text.</li>\n<li>Taking average of all columns and average of non-zero elements.</li></ul></li>\n<li>Cosine similarity of average tfidf/BoW of summary_text per prompt.</li>\n<li>Sentence Transformer embedding features.<ul>\n<li>Cosine similarity between prompt_text and summary_text.</li>\n<li>Cosine similarity between question and summary_text.</li>\n<li>Cosine similarity between average embedding of summary_text per prompt.</li>\n<li>Calculate cosine similarity between embeddings of sentences split from prompt_text and summary_text, and take the standard deviation of similarity for all sentences.</li>\n<li>Similarly, calculate cosine similarity between embeddings of sentences split from summary_text and prompt_text, and take the standard deviation of similarity for all sentences.</li></ul></li>\n<li>kNN-based features.<ul>\n<li>For each prompt, kNN with the embedding of summary_text.</li>\n<li>Feature extraction of the average and cosine similarity of the top 5% of oof features with high similarity for each record.</li>\n<li>The average of oof features also includes the difference from the average oof features per prompt.</li></ul></li>\n<li>Text similarity metrics:<ul>\n<li>BERT Score</li>\n<li>ROUGE</li>\n<li>BLEU</li></ul></li>\n<li>Learning Word2Vec with prompt_text for each prompt. Cosine similarity of the vectors of prompt_text and summary_text.</li>\n<li>Include prediction values from the model of Feedback competition 3 as features.<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference?scriptVersionId=104649006\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference?scriptVersionId=104649006</a></li></ul></li>\n<li>Features from pyreadability.</li>\n<li>Average the general frequency of words present only in summary_text and not in prompt_text for each summary_text.<ul>\n<li>For bert_score and embedding features, the average embedding was taken from extracting text from the beginning, middle, and end for long prompts.</li></ul></li>\n</ul>\n<h2>Ensemble</h2>\n<ul>\n<li>Weight optimization using Nelder-Mead.<ul>\n<li>Ultimately, only use those above 0 and round to a total of 1.0.</li></ul></li>\n<li>Change ensemble weights according to the token count of summary_text + prompt_text.<ul>\n<li>For those with a certain number of tokens or more, gradually reduce the weight of raw Transformer predictions and increase the weight of LightGBM (linearly adjust the weight by token count).</li>\n<li>It is assumed that when the token count is high, the prompt_text does not fit entirely in the Transformer with prompt_text input, reducing accuracy.</li></ul></li>\n</ul>",
  "comments": [
    {
      "id": 2478588,
      "author_name": "Anil Reddy",
      "author_url": "",
      "post_date": "2023-10-12T05:12:16.010000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/go5kuramubon\" target=\"_blank\">@go5kuramubon</a>, what is the difference between models t1, t2 and between t3, t4?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2478635,
          "author_name": "takai380",
          "author_url": "",
          "post_date": "2023-10-12T05:56:32.673000",
          "content": "<p>Difference of t1 and t2 is pooling<br>\nt1=&gt;weighted layer pooling (layer = 4)<br>\nt2=&gt;concat pooling (layer = 10)</p>\n<p>Difference of t1 and t2 is input<br>\nt3=&gt; train['text']+' [SEP] source: '+train['prompt_text']<br>\nt4=&gt; '[SEP] question: '+train['prompt_question']+' [SEP] summary: '+train['text']+' [SEP] source: '+train['prompt_text']</p>",
          "votes": 2,
          "replies": [
            {
              "id": 2478641,
              "author_name": "Anil Reddy",
              "author_url": "",
              "post_date": "2023-10-12T06:02:15.150000",
              "content": "<p>Congratulations to you <a href=\"https://www.kaggle.com/takai380\" target=\"_blank\">@takai380</a> as well</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2480571,
      "author_name": "DL",
      "author_url": "",
      "post_date": "2023-10-13T12:10:41.550000",
      "content": "<p>Thanks for sharing, look forward to your code sharing if possible.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2479385,
      "author_name": "etcnet",
      "author_url": "",
      "post_date": "2023-10-12T15:37:44.330000",
      "content": "<p>Thank you for sharing your solution with us!</p>\n<p>Your idea of tweaking transformer/LGBM weighting based on token counts is quite clever.</p>\n<p>May I ask, how did you split your data? Did you use groupkfold, stratified fold, or some other method?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2479393,
          "author_name": "takai380",
          "author_url": "",
          "post_date": "2023-10-12T15:44:17.453000",
          "content": "<p>group-k-fold by prompt_id</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 6,
  "messages": [
    {
      "id": 2478459,
      "postDate": "2023-10-12T03:08:56.473Z",
      "content": "<p>I'd like to start by expressing my gratitude to the organizers for hosting this competition and inspiring engaging discussions.<br>\nA special congratulations to the top participants for their outstanding achievements.<br>\nPersonally, I was on the cusp of becoming a GM with a gold medal, and it's a bit disappointing to have come so close yet fall short. Nevertheless, your hard work and insights have truly made this a memorable event for everyone. <br>\nThank you all for making it such a rewarding experience.</p>\n<p>And also thank you for my teammate takai380. </p>\n<h2>Overview</h2>\n<ul>\n<li>A two-stage structure with the 1st stage being a Transformer and the 2nd stage being a GBDT (LightGBM).</li>\n<li>For the 2nd stage GBDT, a model is created for each out-of-fold (oof) of the Transformer, and finally, a Weighted Average is taken.<ul>\n<li>Some raw prediction values from the Transformers are also included in the final Weighted Average.</li></ul></li>\n<li>Adjust the weight according to the total token count of summary_text and prompt_text.</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1305588%2F9c362ac762756babd02f63e96e36e971%2Foverview.png?generation=1697077612844422&amp;alt=media\" alt=\"overview\"></p>\n<h2>Final Model</h2>\n<p>Three models with different features and weights were selected for the final submission:</p>\n<ul>\n<li>Sub1 (TrustLB)  LB:0.422 / CV:0.463-0.465 (Different CV calculation method)</li>\n<li>Sub2 (TrustCV)  LB:0.448 / CV:0.458</li>\n<li>Sub3 (Balance)  LB:0.441 / CV:0.460</li>\n</ul>\n<h2>Transformer</h2>\n<ul>\n<li>pao<ul>\n<li>model-p1: deberta-v3-large (question + summary_text)</li>\n<li>model-p2: deberta-v2-xlarge (question + summary_text)</li></ul></li>\n<li>takai<ul>\n<li>model-t1: deberta-v3-large (question + summary_text)</li>\n<li>model-t2: deberta-v3-large (question + summary_text)</li>\n<li>model-t3: deberta-v3-large (summary_text + prompt_text)</li>\n<li>model-t4: deberta-v3-large (summary_text + question + prompt_text)</li></ul></li>\n</ul>\n<p>Predictions from model-t3 and t4 are also used in the final ensemble.<br>\nModel with prompt_text is strong, but it takes long time to inference, so just 2 models.</p>\n<ul>\n<li>For pao model:<ul>\n<li>Freezing layer</li>\n<li>Concatenate the embedding of deberta-v3-large before training to the final layer (only for model-p1)</li>\n<li>Prediction using CLS token</li>\n<li>Linear scheduler with warmup, 3 epochs</li></ul></li>\n<li>For takai model'<ul>\n<li>Freezing layer</li>\n<li>concat pooling (concat multiple layer cls tokens)</li></ul></li>\n</ul>\n<h2>LightGBM</h2>\n<ul>\n<li>Ultimately, about 80 features.</li>\n<li>10 seeds x 4 folds of LightGBM Average for each Transformer model.</li>\n<li>To add features, check if the CV is improved with a score from a 50 seed average.</li>\n</ul>\n<h3>Features</h3>\n<ul>\n<li>From public notebooks: (great thanks to <a href=\"https://www.kaggle.com/nogawanogawa\" target=\"_blank\">@nogawanogawa</a>)<ul>\n<li><a href=\"https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering\" target=\"_blank\">https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering</a></li></ul></li>\n<li>Jaccard coefficient between question/prompt_text and summary_text.</li>\n<li>Calculate tfidf for summary_text per prompt, and take the average tfidf per record.<ul>\n<li>For all of summary_text and only words not in prompt_text.</li>\n<li>Taking average of all columns and average of non-zero elements.</li></ul></li>\n<li>Cosine similarity of average tfidf/BoW of summary_text per prompt.</li>\n<li>Sentence Transformer embedding features.<ul>\n<li>Cosine similarity between prompt_text and summary_text.</li>\n<li>Cosine similarity between question and summary_text.</li>\n<li>Cosine similarity between average embedding of summary_text per prompt.</li>\n<li>Calculate cosine similarity between embeddings of sentences split from prompt_text and summary_text, and take the standard deviation of similarity for all sentences.</li>\n<li>Similarly, calculate cosine similarity between embeddings of sentences split from summary_text and prompt_text, and take the standard deviation of similarity for all sentences.</li></ul></li>\n<li>kNN-based features.<ul>\n<li>For each prompt, kNN with the embedding of summary_text.</li>\n<li>Feature extraction of the average and cosine similarity of the top 5% of oof features with high similarity for each record.</li>\n<li>The average of oof features also includes the difference from the average oof features per prompt.</li></ul></li>\n<li>Text similarity metrics:<ul>\n<li>BERT Score</li>\n<li>ROUGE</li>\n<li>BLEU</li></ul></li>\n<li>Learning Word2Vec with prompt_text for each prompt. Cosine similarity of the vectors of prompt_text and summary_text.</li>\n<li>Include prediction values from the model of Feedback competition 3 as features.<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference?scriptVersionId=104649006\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference?scriptVersionId=104649006</a></li></ul></li>\n<li>Features from pyreadability.</li>\n<li>Average the general frequency of words present only in summary_text and not in prompt_text for each summary_text.<ul>\n<li>For bert_score and embedding features, the average embedding was taken from extracting text from the beginning, middle, and end for long prompts.</li></ul></li>\n</ul>\n<h2>Ensemble</h2>\n<ul>\n<li>Weight optimization using Nelder-Mead.<ul>\n<li>Ultimately, only use those above 0 and round to a total of 1.0.</li></ul></li>\n<li>Change ensemble weights according to the token count of summary_text + prompt_text.<ul>\n<li>For those with a certain number of tokens or more, gradually reduce the weight of raw Transformer predictions and increase the weight of LightGBM (linearly adjust the weight by token count).</li>\n<li>It is assumed that when the token count is high, the prompt_text does not fit entirely in the Transformer with prompt_text input, reducing accuracy.</li></ul></li>\n</ul>",
      "rawMarkdown": "I'd like to start by expressing my gratitude to the organizers for hosting this competition and inspiring engaging discussions.\nA special congratulations to the top participants for their outstanding achievements.\nPersonally, I was on the cusp of becoming a GM with a gold medal, and it's a bit disappointing to have come so close yet fall short. Nevertheless, your hard work and insights have truly made this a memorable event for everyone. \nThank you all for making it such a rewarding experience.\n\nAnd also thank you for my teammate takai380. \n\n## Overview\n\n- A two-stage structure with the 1st stage being a Transformer and the 2nd stage being a GBDT (LightGBM).\n- For the 2nd stage GBDT, a model is created for each out-of-fold (oof) of the Transformer, and finally, a Weighted Average is taken.\n    - Some raw prediction values from the Transformers are also included in the final Weighted Average.\n- Adjust the weight according to the total token count of summary_text and prompt_text.\n\n![overview](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1305588%2F9c362ac762756babd02f63e96e36e971%2Foverview.png?generation=1697077612844422&alt=media)\n\n## Final Model\n\nThree models with different features and weights were selected for the final submission:\n\n- Sub1 (TrustLB)  LB:0.422 / CV:0.463-0.465 (Different CV calculation method)\n- Sub2 (TrustCV)  LB:0.448 / CV:0.458\n- Sub3 (Balance)  LB:0.441 / CV:0.460\n\n## Transformer\n\n- pao\n  - model-p1: deberta-v3-large (question + summary_text)\n  - model-p2: deberta-v2-xlarge (question + summary_text)\n- takai\n  - model-t1: deberta-v3-large (question + summary_text)\n  - model-t2: deberta-v3-large (question + summary_text)\n  - model-t3: deberta-v3-large (summary_text + prompt_text)\n  - model-t4: deberta-v3-large (summary_text + question + prompt_text)\n\nPredictions from model-t3 and t4 are also used in the final ensemble.\nModel with prompt_text is strong, but it takes long time to inference, so just 2 models.\n\n- For pao model:\n  - Freezing layer\n  - Concatenate the embedding of deberta-v3-large before training to the final layer (only for model-p1)\n  - Prediction using CLS token\n  - Linear scheduler with warmup, 3 epochs\n- For takai model'\n  - Freezing layer\n  - concat pooling (concat multiple layer cls tokens)\n\n## LightGBM\n\n- Ultimately, about 80 features.\n- 10 seeds x 4 folds of LightGBM Average for each Transformer model.\n- To add features, check if the CV is improved with a score from a 50 seed average.\n\n\n### Features\n  - From public notebooks: (great thanks to @nogawanogawa)\n    - https://www.kaggle.com/code/tsunotsuno/updated-debertav3-lgbm-with-feature-engineering\n  - Jaccard coefficient between question/prompt_text and summary_text.\n  - Calculate tfidf for summary_text per prompt, and take the average tfidf per record.\n    - For all of summary_text and only words not in prompt_text.\n    - Taking average of all columns and average of non-zero elements.\n  - Cosine similarity of average tfidf/BoW of summary_text per prompt.\n  - Sentence Transformer embedding features.\n    - Cosine similarity between prompt_text and summary_text.\n    - Cosine similarity between question and summary_text.\n    - Cosine similarity between average embedding of summary_text per prompt.\n    - Calculate cosine similarity between embeddings of sentences split from prompt_text and summary_text, and take the standard deviation of similarity for all sentences.\n    - Similarly, calculate cosine similarity between embeddings of sentences split from summary_text and prompt_text, and take the standard deviation of similarity for all sentences.\n  - kNN-based features.\n    - For each prompt, kNN with the embedding of summary_text.\n    - Feature extraction of the average and cosine similarity of the top 5% of oof features with high similarity for each record.\n    - The average of oof features also includes the difference from the average oof features per prompt.\n  - Text similarity metrics:\n    - BERT Score\n    - ROUGE\n    - BLEU\n  - Learning Word2Vec with prompt_text for each prompt. Cosine similarity of the vectors of prompt_text and summary_text.\n  - Include prediction values from the model of Feedback competition 3 as features.\n    - https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference?scriptVersionId=104649006\n  - Features from pyreadability.\n  - Average the general frequency of words present only in summary_text and not in prompt_text for each summary_text.\n- For bert_score and embedding features, the average embedding was taken from extracting text from the beginning, middle, and end for long prompts.\n\n## Ensemble\n\n- Weight optimization using Nelder-Mead.\n  - Ultimately, only use those above 0 and round to a total of 1.0.\n- Change ensemble weights according to the token count of summary_text + prompt_text.\n  - For those with a certain number of tokens or more, gradually reduce the weight of raw Transformer predictions and increase the weight of LightGBM (linearly adjust the weight by token count).\n  - It is assumed that when the token count is high, the prompt_text does not fit entirely in the Transformer with prompt_text input, reducing accuracy.\n\n\n",
      "votes": 36
    },
    {
      "id": 2478588,
      "postDate": "2023-10-12T05:12:16.010Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/go5kuramubon\" target=\"_blank\">@go5kuramubon</a>, what is the difference between models t1, t2 and between t3, t4?</p>",
      "rawMarkdown": "Congratulations @go5kuramubon, what is the difference between models t1, t2 and between t3, t4?",
      "votes": 1,
      "replies": [
        {
          "id": 2478635,
          "postDate": "2023-10-12T05:56:32.673Z",
          "content": "<p>Difference of t1 and t2 is pooling<br>\nt1=&gt;weighted layer pooling (layer = 4)<br>\nt2=&gt;concat pooling (layer = 10)</p>\n<p>Difference of t1 and t2 is input<br>\nt3=&gt; train['text']+' [SEP] source: '+train['prompt_text']<br>\nt4=&gt; '[SEP] question: '+train['prompt_question']+' [SEP] summary: '+train['text']+' [SEP] source: '+train['prompt_text']</p>",
          "rawMarkdown": "Difference of t1 and t2 is pooling\nt1=>weighted layer pooling (layer = 4)\nt2=>concat pooling (layer = 10)\n\nDifference of t1 and t2 is input\nt3=> train['text']+' [SEP] source: '+train['prompt_text']\nt4=> '[SEP] question: '+train['prompt_question']+' [SEP] summary: '+train['text']+' [SEP] source: '+train['prompt_text']",
          "votes": 2,
          "replies": [
            {
              "id": 2478641,
              "postDate": "2023-10-12T06:02:15.150Z",
              "content": "<p>Congratulations to you <a href=\"https://www.kaggle.com/takai380\" target=\"_blank\">@takai380</a> as well</p>",
              "rawMarkdown": "Congratulations to you @takai380 as well"
            }
          ]
        }
      ]
    },
    {
      "id": 2480571,
      "postDate": "2023-10-13T12:10:41.550Z",
      "content": "<p>Thanks for sharing, look forward to your code sharing if possible.</p>",
      "rawMarkdown": "Thanks for sharing, look forward to your code sharing if possible."
    },
    {
      "id": 2479385,
      "postDate": "2023-10-12T15:37:44.330Z",
      "content": "<p>Thank you for sharing your solution with us!</p>\n<p>Your idea of tweaking transformer/LGBM weighting based on token counts is quite clever.</p>\n<p>May I ask, how did you split your data? Did you use groupkfold, stratified fold, or some other method?</p>",
      "rawMarkdown": "Thank you for sharing your solution with us!\n\nYour idea of tweaking transformer/LGBM weighting based on token counts is quite clever.\n\nMay I ask, how did you split your data? Did you use groupkfold, stratified fold, or some other method?",
      "replies": [
        {
          "id": 2479393,
          "postDate": "2023-10-12T15:44:17.453Z",
          "content": "<p>group-k-fold by prompt_id</p>",
          "rawMarkdown": "group-k-fold by prompt_id",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:01:09Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10676170,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/takai380-pao-19th-place-solution"
}