{
  "id": 446872,
  "title": "16th place solution",
  "content": "<p>Thanks, Kaggle and <a href=\"https://www.kaggle.com/cookiecutters\" target=\"_blank\">@cookiecutters</a> for this interesting competition. Congratulations to all winners, and gold medalists. I would like to give credit to my talented and hard-working teammates <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> and <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a>.</p>\n<p>Kudos to <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> for their fantastic work on LGBM trick, and all participants of this competition for their sharing and hard work.</p>\n<h1>First words:</h1>\n<ul>\n<li>We think this competition is a perfect example of out-of-distribution forecasting, therefore, we don’t fully trust either CV or LB. Instead, we judge the models by our sense of training stability, partly CV, partly LB, and diversity;</li>\n<li>Therefore, we try to think of as many model architectures, and model training techniques as possible;</li>\n</ul>\n<h1>Model 1 - Transformer + Catboost</h1>\n<h3>Transformer model</h3>\n<ul>\n<li>CV/LB/PB: 0.552/0.488/0.556</li>\n<li>Input: text + [SEP] + prompt_question</li>\n<li>Loss: MCRMSE </li>\n<li>Backbone: deberta-v3-large</li>\n<li>Hyperparams:<ul>\n<li>max_length: 256</li>\n<li>pooling: attention</li>\n<li>random-reinit: 1</li>\n<li>differential learning rate factor: 2.6</li>\n<li>lr: 1e-5</li></ul></li>\n<li>epochs: 4</li>\n<li>evaluate at the end of each epoch</li>\n<li>add additional 2x full-fit checkpoints, it helps to gain 0.003 in PB</li>\n</ul>\n<h3>GBDT - Catboost</h3>\n<ul>\n<li>CV/LB: 0.513/0.443/0.484</li>\n<li>Hyperparams:<ul>\n<li>learning_rate: 0.05</li>\n<li>Cat_features = [“summary_n_sentence”]</li></ul></li>\n<li>Features:<ul>\n<li>summary_length, summary_n_sentence, mean_edit_distance of summary sentences, …</li>\n<li>remove many features like spell errors, POS tag features, …</li>\n<li>removing prompt_length improves CV a bit, but no help in LB/PB</li></ul></li>\n<li>Total inference time: 32mins</li>\n</ul>\n<h1>Model 2 - Span Model</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2F6170216dc252bbe8f8ef098e626a3b4b%2FSpanModel.png?generation=1697201774978961&amp;alt=media\" alt=\"\"></p>\n<ul>\n<li>Reformulated the problem as a span-regression task by randomly grouping N summaries associated with a particular prompt and predicting their scores together </li>\n<li>Predictions were conditioned on a context that consisted of the top 8 sentences from the prompt text as ranked by their importance. </li>\n<li>The importance value of a sentence from the prompt text was determined as the average cosine similarity score between the sentence and all summaries for the prompt</li>\n<li>Used MSE Loss on the context and wording targets</li>\n<li>Discretized the content and wording scores into 16 bins to create auxiliary targets and used ArcFace loss on aux targets</li>\n<li>Used ranking loss among summaries in each example span</li>\n<li>During inference, the random grouping of N summaries provided a TTA-like impact</li>\n<li>Private LB by using 2x full-fit checkpoints: 0.473</li>\n</ul>\n<h1>Model 3 - Search Approach</h1>\n<ul>\n<li>Input text is comprised of a context and the student summary</li>\n<li>Context:<ul>\n<li>Retrieved the top 3 sentences from the prompt text based on cosine similarity scores with the summary. </li>\n<li>Concatenated prompt title and prompt question </li></ul></li>\n<li>Trained the model using BCEWithLogitsLoss<ul>\n<li>Scaled the scores between 0-1 </li>\n<li>Used BCEWithLogitsLoss to compute the loss between scaled scores and logits</li></ul></li>\n<li>Private LB by using 2x full-fit checkpoints: 0.476</li>\n</ul>\n<h1>Model 4 - Long Context</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2F9184846a10fbdd62ab5d384d294d81d4%2FLongContext.png?generation=1697201712777096&amp;alt=media\" alt=\"\"></p>\n<ul>\n<li>Remarks:<ul>\n<li>max_len: 1024</li>\n<li>lr: 2e-5</li>\n<li>weight_decay: 1e-2</li>\n<li>n_epochs: 3</li>\n<li>AWP and EMA after 1 epoch</li>\n<li>evaluate 10 times/epoch</li>\n<li>Inference time: 205 mins (4 folds)</li>\n<li>CV/Public LB/Private LB: 0.499/0.438/0.462</li>\n<li>Backbone: OpenAssistant/reward-model-deberta-v3-large-v2</li>\n<li>We have another model with this structure and its CV/Public LB/Private LB: 0.494/0.448/0.459. Unfortunately, we didn’t include this model in the ensemble</li></ul></li>\n<li>Problems:<ul>\n<li>Inference time is too long due to the long context → not good for ensemble</li></ul></li>\n<li>Code:<ul>\n<li>Training: <a href=\"https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv5e\" target=\"_blank\">https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv5e</a></li>\n<li>Inference: <a href=\"https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v5e-infer\" target=\"_blank\">https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v5e-infer</a> </li></ul></li>\n</ul>\n<h1>Model 5 - Dual Encoder with Transfer Learning</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2Fde86838dc52ff852ab461f461ab2a4b7%2FDualEncoder.png?generation=1697201727858488&amp;alt=media\" alt=\"\"></p>\n<ul>\n<li>Model Ideas:<ul>\n<li>Instead of concatenating text and prompt info (prompt text, question, title) before passing through the backbone, we can concatenate them after the transformer. One benefit is that many summaries share the same prompts, we can compute them only once and put them into a hashing table, once the embedding of a prompt is computed, we can just query it instead of computing it again.</li>\n<li>However, I think this way is not good in terms of performance because there is no connection between text and prompt (they’re processed separately in spite of being processed by the same backbone). To overcome it, I load the trained backbone from the long-context model (trained with concatenated text and prompt, with a long sequence length), freeze the top 12 layers, and train the second stage model on top.</li></ul></li>\n<li>Remarks:<ul>\n<li>max_len context: 1536</li>\n<li>max_len text: 512</li>\n<li>n_epochs: 1</li>\n<li>AWP and EMA enabled after 50 steps</li>\n<li>Use LGBM trick</li>\n<li>Inference time: 39 mins (4 folds)</li>\n<li>CV/Public LB/Private LB: 0.500/0.445/0.470</li>\n<li>Backbone: OpenAssistant/reward-model-deberta-v3-large-v2</li></ul></li>\n<li>Code:<ul>\n<li>Training: <a href=\"https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv6c\" target=\"_blank\">https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv6c</a> </li>\n<li>Inference: <a href=\"https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v6c-infer\" target=\"_blank\">https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v6c-infer</a> </li></ul></li>\n</ul>\n<h1>Final ensemble</h1>\n<p>We make ensembles of these above models and the public models by <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> (we actually train it by ourselves rather than get the model from them directly). We observe a super messy correlation between CV and Public LB here, at the ensemble phase. For example, we simply took the average of the component models, the weights were optimized with some constraints to deal with overfitting.</p>\n<h1>What didn’t work</h1>\n<ul>\n<li>We tried to generate synthetic data from LLMs (T5 and Llama), but it didn’t work at all;</li>\n<li>Training with LoRA for target-wise models worked and got CV 0.53+, but didn’t help in the local ensemble;</li>\n</ul>\n<h1>Team member</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a></li>\n<li><a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a></li>\n<li><a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2F1ff6a8913b16a36c1474d8418353be2a%2Fteam.PNG?generation=1697202450769081&amp;alt=media\" alt=\"\"></li>\n</ul>",
  "comments": [
    {
      "id": 2482723,
      "author_name": "DeepUnderstanding",
      "author_url": "",
      "post_date": "2023-10-15T08:24:14.870000",
      "content": "<p>Very innovative approaches! My favourite is Model 2.<br>\nAlthough could you explain what are auxiliary targets really and what exactly is ranking loss, like what is the ground truth used</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2482804,
          "author_name": "Minh Tri Phan",
          "author_url": "",
          "post_date": "2023-10-15T09:37:10.200000",
          "content": "<p>Thanks, this model is developed by Raja. <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a>, Could you please explain the model to him?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2483872,
          "author_name": "Raja Biswas",
          "author_url": "",
          "post_date": "2023-10-16T04:08:41.713000",
          "content": "<p>Many thanks for the questions. <br>\n<code>auxiliary targets</code>: It's basically a mapping between raw scores and bin index. For examples: <code>content &lt; -2.0</code> is mapped to 0, <code>-2.0 &lt; content &lt;  - 1.75</code> is mapped to 1 …<br>\n<code>ranking loss</code> is defined as below:</p>\n<pre><code> ():\n    labels1 = labels.unsqueeze()\n    labels2 = labels.unsqueeze()\n\n    logits1 = logits.unsqueeze()\n    logits2 = logits.unsqueeze()\n\n    y_ij = labels1 - labels2\n    r_ij = (logits1 &gt;= logits2).() - (logits1 &lt; logits2).()\n\n    residual = torch.clamp(-r_ij*y_ij + margin, =)\n    loss = residual.mean()\n     loss\n</code></pre>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2482373,
      "author_name": "Jun Bai",
      "author_url": "",
      "post_date": "2023-10-15T01:13:55.113000",
      "content": "<p>Impressive!</p>",
      "votes": 2,
      "replies": []
    }
  ],
  "comment_count": 4,
  "messages": [
    {
      "id": 2480629,
      "postDate": "2023-10-13T12:56:52.237Z",
      "content": "<p>Thanks, Kaggle and <a href=\"https://www.kaggle.com/cookiecutters\" target=\"_blank\">@cookiecutters</a> for this interesting competition. Congratulations to all winners, and gold medalists. I would like to give credit to my talented and hard-working teammates <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a> and <a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a>.</p>\n<p>Kudos to <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> for their fantastic work on LGBM trick, and all participants of this competition for their sharing and hard work.</p>\n<h1>First words:</h1>\n<ul>\n<li>We think this competition is a perfect example of out-of-distribution forecasting, therefore, we don’t fully trust either CV or LB. Instead, we judge the models by our sense of training stability, partly CV, partly LB, and diversity;</li>\n<li>Therefore, we try to think of as many model architectures, and model training techniques as possible;</li>\n</ul>\n<h1>Model 1 - Transformer + Catboost</h1>\n<h3>Transformer model</h3>\n<ul>\n<li>CV/LB/PB: 0.552/0.488/0.556</li>\n<li>Input: text + [SEP] + prompt_question</li>\n<li>Loss: MCRMSE </li>\n<li>Backbone: deberta-v3-large</li>\n<li>Hyperparams:<ul>\n<li>max_length: 256</li>\n<li>pooling: attention</li>\n<li>random-reinit: 1</li>\n<li>differential learning rate factor: 2.6</li>\n<li>lr: 1e-5</li></ul></li>\n<li>epochs: 4</li>\n<li>evaluate at the end of each epoch</li>\n<li>add additional 2x full-fit checkpoints, it helps to gain 0.003 in PB</li>\n</ul>\n<h3>GBDT - Catboost</h3>\n<ul>\n<li>CV/LB: 0.513/0.443/0.484</li>\n<li>Hyperparams:<ul>\n<li>learning_rate: 0.05</li>\n<li>Cat_features = [“summary_n_sentence”]</li></ul></li>\n<li>Features:<ul>\n<li>summary_length, summary_n_sentence, mean_edit_distance of summary sentences, …</li>\n<li>remove many features like spell errors, POS tag features, …</li>\n<li>removing prompt_length improves CV a bit, but no help in LB/PB</li></ul></li>\n<li>Total inference time: 32mins</li>\n</ul>\n<h1>Model 2 - Span Model</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2F6170216dc252bbe8f8ef098e626a3b4b%2FSpanModel.png?generation=1697201774978961&amp;alt=media\" alt=\"\"></p>\n<ul>\n<li>Reformulated the problem as a span-regression task by randomly grouping N summaries associated with a particular prompt and predicting their scores together </li>\n<li>Predictions were conditioned on a context that consisted of the top 8 sentences from the prompt text as ranked by their importance. </li>\n<li>The importance value of a sentence from the prompt text was determined as the average cosine similarity score between the sentence and all summaries for the prompt</li>\n<li>Used MSE Loss on the context and wording targets</li>\n<li>Discretized the content and wording scores into 16 bins to create auxiliary targets and used ArcFace loss on aux targets</li>\n<li>Used ranking loss among summaries in each example span</li>\n<li>During inference, the random grouping of N summaries provided a TTA-like impact</li>\n<li>Private LB by using 2x full-fit checkpoints: 0.473</li>\n</ul>\n<h1>Model 3 - Search Approach</h1>\n<ul>\n<li>Input text is comprised of a context and the student summary</li>\n<li>Context:<ul>\n<li>Retrieved the top 3 sentences from the prompt text based on cosine similarity scores with the summary. </li>\n<li>Concatenated prompt title and prompt question </li></ul></li>\n<li>Trained the model using BCEWithLogitsLoss<ul>\n<li>Scaled the scores between 0-1 </li>\n<li>Used BCEWithLogitsLoss to compute the loss between scaled scores and logits</li></ul></li>\n<li>Private LB by using 2x full-fit checkpoints: 0.476</li>\n</ul>\n<h1>Model 4 - Long Context</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2F9184846a10fbdd62ab5d384d294d81d4%2FLongContext.png?generation=1697201712777096&amp;alt=media\" alt=\"\"></p>\n<ul>\n<li>Remarks:<ul>\n<li>max_len: 1024</li>\n<li>lr: 2e-5</li>\n<li>weight_decay: 1e-2</li>\n<li>n_epochs: 3</li>\n<li>AWP and EMA after 1 epoch</li>\n<li>evaluate 10 times/epoch</li>\n<li>Inference time: 205 mins (4 folds)</li>\n<li>CV/Public LB/Private LB: 0.499/0.438/0.462</li>\n<li>Backbone: OpenAssistant/reward-model-deberta-v3-large-v2</li>\n<li>We have another model with this structure and its CV/Public LB/Private LB: 0.494/0.448/0.459. Unfortunately, we didn’t include this model in the ensemble</li></ul></li>\n<li>Problems:<ul>\n<li>Inference time is too long due to the long context → not good for ensemble</li></ul></li>\n<li>Code:<ul>\n<li>Training: <a href=\"https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv5e\" target=\"_blank\">https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv5e</a></li>\n<li>Inference: <a href=\"https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v5e-infer\" target=\"_blank\">https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v5e-infer</a> </li></ul></li>\n</ul>\n<h1>Model 5 - Dual Encoder with Transfer Learning</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2Fde86838dc52ff852ab461f461ab2a4b7%2FDualEncoder.png?generation=1697201727858488&amp;alt=media\" alt=\"\"></p>\n<ul>\n<li>Model Ideas:<ul>\n<li>Instead of concatenating text and prompt info (prompt text, question, title) before passing through the backbone, we can concatenate them after the transformer. One benefit is that many summaries share the same prompts, we can compute them only once and put them into a hashing table, once the embedding of a prompt is computed, we can just query it instead of computing it again.</li>\n<li>However, I think this way is not good in terms of performance because there is no connection between text and prompt (they’re processed separately in spite of being processed by the same backbone). To overcome it, I load the trained backbone from the long-context model (trained with concatenated text and prompt, with a long sequence length), freeze the top 12 layers, and train the second stage model on top.</li></ul></li>\n<li>Remarks:<ul>\n<li>max_len context: 1536</li>\n<li>max_len text: 512</li>\n<li>n_epochs: 1</li>\n<li>AWP and EMA enabled after 50 steps</li>\n<li>Use LGBM trick</li>\n<li>Inference time: 39 mins (4 folds)</li>\n<li>CV/Public LB/Private LB: 0.500/0.445/0.470</li>\n<li>Backbone: OpenAssistant/reward-model-deberta-v3-large-v2</li></ul></li>\n<li>Code:<ul>\n<li>Training: <a href=\"https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv6c\" target=\"_blank\">https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv6c</a> </li>\n<li>Inference: <a href=\"https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v6c-infer\" target=\"_blank\">https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v6c-infer</a> </li></ul></li>\n</ul>\n<h1>Final ensemble</h1>\n<p>We make ensembles of these above models and the public models by <a href=\"https://www.kaggle.com/tsunotsuno\" target=\"_blank\">@tsunotsuno</a> (we actually train it by ourselves rather than get the model from them directly). We observe a super messy correlation between CV and Public LB here, at the ensemble phase. For example, we simply took the average of the component models, the weights were optimized with some constraints to deal with overfitting.</p>\n<h1>What didn’t work</h1>\n<ul>\n<li>We tried to generate synthetic data from LLMs (T5 and Llama), but it didn’t work at all;</li>\n<li>Training with LoRA for target-wise models worked and got CV 0.53+, but didn’t help in the local ensemble;</li>\n</ul>\n<h1>Team member</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a></li>\n<li><a href=\"https://www.kaggle.com/syhens\" target=\"_blank\">@syhens</a></li>\n<li><a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2F1ff6a8913b16a36c1474d8418353be2a%2Fteam.PNG?generation=1697202450769081&amp;alt=media\" alt=\"\"></li>\n</ul>",
      "rawMarkdown": "Thanks, Kaggle and @cookiecutters for this interesting competition. Congratulations to all winners, and gold medalists. I would like to give credit to my talented and hard-working teammates @conjuring92 and @syhens.\n\nKudos to @tsunotsuno for their fantastic work on LGBM trick, and all participants of this competition for their sharing and hard work.\n\n# First words:\n- We think this competition is a perfect example of out-of-distribution forecasting, therefore, we don’t fully trust either CV or LB. Instead, we judge the models by our sense of training stability, partly CV, partly LB, and diversity;\n- Therefore, we try to think of as many model architectures, and model training techniques as possible;\n\n# Model 1 - Transformer + Catboost\n### Transformer model\n- CV/LB/PB: 0.552/0.488/0.556\n- Input: text + [SEP] + prompt_question\n- Loss: MCRMSE \n- Backbone: deberta-v3-large\n- Hyperparams:\n    * max_length: 256\n    * pooling: attention\n    * random-reinit: 1\n    * differential learning rate factor: 2.6\n    * lr: 1e-5\n* epochs: 4\n* evaluate at the end of each epoch\n- add additional 2x full-fit checkpoints, it helps to gain 0.003 in PB\n\n### GBDT - Catboost\n- CV/LB: 0.513/0.443/0.484\n- Hyperparams:\n    * learning_rate: 0.05\n    * Cat_features = [“summary_n_sentence”]\n- Features:\n    * summary_length, summary_n_sentence, mean_edit_distance of summary sentences, …\n    * remove many features like spell errors, POS tag features, …\n    * removing prompt_length improves CV a bit, but no help in LB/PB\n- Total inference time: 32mins\n\n# Model 2 - Span Model\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2F6170216dc252bbe8f8ef098e626a3b4b%2FSpanModel.png?generation=1697201774978961&alt=media)\n- Reformulated the problem as a span-regression task by randomly grouping N summaries associated with a particular prompt and predicting their scores together \n- Predictions were conditioned on a context that consisted of the top 8 sentences from the prompt text as ranked by their importance. \n- The importance value of a sentence from the prompt text was determined as the average cosine similarity score between the sentence and all summaries for the prompt\n- Used MSE Loss on the context and wording targets\n- Discretized the content and wording scores into 16 bins to create auxiliary targets and used ArcFace loss on aux targets\n- Used ranking loss among summaries in each example span\n- During inference, the random grouping of N summaries provided a TTA-like impact\n- Private LB by using 2x full-fit checkpoints: 0.473\n\n# Model 3 - Search Approach\n\n- Input text is comprised of a context and the student summary\n- Context:\n    * Retrieved the top 3 sentences from the prompt text based on cosine similarity scores with the summary. \n    * Concatenated prompt title and prompt question \n- Trained the model using BCEWithLogitsLoss\n    * Scaled the scores between 0-1 \n    * Used BCEWithLogitsLoss to compute the loss between scaled scores and logits\n- Private LB by using 2x full-fit checkpoints: 0.476\n\n# Model 4 - Long Context\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2F9184846a10fbdd62ab5d384d294d81d4%2FLongContext.png?generation=1697201712777096&alt=media)\n- Remarks:\n    * max_len: 1024\n    * lr: 2e-5\n    * weight_decay: 1e-2\n    * n_epochs: 3\n    * AWP and EMA after 1 epoch\n    * evaluate 10 times/epoch\n    * Inference time: 205 mins (4 folds)\n    * CV/Public LB/Private LB: 0.499/0.438/0.462\n    * Backbone: OpenAssistant/reward-model-deberta-v3-large-v2\n    * We have another model with this structure and its CV/Public LB/Private LB: 0.494/0.448/0.459. Unfortunately, we didn’t include this model in the ensemble\n- Problems:\n    * Inference time is too long due to the long context → not good for ensemble\n- Code:\n    * Training: https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv5e\n    * Inference: https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v5e-infer \n\n# Model 5 - Dual Encoder with Transfer Learning\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2Fde86838dc52ff852ab461f461ab2a4b7%2FDualEncoder.png?generation=1697201727858488&alt=media)\n- Model Ideas:\n    * Instead of concatenating text and prompt info (prompt text, question, title) before passing through the backbone, we can concatenate them after the transformer. One benefit is that many summaries share the same prompts, we can compute them only once and put them into a hashing table, once the embedding of a prompt is computed, we can just query it instead of computing it again.\n    * However, I think this way is not good in terms of performance because there is no connection between text and prompt (they’re processed separately in spite of being processed by the same backbone). To overcome it, I load the trained backbone from the long-context model (trained with concatenated text and prompt, with a long sequence length), freeze the top 12 layers, and train the second stage model on top.\n- Remarks:\n    * max_len context: 1536\n    * max_len text: 512\n    * n_epochs: 1\n    * AWP and EMA enabled after 50 steps\n    * Use LGBM trick\n    * Inference time: 39 mins (4 folds)\n    * CV/Public LB/Private LB: 0.500/0.445/0.470\n    * Backbone: OpenAssistant/reward-model-deberta-v3-large-v2\n- Code:\n    * Training: https://www.kaggle.com/datasets/shinomoriaoshi/commonlitsummaryv6c \n    * Inference: https://www.kaggle.com/code/shinomoriaoshi/commonlitsummary-v6c-infer \n\n# Final ensemble\nWe make ensembles of these above models and the public models by @tsunotsuno (we actually train it by ourselves rather than get the model from them directly). We observe a super messy correlation between CV and Public LB here, at the ensemble phase. For example, we simply took the average of the component models, the weights were optimized with some constraints to deal with overfitting.\n\n# What didn’t work\n- We tried to generate synthetic data from LLMs (T5 and Llama), but it didn’t work at all;\n- Training with LoRA for target-wise models worked and got CV 0.53+, but didn’t help in the local ensemble;\n\n# Team member\n- @conjuring92\n- @syhens\n- @shinomoriaoshi\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2176524%2F1ff6a8913b16a36c1474d8418353be2a%2Fteam.PNG?generation=1697202450769081&alt=media)",
      "votes": 30
    },
    {
      "id": 2482723,
      "postDate": "2023-10-15T08:24:14.870Z",
      "content": "<p>Very innovative approaches! My favourite is Model 2.<br>\nAlthough could you explain what are auxiliary targets really and what exactly is ranking loss, like what is the ground truth used</p>",
      "rawMarkdown": "Very innovative approaches! My favourite is Model 2.\nAlthough could you explain what are auxiliary targets really and what exactly is ranking loss, like what is the ground truth used",
      "votes": 3,
      "replies": [
        {
          "id": 2482804,
          "postDate": "2023-10-15T09:37:10.200Z",
          "content": "<p>Thanks, this model is developed by Raja. <a href=\"https://www.kaggle.com/conjuring92\" target=\"_blank\">@conjuring92</a>, Could you please explain the model to him?</p>",
          "rawMarkdown": "Thanks, this model is developed by Raja. @conjuring92, Could you please explain the model to him?",
          "votes": 1
        },
        {
          "id": 2483872,
          "postDate": "2023-10-16T04:08:41.713Z",
          "content": "<p>Many thanks for the questions. <br>\n<code>auxiliary targets</code>: It's basically a mapping between raw scores and bin index. For examples: <code>content &lt; -2.0</code> is mapped to 0, <code>-2.0 &lt; content &lt;  - 1.75</code> is mapped to 1 …<br>\n<code>ranking loss</code> is defined as below:</p>\n<pre><code> ():\n    labels1 = labels.unsqueeze()\n    labels2 = labels.unsqueeze()\n\n    logits1 = logits.unsqueeze()\n    logits2 = logits.unsqueeze()\n\n    y_ij = labels1 - labels2\n    r_ij = (logits1 &gt;= logits2).() - (logits1 &lt; logits2).()\n\n    residual = torch.clamp(-r_ij*y_ij + margin, =)\n    loss = residual.mean()\n     loss\n</code></pre>",
          "rawMarkdown": "Many thanks for the questions. \n`auxiliary targets`: It's basically a mapping between raw scores and bin index. For examples: `content < -2.0` is mapped to 0, `-2.0 < content <  - 1.75` is mapped to 1 ...\n`ranking loss` is defined as below:\n\n```python\ndef get_ranking_loss(logits, labels, margin=0.0):\n    labels1 = labels.unsqueeze(1)\n    labels2 = labels.unsqueeze(0)\n\n    logits1 = logits.unsqueeze(1)\n    logits2 = logits.unsqueeze(0)\n\n    y_ij = labels1 - labels2\n    r_ij = (logits1 >= logits2).float() - (logits1 < logits2).float()\n\n    residual = torch.clamp(-r_ij*y_ij + margin, min=0.0)\n    loss = residual.mean()\n    return loss\n```",
          "votes": 2
        }
      ]
    },
    {
      "id": 2482373,
      "postDate": "2023-10-15T01:13:55.113Z",
      "content": "<p>Impressive!</p>",
      "rawMarkdown": "Impressive!",
      "votes": 2
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:05:18Z",
  "competition_id": 53482,
  "competition_slug": "commonlit-evaluate-student-summaries",
  "team_id": 10674602,
  "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/int8-16th-place-solution"
}