{
  "id": 369609,
  "title": "3rd Place Solution - Congratulations New Competition Grandmaster Amed!",
  "content": "<h1>Congratulations Grandmaster Amed !</h1>\n<p>Congratulations Amed ( <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> ) for becoming Kaggle's newest Competition Grandmaster! Amed was our teams' superstar who designed most of our models. He truly earns the title Kaggle Competition Grandmaster!</p>\n<h1>3rd Place Solution Summary</h1>\n<p>Our solution is an ensemble of 24 models without using pseudo labels. Most models use Deberta-v3-large backbone and a variety of tricks. The ensemble was chosen using hill climbing. Teaming with Amed ( <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> ) and CroDoc ( <a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a> ) was a pleasure. Thank you guys!</p>\n<h1>Hill Climbing - CV 0.4420</h1>\n<p>I love hill climbing because it can take lots of models and pick the best small subset of models. (i.e. its like Lasso regression) And it computes ensemble model weights. </p>\n<p>We start with the single model with best CV score (for us was 0.4470). Then we iterate through all other models and pick a second model which helps the most. Then we pick a third etc until new models do not help. Our ensemble without pseudo achieves CV 0.4420. We allowed hill climbing to use <strong>negative weights</strong> which boosted CV <code>+0.0010</code> and private LB <code>+0.0060</code> versus using positive weights only. </p>\n<pre><code>while not STOP:\n    potential_new_best_cv_score = GET_BEST()\n    for k in range( len(MODELS) ):\n        for wgt in range(-0.5,0.51,0.01):\n            potential_ensemble = (1-wgt) * current_best_ensemble + wgt * MODELS[k]\n            cv_score = compute_metric( potential_ensemble )\n            if cv_score &lt; potential_new_best_cv_score: \n                potential_new_best_cv_score = cv_score\n                REMEMBER_THIS_MODEL(k,wgt)\n    current_best_ensemble, STOP = UPDATE_BEST()\n</code></pre>\n<p><img src=\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/hill_climb.png\" alt=\"\"></p>\n<h1>Models - CV 0.4470</h1>\n<p>Each day, we train new diverse models. Then we run hill climbing to see if the new model gets chosen. We don't need to make a new model with a great CV score, we just need to make new models that are diverse. Below are the choices in order which hill climbing picked from our 50 models. We observe that the best CV score models are not chosen first. Instead hill climbing chooses <strong>diverse models</strong>.</p>\n<p>The improvement in ensemble CV score is shown in the plot above. The table below displays single model CV scores. Our best single model without pseudo achieves CV 0.4470</p>\n<table>\n<thead>\n<tr>\n<th>order selected</th>\n<th>backbone</th>\n<th>cv score</th>\n<th>weight</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>deberta-v3-large</td>\n<td>0.447</td>\n<td>0.190</td>\n</tr>\n<tr>\n<td>2</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.4524</td>\n<td>0.142</td>\n</tr>\n<tr>\n<td>3</td>\n<td>deberta-v3-large</td>\n<td>0.4498</td>\n<td>0.124</td>\n</tr>\n<tr>\n<td>4</td>\n<td>deberta-large-mnli</td>\n<td>0.4548</td>\n<td>0.068</td>\n</tr>\n<tr>\n<td>5</td>\n<td>deberta-v3-large</td>\n<td>0.4492</td>\n<td>0.133</td>\n</tr>\n<tr>\n<td>6</td>\n<td>xlm-roberta-large</td>\n<td>0.4575</td>\n<td>0.092</td>\n</tr>\n<tr>\n<td>7</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.457</td>\n<td>-0.160</td>\n</tr>\n<tr>\n<td>8</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.4525</td>\n<td>0.101</td>\n</tr>\n<tr>\n<td>9</td>\n<td>deberta-v3-large</td>\n<td>0.4489</td>\n<td>0.125</td>\n</tr>\n<tr>\n<td>10</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.4565</td>\n<td>-0.124</td>\n</tr>\n<tr>\n<td>11</td>\n<td><a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">RAPIDS-SVR</a></td>\n<td>0.4526</td>\n<td>0.083</td>\n</tr>\n<tr>\n<td>12</td>\n<td>deberta-v3-large</td>\n<td>0.4495</td>\n<td>0.102</td>\n</tr>\n<tr>\n<td>13</td>\n<td>deberta-v3-large</td>\n<td>0.4502</td>\n<td>0.081</td>\n</tr>\n<tr>\n<td>14</td>\n<td><a href=\"https://www.kaggle.com/code/electro/deberta-layerwiselr-lastlayerreinit-tensorflow\" target=\"_blank\">TF-deberta-v3-base</a></td>\n<td>0.4554</td>\n<td>0.061</td>\n</tr>\n<tr>\n<td>15</td>\n<td>deberta-v3-large</td>\n<td>0.4527</td>\n<td>-0.069</td>\n</tr>\n<tr>\n<td>16</td>\n<td>deberta-v3-large</td>\n<td>0.4522</td>\n<td>-0.058</td>\n</tr>\n<tr>\n<td>17</td>\n<td>deberta-v3-large</td>\n<td>0.45</td>\n<td>0.061</td>\n</tr>\n<tr>\n<td>18</td>\n<td>deberta-v3-large</td>\n<td>0.4516</td>\n<td>-0.058</td>\n</tr>\n<tr>\n<td>19</td>\n<td>deberta-v3-large</td>\n<td>0.4509</td>\n<td>0.071</td>\n</tr>\n<tr>\n<td>20</td>\n<td>deberta-v3-base</td>\n<td>0.4575</td>\n<td>-0.039</td>\n</tr>\n<tr>\n<td>21</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.4501</td>\n<td>0.051</td>\n</tr>\n<tr>\n<td>22</td>\n<td>deberta-v3-large</td>\n<td>0.4512</td>\n<td>-0.049</td>\n</tr>\n<tr>\n<td>23</td>\n<td>roberta-large</td>\n<td>0.4571</td>\n<td>0.030</td>\n</tr>\n<tr>\n<td>24</td>\n<td>deberta-v3-large</td>\n<td>0.45</td>\n<td>0.040</td>\n</tr>\n</tbody>\n</table>\n<h1>What Worked - Tricks</h1>\n<p>Our models use a diversity of regression heads, pooling techniques, learning schedules, and backbones. Some models used the following tricks:</p>\n<ul>\n<li>Different loss rates per target <code>{'cohesion':0.21, 'syntax':0.16, 'vocabulary':0.10, 'phraseology':0.16, 'grammar':0.21, 'conventions':0.16}</code> [High Impact] </li>\n<li>Clip grad norm with <code>max_norm = 10</code> [High Impact]</li>\n<li><code>hidden_dropout_prob = 0.0</code> and <code>attention_probs_dropout_prob = 0.0</code> [High Impact]</li>\n<li>Stride window for non deberta models [Medium Impact] </li>\n<li>Since Deberta tokenizer ignores \"\\n\" we replaced \"\\n\\n\" with \"|\" [Medium Impact] </li>\n<li>Last layer reinitialisation [Medium Impact] </li>\n<li>2 stage pooling. First pool either words, sentences, or paragraphs. Then pool that result. [Medium Impact] </li>\n<li>Train with <code>max_len=2048</code>, infer <code>max_len = 640</code> [High Impact] </li>\n<li>Train with <code>batch_size = 1</code> [High Impact] </li>\n<li>RAPIDS SVR using embeddings without train on comp data [Medium Impact] </li>\n</ul>\n<h1>What Didn't Work - Pseudo Labels</h1>\n<p>Our final submission <strong>does not</strong> use pseudo labels. We tried using pseudo labels during the competition. We were careful to avoid leaks, but none-the-less using pseudo labels from Feedback Prize 1 competition boosted our single model best CV score from 0.4470 to 0.4370 and did not improve our LB score. There must be a leak somewhere but we still haven't found it.</p>\n<p>We noticed that the distribution of targets from FP1 is different than FP3. So care must to be taken to account for this difference in distribution. Below are histogram of pseudo labels on FP1 vs. pseudo labels on FP3. We observe that the old Feedback Prize 1 comp has greater target values than current Feedback Prize 3 comp:</p>\n<p><img src=\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/shift.png\" alt=\"\"></p>\n<h1>Team Members</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a></li>\n<li><a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a></li>\n<li><a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> </li>\n</ul>\n<h1>Solution Code Published</h1>\n<p>We published our training code on GitHub <a href=\"https://github.com/Amed1710/Feedback-Prize--English-Language-Learning\" target=\"_blank\">here</a> and we published our inference code on Kaggle <a href=\"https://www.kaggle.com/code/cdeotte/3rd-place-solution-lb-0-4337-cv-0-4420\" target=\"_blank\">here</a></p>",
  "comments": [
    {
      "id": 2050365,
      "author_name": "Ertuğrul Demir",
      "author_url": "",
      "post_date": "2022-11-30T17:34:15.240000",
      "content": "<p>Thanks for sharing your results and congrats to all, especially <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> ! Many of tricks we experimented or thought of during the competition phrase but this one got my attention:</p>\n<p><code>Train with max_len=2048, infer max_len = 640</code></p>\n<p>What's the exact motivation behind this trick? Thanks…</p>",
      "votes": 9,
      "replies": [
        {
          "id": 2050372,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-11-30T17:38:26.340000",
          "content": "<p>IMO Models want the most train data possible. Training with <code>max_len=2048</code> allows the model to learn from all the provided train text. Once we have a smart model, it does not need to see the entire test essay to predict the targets. (And just like humans if you show it too much maybe it gets confused or overwhelmed). Hence during inference, it can use less. (i.e. Humans read the first paragraph and already have a good sense of the essay's final target scores).</p>\n<p>We confirmed this on CV and LB. First train a model with 2048. Then create OOF (using the same model again and again) with different inference lengths like 512, 640, 768, 1024 etc. Compute CV score and submit to LB. Using 640 achieved best CV and LB.</p>",
          "votes": 29,
          "replies": []
        },
        {
          "id": 2050394,
          "author_name": "Ertuğrul Demir",
          "author_url": "",
          "post_date": "2022-11-30T17:45:28.580000",
          "content": "<p>Thanks for the explanation! Makes sense.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 2050434,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-11-30T18:21:59.613000",
          "content": "<p><a href=\"https://www.kaggle.com/datafan07\" target=\"_blank\">@datafan07</a> Also note that Deberta uses \"relative embeddings\". Therefore if WORD_A in position 1500 relates to the word in position 1501. Then the model learns that WORD_A needs to pay attention to the word in position +1. Therefore when we infer with length 640 and the model sees WORD_A in position 550, it will look at position 551. In other words, the learnings from training with max_len=2028 will generalize to inference at max_len=640. (I think this is how it works)</p>",
          "votes": 20,
          "replies": []
        }
      ]
    },
    {
      "id": 2050709,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-11-30T23:48:17.800000",
      "content": "<p>Congratulations! Thanks for the excellent sharing!  Your team did geat work to make diverse models to help ensemble!<br>\nI learned a lot especially below:</p>\n<ol>\n<li>Non deberta using slide window seems help a lot, I could see you have non deberta models like roberta with good cv.</li>\n<li>Train long and infer short is creative, as before we always train infer using same, or train using short and infer using long(only concidering training using more GPU mem), and I only tried same length train/infer here, for me larger length work a bit better.</li>\n<li>Analysis error rate for each target and tune for loss rate per target is cool</li>\n<li>hill climb, negative model weights helps, another sharing also mentioned this.</li>\n</ol>",
      "votes": 8,
      "replies": []
    },
    {
      "id": 2050412,
      "author_name": "Ravi Ramakrishnan",
      "author_url": "",
      "post_date": "2022-11-30T18:01:04.473000",
      "content": "<p>Hearty congratulations to <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a>, <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> and <a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a> for a great result and thanks to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a>  for the detailed approach note too!</p>",
      "votes": 5,
      "replies": []
    },
    {
      "id": 2050362,
      "author_name": "CroDoc",
      "author_url": "",
      "post_date": "2022-11-30T17:30:41.533000",
      "content": "<p>Amazing teammates! It was really nice to work with both of you. Also, I think you really wrote nicely the key points that contributed to our final solution.</p>",
      "votes": 5,
      "replies": []
    },
    {
      "id": 2050377,
      "author_name": "Amed",
      "author_url": "",
      "post_date": "2022-11-30T17:40:17.780000",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a>  for your kind words.<br>\nI really enjoyed working with you.</p>",
      "votes": 6,
      "replies": []
    },
    {
      "id": 2058634,
      "author_name": "Yuji.K",
      "author_url": "",
      "post_date": "2022-12-08T05:10:25.640000",
      "content": "<p>Thank you for sharing!<br>\nDid you try MLM?<br>\nI have a question about MLM.<br>\nWhen doing MLM using training data, should I create a model for each fold to account for data leakages?</p>\n<p>Ex)<br>\n[Setting] df = (FB3 data), train = df[df['fold'] != 0], valid = df[df['fold'] == 0],  pl_data = (FB1 data)<br>\nIn this situation, do MLM training with train&amp;pl_data and evaluation with valid.<br>\nDo this for the number of folds.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2059319,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-12-08T17:51:30.990000",
          "content": "<p>Our final solution does not use MLM and I did not try MLM. Perhaps <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> or <a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a> tried MLM.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 2059452,
          "author_name": "CroDoc",
          "author_url": "",
          "post_date": "2022-12-08T21:29:59.047000",
          "content": "<p>I think Amed tried it and said it didn't help.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 2059521,
          "author_name": "Amed",
          "author_url": "",
          "post_date": "2022-12-09T00:55:19.580000",
          "content": "<p>Yes, I tried MLM (and the pre-trained weight on Feedback 2) without success. Which was quite surprising.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2060106,
          "author_name": "Yuji.K",
          "author_url": "",
          "post_date": "2022-12-09T14:53:16.757000",
          "content": "<p>Thank you! How about how to do MLM?Did you do it like above?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2060285,
          "author_name": "Amed",
          "author_url": "",
          "post_date": "2022-12-09T17:23:22.777000",
          "content": "<p>Yes.<br>\nBut you need to remove the duplicates essays between FB1 and FB3 before doing MLM.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2060291,
          "author_name": "Yuji.K",
          "author_url": "",
          "post_date": "2022-12-09T17:38:36.857000",
          "content": "<p>Thank you.<br>\nCongratulations on new GM!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2056471,
      "author_name": "Apa",
      "author_url": "",
      "post_date": "2022-12-06T06:45:07.340000",
      "content": "<p>Can you explain what exactly \"Stride window for non deberta models\" does in your solution?😄</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2057238,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-12-06T22:00:00.377000",
          "content": "<p>Models like RoBERTa can only accept input with <code>max_len = 512</code> tokens. So to train with sequences of len 2048, we need to train and infer using a sliding window.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2057325,
          "author_name": "Apa",
          "author_url": "",
          "post_date": "2022-12-07T01:31:17.427000",
          "content": "<p>For example, the length of 2048 can be split into four lengths of 512, that is, one sample is split into four samples, and they have the same label. Then will it train on these 4 samples?</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2057341,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-12-07T01:43:43.077000",
          "content": "<p>Yes basically. However with sliding window, we also pick a stride length. So if <code>max_len=512</code> and <code>doc_stride=256</code>. Then the you break it into 7 pieces, namely 0:512, 256:768, 512:1024, 768:1280, 1024:1536, 1280:1792, 1536:2024. The pieces overlap by <code>max_len minus doc_stride</code>.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 2057362,
          "author_name": "Apa",
          "author_url": "",
          "post_date": "2022-12-07T03:06:17.693000",
          "content": "<p>Thank you for your detailed reply, and best of luck to you in the future</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2051569,
      "author_name": "Bootstrapping",
      "author_url": "",
      "post_date": "2022-12-01T13:57:44.467000",
      "content": "<p>Can you explain how to choose max_norm=10 for gradient clipping?</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2052397,
          "author_name": "Amed",
          "author_url": "",
          "post_date": "2022-12-02T06:58:25.983000",
          "content": "<p>gradient clipping made our models more stable.<br>\nIn the end, It was just tuning this parameter. I found that max_norm=10 was the best.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2051431,
      "author_name": "Optimo",
      "author_url": "",
      "post_date": "2022-12-01T12:08:23.763000",
      "content": "<p><a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> congratulations!! Impressive achievement unlocked! </p>",
      "votes": 3,
      "replies": [
        {
          "id": 2052405,
          "author_name": "Amed",
          "author_url": "",
          "post_date": "2022-12-02T07:09:14.380000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/optimo\" target=\"_blank\">@optimo</a> :).<br>\nI hope you will soon.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2050836,
      "author_name": "Takamichi Toda",
      "author_url": "",
      "post_date": "2022-12-01T02:59:03.227000",
      "content": "<p>Congratulations on Gold Medal🥇 and thank you for sharing your solution.</p>\n<p>I have one question.</p>\n<blockquote>\n  <p>Train with batch_size = 1</p>\n</blockquote>\n<p>I’m not sure about it. It is not mini-batch gradient descent.<br>\nIs simple online stochastic gradient descent (not using mini-batch) better in this competition?</p>",
      "votes": 4,
      "replies": [
        {
          "id": 2052412,
          "author_name": "Amed",
          "author_url": "",
          "post_date": "2022-12-02T07:14:22.737000",
          "content": "<blockquote>\n  <p>Is simple online stochastic gradient descent (not using mini-batch) better in this competition?</p>\n</blockquote>\n<p>It was better for us.<br>\nAnd our models with '2 x Pooling' only support <code>batch_size = 1</code></p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 2053139,
          "author_name": "Takamichi Toda",
          "author_url": "",
          "post_date": "2022-12-02T21:28:11.990000",
          "content": "<p>Thank you for your response!<br>\nI have never tried batch size=1 before, so I will add it to my experiments next time😊</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 2050541,
      "author_name": "Mohamed Eltayeb",
      "author_url": "",
      "post_date": "2022-11-30T20:25:36.760000",
      "content": "<p>Congratulations! Really deserve it!<br>\nJust one question, could I know what is your validation scheme? and whether it was correlating well with the LB? Jumping really high in the lb has definietly something to do with the validation right? :)</p>",
      "votes": 4,
      "replies": [
        {
          "id": 2050565,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-11-30T20:48:41.923000",
          "content": "<p>Validation is key. \"Trust Your CV\" is king. We picked all our final submissions based on CV score and ignored LB score because LB is the size of 1 fold, and train data CV is the size of 5 folds. A single fold is not as reliable as the full 5 folds CV score to estimate private LB score. </p>\n<p>All our models were trained using the same 5 MultilabelStratifiedKFold (seed 42) as seen in public notebooks. For the most part improvements in CV resulted in improvements in LB. But the LB data was the size of 1 fold of 5 fold. And behaved a little different than CV score at times.</p>",
          "votes": 8,
          "replies": []
        },
        {
          "id": 2050645,
          "author_name": "Mohamed Eltayeb",
          "author_url": "",
          "post_date": "2022-11-30T22:35:23.060000",
          "content": "<p>Thanks for the clarification! Honestly, I always be so happy when finding you in a competition I work on. You and the others top GMs make it so fun! Hope one day I can team up with you.</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 2050420,
      "author_name": "Noir.s",
      "author_url": "",
      "post_date": "2022-11-30T18:10:48.780000",
      "content": "<p>Congratulations to your team for landing a top 3 finish!! </p>\n<p>I had a couple of additional questions</p>\n<hr>\n<blockquote>\n  <p>Different loss rates per target {'cohesion':0.21, 'syntax':0.16, 'vocabulary':0.10, 'phraseology':0.16, 'grammar':0.21, 'conventions':0.16}</p>\n</blockquote>\n<p>What was the methodology to determine the loss rates? </p>\n<hr>\n<blockquote>\n  <p>2 stage pooling. First pool either words, sentences, or paragraphs. Then pool that result.</p>\n</blockquote>\n<p>Can you provide some more detail regarding this? I am not able to follow what this is</p>\n<hr>\n<p>Thank you for sharing your approach to the competition!</p>",
      "votes": 4,
      "replies": [
        {
          "id": 2050441,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-11-30T18:29:31.720000",
          "content": "<p><a href=\"https://www.kaggle.com/amed\" target=\"_blank\">@amed</a> can share the details. These are two of his tricks.</p>\n<p>I think the way \"2 x pooling\" works is as follows. Amed please correct me if i'm wrong. A transformer model outputs embeddings for each token which are pieces of words. Most popular notebook, use mean pooling over all these token outputs. Therefore each token has equal importance.</p>\n<p>When doing paragraph pooling, we first takes the mean of all tokens in each paragraph separately. We now have a single embedding for each paragraph. Next we take the mean of these embeddings. This gives each paragraph equal weight. </p>\n<p>This is different than normal mean pooling. With normal mean pooling, if one paragraph has twice as many tokens as another paragraph, then the first paragraph would be twice as important when taking average of tokens.</p>\n<p>Amed, can you share how you discovered the different learning rates per target?</p>",
          "votes": 10,
          "replies": []
        },
        {
          "id": 2050521,
          "author_name": "Amed",
          "author_url": "",
          "post_date": "2022-11-30T20:09:52.530000",
          "content": "<p>Regarding <strong>\"2 x Pooling\",</strong> <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> explained it really well.<br>\nWe tried different types of Pooling for each level.<br>\nFor example :</p>\n<ul>\n<li>MaxPooling on each paragraph and MeanPooling on that result</li>\n<li>AttentionHead on each paragraph and MeanPooling on that result</li>\n<li>etc…</li>\n</ul>\n<blockquote>\n  <p>Amed, can you share how you discovered the different learning rates per target?</p>\n</blockquote>\n<p>When training our models, we could see in the image below that the <strong>cohesion</strong> and <strong>grammar</strong>  targets had the highest error rate, which explains why we gave more weight to these targets when computing the loss function.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2852238%2Fb24cd06928c67ca91f3e1cfd978a9c2d%2Fimage%20(1).png?generation=1669837101207255&amp;alt=media\" alt=\"\"></p>\n<p>We tried different combinaison of loss rates per target, and best combinaison was: <code>{'cohesion':0.21, 'syntax':0.16, 'vocabulary':0.10, 'phraseology':0.16, 'grammar':0.21, 'conventions':0.16}</code></p>",
          "votes": 14,
          "replies": []
        },
        {
          "id": 2050701,
          "author_name": "Noir.s",
          "author_url": "",
          "post_date": "2022-11-30T23:41:44.470000",
          "content": "<p>Thank you both for clarifying! Happy to learn these awesome tricks! 😄</p>\n<p>Congratulations again!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2050390,
      "author_name": "Romuska Palo Ur Laputa",
      "author_url": "",
      "post_date": "2022-11-30T17:44:09.490000",
      "content": "<p>Congratulations on your gold medal!!<br>\nI'm always learning from you!💪</p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 2054442,
      "author_name": "CocoBear",
      "author_url": "",
      "post_date": "2022-12-04T06:35:16.590000",
      "content": "<p>Congratulations!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2071297,
      "author_name": "Marília Prata",
      "author_url": "",
      "post_date": "2022-12-20T21:03:15.223000",
      "content": "<p>A huge Congratulations Amed for reaching the Most difficult Kaggle step: Competition GM.</p>\n<p>Also congratulations for the whole team (amedprof, crodoc, cdeotte) and the amazing solution. </p>",
      "votes": 2,
      "replies": [
        {
          "id": 2071300,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-12-20T21:09:20.547000",
          "content": "<p>Thanks Marília!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 2058009,
      "author_name": "pineapple",
      "author_url": "",
      "post_date": "2022-12-07T14:39:40.940000",
      "content": "<p>Congrats Amed! Do you have any idea as to why pseudo labels didn't work?</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2058827,
          "author_name": "Amed",
          "author_url": "",
          "post_date": "2022-12-08T08:40:53.773000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/perrypineapple\" target=\"_blank\">@perrypineapple</a>.<br>\nWhen doing pseudo-labels, you need to be sure that all the models used to generate the pseudo-labels are trained on the same cv split.<br>\nI suspect that one of our models was trained on a different split. This might have introduced a leak.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2057153,
      "author_name": "Muhammad Ammar Jamshed",
      "author_url": "",
      "post_date": "2022-12-06T19:57:49.173000",
      "content": "<p>Amazing Work !!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2056487,
      "author_name": "tarekbahgats",
      "author_url": "",
      "post_date": "2022-12-06T06:56:28.163000",
      "content": "<p>Congratulations</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2056374,
      "author_name": "Prakhar Prasad",
      "author_url": "",
      "post_date": "2022-12-06T04:28:51.760000",
      "content": "<p>congratulations, this is very inspiring and a new insight for me. </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2055216,
      "author_name": "Detr1st",
      "author_url": "",
      "post_date": "2022-12-04T22:04:54.400000",
      "content": "<p>Nice Work!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2055151,
      "author_name": "Giuliana",
      "author_url": "",
      "post_date": "2022-12-04T19:03:44.033000",
      "content": "<p>Congratulations!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2054765,
      "author_name": "Alhamdou",
      "author_url": "",
      "post_date": "2022-12-04T12:51:34.700000",
      "content": "<p>congratulations!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2054298,
      "author_name": "Yingshu Li",
      "author_url": "",
      "post_date": "2022-12-04T02:25:07.540000",
      "content": "<p>Congratulations! Really deserve it! i'm really interested in Hill Climbing. Could you please show some detailed infer about this part.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2053884,
      "author_name": "Maciek",
      "author_url": "",
      "post_date": "2022-12-03T17:24:55.613000",
      "content": "<p>Well done mate. </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2053344,
      "author_name": "sahilsingh2402",
      "author_url": "",
      "post_date": "2022-12-03T06:20:01.743000",
      "content": "<p>Congratulations!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2052158,
      "author_name": "Jaewook Kim",
      "author_url": "",
      "post_date": "2022-12-02T01:08:01.430000",
      "content": "<p>Congratulations! I was able to experiment a lot with the notebook you shared.</p>\n<p>I have a question. I know that the larger the number of fold, the smaller the bias.<br>\nIs it possible to make an absolute comparison of CV scores calculated with different number of fold?<br>\nFor example, can we say that cv_score=0.45 with k=4 and cv_score=0.45 with k=5 are the same performance?</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2052161,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-12-02T01:10:43.343000",
          "content": "<p>When train data is small, then the more folds will achieve a better CV and LB. This is because 4 folds uses 75% train data to train a fold model whereas 5 folds, 10 folds, 25 folds, use 80%, 90%, and 95% respectively.</p>\n<p>When training data is very large like 100's of thousands of samples than even 75% is plenty. But when train data only has a few thousand then every extra sample to train with makes a noticeable difference.</p>",
          "votes": 6,
          "replies": []
        }
      ]
    },
    {
      "id": 2051946,
      "author_name": "Wilmer E. Henao",
      "author_url": "",
      "post_date": "2022-12-01T18:44:12.667000",
      "content": "<p>I see that you tried the deberta versions finetuned with squad2. I also tried with other finetuned versions such as tner/deberta-v3-large-ontonotes5 and Elron/deberta-v3-large-sentiment.</p>\n<p>Those helped me a lot with the CV, however they performed really bad in the leaderboard. I achieved my medal with a model including only plain vanilla versions of roberta/deberta. Any idea why these finetuned versions didn't work?</p>\n<p>I realize in the case of tner, they are trained for NER, but still… they did well with the CV.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2052394,
          "author_name": "Amed",
          "author_url": "",
          "post_date": "2022-12-02T06:53:16.477000",
          "content": "<p>With our setup, we could achieve a good CV score with <code>deberta-v3-large-squad2</code>, and it brings diversity to our ensemble.<br>\nAs you can see in the table above, <code>deberta-v3-large-squad2</code> was the second model selected by hill climbing.<br>\nI guess the LB score depends on how you ensemble your models.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2051925,
      "author_name": "GMG",
      "author_url": "",
      "post_date": "2022-12-01T18:14:10.153000",
      "content": "<p>Nice work!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2051344,
      "author_name": "Johnny Tay",
      "author_url": "",
      "post_date": "2022-12-01T10:59:23.440000",
      "content": "<p>Congratulations and thanks for sharing! i have learned something!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2051244,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-12-01T09:47:40.593000",
      "content": "",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2050482,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-11-30T19:33:45.127000",
      "content": "",
      "votes": 2,
      "replies": [
        {
          "id": 2050503,
          "author_name": "",
          "author_url": "",
          "post_date": "2022-11-30T19:54:47.607000",
          "content": "",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 2481456,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-10-14T06:28:18.407000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2407185,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-08-24T23:20:39.343000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2113049,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-01-24T03:52:00.367000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 61,
  "messages": [
    {
      "id": 2050354,
      "postDate": "2022-11-30T17:24:44.847Z",
      "content": "<h1>Congratulations Grandmaster Amed !</h1>\n<p>Congratulations Amed ( <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> ) for becoming Kaggle's newest Competition Grandmaster! Amed was our teams' superstar who designed most of our models. He truly earns the title Kaggle Competition Grandmaster!</p>\n<h1>3rd Place Solution Summary</h1>\n<p>Our solution is an ensemble of 24 models without using pseudo labels. Most models use Deberta-v3-large backbone and a variety of tricks. The ensemble was chosen using hill climbing. Teaming with Amed ( <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> ) and CroDoc ( <a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a> ) was a pleasure. Thank you guys!</p>\n<h1>Hill Climbing - CV 0.4420</h1>\n<p>I love hill climbing because it can take lots of models and pick the best small subset of models. (i.e. its like Lasso regression) And it computes ensemble model weights. </p>\n<p>We start with the single model with best CV score (for us was 0.4470). Then we iterate through all other models and pick a second model which helps the most. Then we pick a third etc until new models do not help. Our ensemble without pseudo achieves CV 0.4420. We allowed hill climbing to use <strong>negative weights</strong> which boosted CV <code>+0.0010</code> and private LB <code>+0.0060</code> versus using positive weights only. </p>\n<pre><code>while not STOP:\n    potential_new_best_cv_score = GET_BEST()\n    for k in range( len(MODELS) ):\n        for wgt in range(-0.5,0.51,0.01):\n            potential_ensemble = (1-wgt) * current_best_ensemble + wgt * MODELS[k]\n            cv_score = compute_metric( potential_ensemble )\n            if cv_score &lt; potential_new_best_cv_score: \n                potential_new_best_cv_score = cv_score\n                REMEMBER_THIS_MODEL(k,wgt)\n    current_best_ensemble, STOP = UPDATE_BEST()\n</code></pre>\n<p><img src=\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/hill_climb.png\" alt=\"\"></p>\n<h1>Models - CV 0.4470</h1>\n<p>Each day, we train new diverse models. Then we run hill climbing to see if the new model gets chosen. We don't need to make a new model with a great CV score, we just need to make new models that are diverse. Below are the choices in order which hill climbing picked from our 50 models. We observe that the best CV score models are not chosen first. Instead hill climbing chooses <strong>diverse models</strong>.</p>\n<p>The improvement in ensemble CV score is shown in the plot above. The table below displays single model CV scores. Our best single model without pseudo achieves CV 0.4470</p>\n<table>\n<thead>\n<tr>\n<th>order selected</th>\n<th>backbone</th>\n<th>cv score</th>\n<th>weight</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>deberta-v3-large</td>\n<td>0.447</td>\n<td>0.190</td>\n</tr>\n<tr>\n<td>2</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.4524</td>\n<td>0.142</td>\n</tr>\n<tr>\n<td>3</td>\n<td>deberta-v3-large</td>\n<td>0.4498</td>\n<td>0.124</td>\n</tr>\n<tr>\n<td>4</td>\n<td>deberta-large-mnli</td>\n<td>0.4548</td>\n<td>0.068</td>\n</tr>\n<tr>\n<td>5</td>\n<td>deberta-v3-large</td>\n<td>0.4492</td>\n<td>0.133</td>\n</tr>\n<tr>\n<td>6</td>\n<td>xlm-roberta-large</td>\n<td>0.4575</td>\n<td>0.092</td>\n</tr>\n<tr>\n<td>7</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.457</td>\n<td>-0.160</td>\n</tr>\n<tr>\n<td>8</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.4525</td>\n<td>0.101</td>\n</tr>\n<tr>\n<td>9</td>\n<td>deberta-v3-large</td>\n<td>0.4489</td>\n<td>0.125</td>\n</tr>\n<tr>\n<td>10</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.4565</td>\n<td>-0.124</td>\n</tr>\n<tr>\n<td>11</td>\n<td><a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">RAPIDS-SVR</a></td>\n<td>0.4526</td>\n<td>0.083</td>\n</tr>\n<tr>\n<td>12</td>\n<td>deberta-v3-large</td>\n<td>0.4495</td>\n<td>0.102</td>\n</tr>\n<tr>\n<td>13</td>\n<td>deberta-v3-large</td>\n<td>0.4502</td>\n<td>0.081</td>\n</tr>\n<tr>\n<td>14</td>\n<td><a href=\"https://www.kaggle.com/code/electro/deberta-layerwiselr-lastlayerreinit-tensorflow\" target=\"_blank\">TF-deberta-v3-base</a></td>\n<td>0.4554</td>\n<td>0.061</td>\n</tr>\n<tr>\n<td>15</td>\n<td>deberta-v3-large</td>\n<td>0.4527</td>\n<td>-0.069</td>\n</tr>\n<tr>\n<td>16</td>\n<td>deberta-v3-large</td>\n<td>0.4522</td>\n<td>-0.058</td>\n</tr>\n<tr>\n<td>17</td>\n<td>deberta-v3-large</td>\n<td>0.45</td>\n<td>0.061</td>\n</tr>\n<tr>\n<td>18</td>\n<td>deberta-v3-large</td>\n<td>0.4516</td>\n<td>-0.058</td>\n</tr>\n<tr>\n<td>19</td>\n<td>deberta-v3-large</td>\n<td>0.4509</td>\n<td>0.071</td>\n</tr>\n<tr>\n<td>20</td>\n<td>deberta-v3-base</td>\n<td>0.4575</td>\n<td>-0.039</td>\n</tr>\n<tr>\n<td>21</td>\n<td>deberta-v3-large-squad2</td>\n<td>0.4501</td>\n<td>0.051</td>\n</tr>\n<tr>\n<td>22</td>\n<td>deberta-v3-large</td>\n<td>0.4512</td>\n<td>-0.049</td>\n</tr>\n<tr>\n<td>23</td>\n<td>roberta-large</td>\n<td>0.4571</td>\n<td>0.030</td>\n</tr>\n<tr>\n<td>24</td>\n<td>deberta-v3-large</td>\n<td>0.45</td>\n<td>0.040</td>\n</tr>\n</tbody>\n</table>\n<h1>What Worked - Tricks</h1>\n<p>Our models use a diversity of regression heads, pooling techniques, learning schedules, and backbones. Some models used the following tricks:</p>\n<ul>\n<li>Different loss rates per target <code>{'cohesion':0.21, 'syntax':0.16, 'vocabulary':0.10, 'phraseology':0.16, 'grammar':0.21, 'conventions':0.16}</code> [High Impact] </li>\n<li>Clip grad norm with <code>max_norm = 10</code> [High Impact]</li>\n<li><code>hidden_dropout_prob = 0.0</code> and <code>attention_probs_dropout_prob = 0.0</code> [High Impact]</li>\n<li>Stride window for non deberta models [Medium Impact] </li>\n<li>Since Deberta tokenizer ignores \"\\n\" we replaced \"\\n\\n\" with \"|\" [Medium Impact] </li>\n<li>Last layer reinitialisation [Medium Impact] </li>\n<li>2 stage pooling. First pool either words, sentences, or paragraphs. Then pool that result. [Medium Impact] </li>\n<li>Train with <code>max_len=2048</code>, infer <code>max_len = 640</code> [High Impact] </li>\n<li>Train with <code>batch_size = 1</code> [High Impact] </li>\n<li>RAPIDS SVR using embeddings without train on comp data [Medium Impact] </li>\n</ul>\n<h1>What Didn't Work - Pseudo Labels</h1>\n<p>Our final submission <strong>does not</strong> use pseudo labels. We tried using pseudo labels during the competition. We were careful to avoid leaks, but none-the-less using pseudo labels from Feedback Prize 1 competition boosted our single model best CV score from 0.4470 to 0.4370 and did not improve our LB score. There must be a leak somewhere but we still haven't found it.</p>\n<p>We noticed that the distribution of targets from FP1 is different than FP3. So care must to be taken to account for this difference in distribution. Below are histogram of pseudo labels on FP1 vs. pseudo labels on FP3. We observe that the old Feedback Prize 1 comp has greater target values than current Feedback Prize 3 comp:</p>\n<p><img src=\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/shift.png\" alt=\"\"></p>\n<h1>Team Members</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a></li>\n<li><a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a></li>\n<li><a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> </li>\n</ul>\n<h1>Solution Code Published</h1>\n<p>We published our training code on GitHub <a href=\"https://github.com/Amed1710/Feedback-Prize--English-Language-Learning\" target=\"_blank\">here</a> and we published our inference code on Kaggle <a href=\"https://www.kaggle.com/code/cdeotte/3rd-place-solution-lb-0-4337-cv-0-4420\" target=\"_blank\">here</a></p>",
      "rawMarkdown": "# Congratulations Grandmaster Amed !\nCongratulations Amed ( @amedprof ) for becoming Kaggle's newest Competition Grandmaster! Amed was our teams' superstar who designed most of our models. He truly earns the title Kaggle Competition Grandmaster!\n\n# 3rd Place Solution Summary\nOur solution is an ensemble of 24 models without using pseudo labels. Most models use Deberta-v3-large backbone and a variety of tricks. The ensemble was chosen using hill climbing. Teaming with Amed ( @amedprof ) and CroDoc ( @crodoc ) was a pleasure. Thank you guys!\n\n# Hill Climbing - CV 0.4420\nI love hill climbing because it can take lots of models and pick the best small subset of models. (i.e. its like Lasso regression) And it computes ensemble model weights. \n  \nWe start with the single model with best CV score (for us was 0.4470). Then we iterate through all other models and pick a second model which helps the most. Then we pick a third etc until new models do not help. Our ensemble without pseudo achieves CV 0.4420. We allowed hill climbing to use **negative weights** which boosted CV `+0.0010` and private LB `+0.0060` versus using positive weights only. \n\n    while not STOP:\n        potential_new_best_cv_score = GET_BEST()\n        for k in range( len(MODELS) ):\n            for wgt in range(-0.5,0.51,0.01):\n                potential_ensemble = (1-wgt) * current_best_ensemble + wgt * MODELS[k]\n                cv_score = compute_metric( potential_ensemble )\n                if cv_score < potential_new_best_cv_score: \n                    potential_new_best_cv_score = cv_score\n                    REMEMBER_THIS_MODEL(k,wgt)\n        current_best_ensemble, STOP = UPDATE_BEST()\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/hill_climb.png)\n\n# Models - CV 0.4470\nEach day, we train new diverse models. Then we run hill climbing to see if the new model gets chosen. We don't need to make a new model with a great CV score, we just need to make new models that are diverse. Below are the choices in order which hill climbing picked from our 50 models. We observe that the best CV score models are not chosen first. Instead hill climbing chooses **diverse models**.\n\nThe improvement in ensemble CV score is shown in the plot above. The table below displays single model CV scores. Our best single model without pseudo achieves CV 0.4470\n\n| order selected | backbone  | cv score | weight |\n| --- | --- | --- | --- |\n|1|deberta-v3-large|0.447|0.190|\n|2|deberta-v3-large-squad2|0.4524|0.142|\n|3|deberta-v3-large|0.4498|0.124|\n|4|deberta-large-mnli|0.4548|0.068|\n|5|deberta-v3-large|0.4492|0.133|\n|6|xlm-roberta-large|0.4575|0.092|\n|7|deberta-v3-large-squad2|0.457|-0.160|\n|8|deberta-v3-large-squad2|0.4525|0.101|\n|9|deberta-v3-large|0.4489|0.125|\n|10|deberta-v3-large-squad2|0.4565|-0.124|\n|11|[RAPIDS-SVR][3]|0.4526|0.083|\n|12|deberta-v3-large|0.4495|0.102|\n|13|deberta-v3-large|0.4502|0.081|\n|14|[TF-deberta-v3-base][4]|0.4554|0.061|\n|15|deberta-v3-large|0.4527|-0.069|\n|16|deberta-v3-large|0.4522|-0.058|\n|17|deberta-v3-large|0.45|0.061|\n|18|deberta-v3-large|0.4516|-0.058|\n|19|deberta-v3-large|0.4509|0.071|\n|20|deberta-v3-base|0.4575|-0.039|\n|21|deberta-v3-large-squad2|0.4501|0.051|\n|22|deberta-v3-large|0.4512|-0.049|\n|23|roberta-large|0.4571|0.030|\n|24|deberta-v3-large|0.45|0.040|\n\n# What Worked - Tricks\nOur models use a diversity of regression heads, pooling techniques, learning schedules, and backbones. Some models used the following tricks:\n  \n* Different loss rates per target `{'cohesion':0.21, 'syntax':0.16, 'vocabulary':0.10, 'phraseology':0.16, 'grammar':0.21, 'conventions':0.16}` [High Impact] \n* Clip grad norm with `max_norm = 10` [High Impact]\n* `hidden_dropout_prob = 0.0` and `attention_probs_dropout_prob = 0.0` [High Impact]\n* Stride window for non deberta models [Medium Impact] \n* Since Deberta tokenizer ignores \"\\n\" we replaced \"\\n\\n\" with \"|\" [Medium Impact] \n* Last layer reinitialisation [Medium Impact] \n* 2 stage pooling. First pool either words, sentences, or paragraphs. Then pool that result. [Medium Impact] \n* Train with `max_len=2048`, infer `max_len = 640` [High Impact] \n* Train with `batch_size = 1` [High Impact] \n* RAPIDS SVR using embeddings without train on comp data [Medium Impact] \n\n# What Didn't Work - Pseudo Labels\nOur final submission **does not** use pseudo labels. We tried using pseudo labels during the competition. We were careful to avoid leaks, but none-the-less using pseudo labels from Feedback Prize 1 competition boosted our single model best CV score from 0.4470 to 0.4370 and did not improve our LB score. There must be a leak somewhere but we still haven't found it.\n\nWe noticed that the distribution of targets from FP1 is different than FP3. So care must to be taken to account for this difference in distribution. Below are histogram of pseudo labels on FP1 vs. pseudo labels on FP3. We observe that the old Feedback Prize 1 comp has greater target values than current Feedback Prize 3 comp:\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Nov-2022/shift.png)\n\n# Team Members\n* @amedprof\n* @crodoc\n* @cdeotte \n\n# Solution Code Published\nWe published our training code on GitHub [here][1] and we published our inference code on Kaggle [here][2]\n\n[1]: https://github.com/Amed1710/Feedback-Prize--English-Language-Learning\n[2]: https://www.kaggle.com/code/cdeotte/3rd-place-solution-lb-0-4337-cv-0-4420\n[3]: https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\n[4]: https://www.kaggle.com/code/electro/deberta-layerwiselr-lastlayerreinit-tensorflow",
      "votes": 141
    },
    {
      "id": 2050365,
      "postDate": "2022-11-30T17:34:15.240Z",
      "content": "<p>Thanks for sharing your results and congrats to all, especially <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> ! Many of tricks we experimented or thought of during the competition phrase but this one got my attention:</p>\n<p><code>Train with max_len=2048, infer max_len = 640</code></p>\n<p>What's the exact motivation behind this trick? Thanks…</p>",
      "rawMarkdown": "Thanks for sharing your results and congrats to all, especially @amedprof ! Many of tricks we experimented or thought of during the competition phrase but this one got my attention:\n\n`Train with max_len=2048, infer max_len = 640`\n\nWhat's the exact motivation behind this trick? Thanks...",
      "votes": 9,
      "replies": [
        {
          "id": 2050372,
          "postDate": "2022-11-30T17:38:26.340Z",
          "content": "<p>IMO Models want the most train data possible. Training with <code>max_len=2048</code> allows the model to learn from all the provided train text. Once we have a smart model, it does not need to see the entire test essay to predict the targets. (And just like humans if you show it too much maybe it gets confused or overwhelmed). Hence during inference, it can use less. (i.e. Humans read the first paragraph and already have a good sense of the essay's final target scores).</p>\n<p>We confirmed this on CV and LB. First train a model with 2048. Then create OOF (using the same model again and again) with different inference lengths like 512, 640, 768, 1024 etc. Compute CV score and submit to LB. Using 640 achieved best CV and LB.</p>",
          "rawMarkdown": "IMO Models want the most train data possible. Training with `max_len=2048` allows the model to learn from all the provided train text. Once we have a smart model, it does not need to see the entire test essay to predict the targets. (And just like humans if you show it too much maybe it gets confused or overwhelmed). Hence during inference, it can use less. (i.e. Humans read the first paragraph and already have a good sense of the essay's final target scores).\n\nWe confirmed this on CV and LB. First train a model with 2048. Then create OOF (using the same model again and again) with different inference lengths like 512, 640, 768, 1024 etc. Compute CV score and submit to LB. Using 640 achieved best CV and LB.",
          "votes": 29
        },
        {
          "id": 2050394,
          "postDate": "2022-11-30T17:45:28.580Z",
          "content": "<p>Thanks for the explanation! Makes sense.</p>",
          "rawMarkdown": "Thanks for the explanation! Makes sense.",
          "votes": 3
        },
        {
          "id": 2050434,
          "postDate": "2022-11-30T18:21:59.613Z",
          "content": "<p><a href=\"https://www.kaggle.com/datafan07\" target=\"_blank\">@datafan07</a> Also note that Deberta uses \"relative embeddings\". Therefore if WORD_A in position 1500 relates to the word in position 1501. Then the model learns that WORD_A needs to pay attention to the word in position +1. Therefore when we infer with length 640 and the model sees WORD_A in position 550, it will look at position 551. In other words, the learnings from training with max_len=2028 will generalize to inference at max_len=640. (I think this is how it works)</p>",
          "rawMarkdown": "@datafan07 Also note that Deberta uses \"relative embeddings\". Therefore if WORD_A in position 1500 relates to the word in position 1501. Then the model learns that WORD_A needs to pay attention to the word in position +1. Therefore when we infer with length 640 and the model sees WORD_A in position 550, it will look at position 551. In other words, the learnings from training with max_len=2028 will generalize to inference at max_len=640. (I think this is how it works)",
          "votes": 20
        }
      ]
    },
    {
      "id": 2050709,
      "postDate": "2022-11-30T23:48:17.800Z",
      "content": "<p>Congratulations! Thanks for the excellent sharing!  Your team did geat work to make diverse models to help ensemble!<br>\nI learned a lot especially below:</p>\n<ol>\n<li>Non deberta using slide window seems help a lot, I could see you have non deberta models like roberta with good cv.</li>\n<li>Train long and infer short is creative, as before we always train infer using same, or train using short and infer using long(only concidering training using more GPU mem), and I only tried same length train/infer here, for me larger length work a bit better.</li>\n<li>Analysis error rate for each target and tune for loss rate per target is cool</li>\n<li>hill climb, negative model weights helps, another sharing also mentioned this.</li>\n</ol>",
      "rawMarkdown": "Congratulations! Thanks for the excellent sharing!  Your team did geat work to make diverse models to help ensemble!\nI learned a lot especially below:\n1. Non deberta using slide window seems help a lot, I could see you have non deberta models like roberta with good cv.\n2. Train long and infer short is creative, as before we always train infer using same, or train using short and infer using long(only concidering training using more GPU mem), and I only tried same length train/infer here, for me larger length work a bit better.\n3. Analysis error rate for each target and tune for loss rate per target is cool\n4. hill climb, negative model weights helps, another sharing also mentioned this.",
      "votes": 8
    },
    {
      "id": 2050412,
      "postDate": "2022-11-30T18:01:04.473Z",
      "content": "<p>Hearty congratulations to <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a>, <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> and <a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a> for a great result and thanks to <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a>  for the detailed approach note too!</p>",
      "rawMarkdown": "Hearty congratulations to @amedprof, @cdeotte and @crodoc for a great result and thanks to @cdeotte  for the detailed approach note too!",
      "votes": 5
    },
    {
      "id": 2050362,
      "postDate": "2022-11-30T17:30:41.533Z",
      "content": "<p>Amazing teammates! It was really nice to work with both of you. Also, I think you really wrote nicely the key points that contributed to our final solution.</p>",
      "rawMarkdown": "Amazing teammates! It was really nice to work with both of you. Also, I think you really wrote nicely the key points that contributed to our final solution.",
      "votes": 5
    },
    {
      "id": 2050377,
      "postDate": "2022-11-30T17:40:17.780Z",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a>  for your kind words.<br>\nI really enjoyed working with you.</p>",
      "rawMarkdown": "Thank you @cdeotte  for your kind words.\nI really enjoyed working with you.",
      "votes": 6
    },
    {
      "id": 2058634,
      "postDate": "2022-12-08T05:10:25.640Z",
      "content": "<p>Thank you for sharing!<br>\nDid you try MLM?<br>\nI have a question about MLM.<br>\nWhen doing MLM using training data, should I create a model for each fold to account for data leakages?</p>\n<p>Ex)<br>\n[Setting] df = (FB3 data), train = df[df['fold'] != 0], valid = df[df['fold'] == 0],  pl_data = (FB1 data)<br>\nIn this situation, do MLM training with train&amp;pl_data and evaluation with valid.<br>\nDo this for the number of folds.</p>",
      "rawMarkdown": "Thank you for sharing!\nDid you try MLM?\nI have a question about MLM.\nWhen doing MLM using training data, should I create a model for each fold to account for data leakages?\n\nEx)\n[Setting] df = (FB3 data), train = df[df['fold'] != 0], valid = df[df['fold'] == 0],  pl_data = (FB1 data)\nIn this situation, do MLM training with train&pl_data and evaluation with valid.\nDo this for the number of folds.",
      "votes": 3,
      "replies": [
        {
          "id": 2059319,
          "postDate": "2022-12-08T17:51:30.990Z",
          "content": "<p>Our final solution does not use MLM and I did not try MLM. Perhaps <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> or <a href=\"https://www.kaggle.com/crodoc\" target=\"_blank\">@crodoc</a> tried MLM.</p>",
          "rawMarkdown": "Our final solution does not use MLM and I did not try MLM. Perhaps @amedprof or @crodoc tried MLM.",
          "votes": 3
        },
        {
          "id": 2059452,
          "postDate": "2022-12-08T21:29:59.047Z",
          "content": "<p>I think Amed tried it and said it didn't help.</p>",
          "rawMarkdown": "I think Amed tried it and said it didn't help.",
          "votes": 3
        },
        {
          "id": 2059521,
          "postDate": "2022-12-09T00:55:19.580Z",
          "content": "<p>Yes, I tried MLM (and the pre-trained weight on Feedback 2) without success. Which was quite surprising.</p>",
          "rawMarkdown": "Yes, I tried MLM (and the pre-trained weight on Feedback 2) without success. Which was quite surprising.",
          "votes": 2
        },
        {
          "id": 2060106,
          "postDate": "2022-12-09T14:53:16.757Z",
          "content": "<p>Thank you! How about how to do MLM?Did you do it like above?</p>",
          "rawMarkdown": "Thank you! How about how to do MLM?Did you do it like above?",
          "votes": 1
        },
        {
          "id": 2060285,
          "postDate": "2022-12-09T17:23:22.777Z",
          "content": "<p>Yes.<br>\nBut you need to remove the duplicates essays between FB1 and FB3 before doing MLM.</p>",
          "rawMarkdown": "Yes.\nBut you need to remove the duplicates essays between FB1 and FB3 before doing MLM.",
          "votes": 1
        },
        {
          "id": 2060291,
          "postDate": "2022-12-09T17:38:36.857Z",
          "content": "<p>Thank you.<br>\nCongratulations on new GM!</p>",
          "rawMarkdown": "Thank you.\nCongratulations on new GM!",
          "votes": 1
        }
      ]
    },
    {
      "id": 2056471,
      "postDate": "2022-12-06T06:45:07.340Z",
      "content": "<p>Can you explain what exactly \"Stride window for non deberta models\" does in your solution?😄</p>",
      "rawMarkdown": "Can you explain what exactly \"Stride window for non deberta models\" does in your solution?😄",
      "votes": 3,
      "replies": [
        {
          "id": 2057238,
          "postDate": "2022-12-06T22:00:00.377Z",
          "content": "<p>Models like RoBERTa can only accept input with <code>max_len = 512</code> tokens. So to train with sequences of len 2048, we need to train and infer using a sliding window.</p>",
          "rawMarkdown": "Models like RoBERTa can only accept input with `max_len = 512` tokens. So to train with sequences of len 2048, we need to train and infer using a sliding window.",
          "votes": 2
        },
        {
          "id": 2057325,
          "postDate": "2022-12-07T01:31:17.427Z",
          "content": "<p>For example, the length of 2048 can be split into four lengths of 512, that is, one sample is split into four samples, and they have the same label. Then will it train on these 4 samples?</p>",
          "rawMarkdown": "For example, the length of 2048 can be split into four lengths of 512, that is, one sample is split into four samples, and they have the same label. Then will it train on these 4 samples?",
          "votes": 2
        },
        {
          "id": 2057341,
          "postDate": "2022-12-07T01:43:43.077Z",
          "content": "<p>Yes basically. However with sliding window, we also pick a stride length. So if <code>max_len=512</code> and <code>doc_stride=256</code>. Then the you break it into 7 pieces, namely 0:512, 256:768, 512:1024, 768:1280, 1024:1536, 1280:1792, 1536:2024. The pieces overlap by <code>max_len minus doc_stride</code>.</p>",
          "rawMarkdown": "Yes basically. However with sliding window, we also pick a stride length. So if `max_len=512` and `doc_stride=256`. Then the you break it into 7 pieces, namely 0:512, 256:768, 512:1024, 768:1280, 1024:1536, 1280:1792, 1536:2024. The pieces overlap by `max_len minus doc_stride`.",
          "votes": 3
        },
        {
          "id": 2057362,
          "postDate": "2022-12-07T03:06:17.693Z",
          "content": "<p>Thank you for your detailed reply, and best of luck to you in the future</p>",
          "rawMarkdown": "Thank you for your detailed reply, and best of luck to you in the future",
          "votes": 2
        }
      ]
    },
    {
      "id": 2051569,
      "postDate": "2022-12-01T13:57:44.467Z",
      "content": "<p>Can you explain how to choose max_norm=10 for gradient clipping?</p>",
      "rawMarkdown": "Can you explain how to choose max_norm=10 for gradient clipping?",
      "votes": 3,
      "replies": [
        {
          "id": 2052397,
          "postDate": "2022-12-02T06:58:25.983Z",
          "content": "<p>gradient clipping made our models more stable.<br>\nIn the end, It was just tuning this parameter. I found that max_norm=10 was the best.</p>",
          "rawMarkdown": "gradient clipping made our models more stable.\nIn the end, It was just tuning this parameter. I found that max_norm=10 was the best.",
          "votes": 2
        }
      ]
    },
    {
      "id": 2051431,
      "postDate": "2022-12-01T12:08:23.763Z",
      "content": "<p><a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> congratulations!! Impressive achievement unlocked! </p>",
      "rawMarkdown": "@amedprof congratulations!! Impressive achievement unlocked! ",
      "votes": 3,
      "replies": [
        {
          "id": 2052405,
          "postDate": "2022-12-02T07:09:14.380Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/optimo\" target=\"_blank\">@optimo</a> :).<br>\nI hope you will soon.</p>",
          "rawMarkdown": "Thanks @optimo :).\nI hope you will soon.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2050836,
      "postDate": "2022-12-01T02:59:03.227Z",
      "content": "<p>Congratulations on Gold Medal🥇 and thank you for sharing your solution.</p>\n<p>I have one question.</p>\n<blockquote>\n  <p>Train with batch_size = 1</p>\n</blockquote>\n<p>I’m not sure about it. It is not mini-batch gradient descent.<br>\nIs simple online stochastic gradient descent (not using mini-batch) better in this competition?</p>",
      "rawMarkdown": "Congratulations on Gold Medal🥇 and thank you for sharing your solution.\n\nI have one question.\n\n> Train with batch_size = 1\n\nI’m not sure about it. It is not mini-batch gradient descent.\nIs simple online stochastic gradient descent (not using mini-batch) better in this competition?",
      "votes": 4,
      "replies": [
        {
          "id": 2052412,
          "postDate": "2022-12-02T07:14:22.737Z",
          "content": "<blockquote>\n  <p>Is simple online stochastic gradient descent (not using mini-batch) better in this competition?</p>\n</blockquote>\n<p>It was better for us.<br>\nAnd our models with '2 x Pooling' only support <code>batch_size = 1</code></p>",
          "rawMarkdown": "> Is simple online stochastic gradient descent (not using mini-batch) better in this competition?\n\nIt was better for us.\nAnd our models with '2 x Pooling' only support `batch_size = 1`",
          "votes": 3
        },
        {
          "id": 2053139,
          "postDate": "2022-12-02T21:28:11.990Z",
          "content": "<p>Thank you for your response!<br>\nI have never tried batch size=1 before, so I will add it to my experiments next time😊</p>",
          "rawMarkdown": "Thank you for your response!\nI have never tried batch size=1 before, so I will add it to my experiments next time😊",
          "votes": 3
        }
      ]
    },
    {
      "id": 2050541,
      "postDate": "2022-11-30T20:25:36.760Z",
      "content": "<p>Congratulations! Really deserve it!<br>\nJust one question, could I know what is your validation scheme? and whether it was correlating well with the LB? Jumping really high in the lb has definietly something to do with the validation right? :)</p>",
      "rawMarkdown": "Congratulations! Really deserve it!\nJust one question, could I know what is your validation scheme? and whether it was correlating well with the LB? Jumping really high in the lb has definietly something to do with the validation right? :)",
      "votes": 4,
      "replies": [
        {
          "id": 2050565,
          "postDate": "2022-11-30T20:48:41.923Z",
          "content": "<p>Validation is key. \"Trust Your CV\" is king. We picked all our final submissions based on CV score and ignored LB score because LB is the size of 1 fold, and train data CV is the size of 5 folds. A single fold is not as reliable as the full 5 folds CV score to estimate private LB score. </p>\n<p>All our models were trained using the same 5 MultilabelStratifiedKFold (seed 42) as seen in public notebooks. For the most part improvements in CV resulted in improvements in LB. But the LB data was the size of 1 fold of 5 fold. And behaved a little different than CV score at times.</p>",
          "rawMarkdown": "Validation is key. \"Trust Your CV\" is king. We picked all our final submissions based on CV score and ignored LB score because LB is the size of 1 fold, and train data CV is the size of 5 folds. A single fold is not as reliable as the full 5 folds CV score to estimate private LB score. \n\nAll our models were trained using the same 5 MultilabelStratifiedKFold (seed 42) as seen in public notebooks. For the most part improvements in CV resulted in improvements in LB. But the LB data was the size of 1 fold of 5 fold. And behaved a little different than CV score at times.",
          "votes": 8
        },
        {
          "id": 2050645,
          "postDate": "2022-11-30T22:35:23.060Z",
          "content": "<p>Thanks for the clarification! Honestly, I always be so happy when finding you in a competition I work on. You and the others top GMs make it so fun! Hope one day I can team up with you.</p>",
          "rawMarkdown": "Thanks for the clarification! Honestly, I always be so happy when finding you in a competition I work on. You and the others top GMs make it so fun! Hope one day I can team up with you.",
          "votes": 3
        }
      ]
    },
    {
      "id": 2050420,
      "postDate": "2022-11-30T18:10:48.780Z",
      "content": "<p>Congratulations to your team for landing a top 3 finish!! </p>\n<p>I had a couple of additional questions</p>\n<hr>\n<blockquote>\n  <p>Different loss rates per target {'cohesion':0.21, 'syntax':0.16, 'vocabulary':0.10, 'phraseology':0.16, 'grammar':0.21, 'conventions':0.16}</p>\n</blockquote>\n<p>What was the methodology to determine the loss rates? </p>\n<hr>\n<blockquote>\n  <p>2 stage pooling. First pool either words, sentences, or paragraphs. Then pool that result.</p>\n</blockquote>\n<p>Can you provide some more detail regarding this? I am not able to follow what this is</p>\n<hr>\n<p>Thank you for sharing your approach to the competition!</p>",
      "rawMarkdown": "Congratulations to your team for landing a top 3 finish!! \n\nI had a couple of additional questions\n\n------\n\n> Different loss rates per target {'cohesion':0.21, 'syntax':0.16, 'vocabulary':0.10, 'phraseology':0.16, 'grammar':0.21, 'conventions':0.16}\n\nWhat was the methodology to determine the loss rates? \n\n-----\n\n> 2 stage pooling. First pool either words, sentences, or paragraphs. Then pool that result.\n\nCan you provide some more detail regarding this? I am not able to follow what this is\n\n-----\n\nThank you for sharing your approach to the competition!",
      "votes": 4,
      "replies": [
        {
          "id": 2050441,
          "postDate": "2022-11-30T18:29:31.720Z",
          "content": "<p><a href=\"https://www.kaggle.com/amed\" target=\"_blank\">@amed</a> can share the details. These are two of his tricks.</p>\n<p>I think the way \"2 x pooling\" works is as follows. Amed please correct me if i'm wrong. A transformer model outputs embeddings for each token which are pieces of words. Most popular notebook, use mean pooling over all these token outputs. Therefore each token has equal importance.</p>\n<p>When doing paragraph pooling, we first takes the mean of all tokens in each paragraph separately. We now have a single embedding for each paragraph. Next we take the mean of these embeddings. This gives each paragraph equal weight. </p>\n<p>This is different than normal mean pooling. With normal mean pooling, if one paragraph has twice as many tokens as another paragraph, then the first paragraph would be twice as important when taking average of tokens.</p>\n<p>Amed, can you share how you discovered the different learning rates per target?</p>",
          "rawMarkdown": "@amed can share the details. These are two of his tricks.\n\nI think the way \"2 x pooling\" works is as follows. Amed please correct me if i'm wrong. A transformer model outputs embeddings for each token which are pieces of words. Most popular notebook, use mean pooling over all these token outputs. Therefore each token has equal importance.\n\nWhen doing paragraph pooling, we first takes the mean of all tokens in each paragraph separately. We now have a single embedding for each paragraph. Next we take the mean of these embeddings. This gives each paragraph equal weight. \n\nThis is different than normal mean pooling. With normal mean pooling, if one paragraph has twice as many tokens as another paragraph, then the first paragraph would be twice as important when taking average of tokens.\n\nAmed, can you share how you discovered the different learning rates per target?",
          "votes": 10
        },
        {
          "id": 2050521,
          "postDate": "2022-11-30T20:09:52.530Z",
          "content": "<p>Regarding <strong>\"2 x Pooling\",</strong> <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> explained it really well.<br>\nWe tried different types of Pooling for each level.<br>\nFor example :</p>\n<ul>\n<li>MaxPooling on each paragraph and MeanPooling on that result</li>\n<li>AttentionHead on each paragraph and MeanPooling on that result</li>\n<li>etc…</li>\n</ul>\n<blockquote>\n  <p>Amed, can you share how you discovered the different learning rates per target?</p>\n</blockquote>\n<p>When training our models, we could see in the image below that the <strong>cohesion</strong> and <strong>grammar</strong>  targets had the highest error rate, which explains why we gave more weight to these targets when computing the loss function.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2852238%2Fb24cd06928c67ca91f3e1cfd978a9c2d%2Fimage%20(1).png?generation=1669837101207255&amp;alt=media\" alt=\"\"></p>\n<p>We tried different combinaison of loss rates per target, and best combinaison was: <code>{'cohesion':0.21, 'syntax':0.16, 'vocabulary':0.10, 'phraseology':0.16, 'grammar':0.21, 'conventions':0.16}</code></p>",
          "rawMarkdown": "Regarding **\"2 x Pooling\",** @cdeotte explained it really well.\nWe tried different types of Pooling for each level.\nFor example :\n- MaxPooling on each paragraph and MeanPooling on that result\n- AttentionHead on each paragraph and MeanPooling on that result\n- etc...\n\n> Amed, can you share how you discovered the different learning rates per target?\n\nWhen training our models, we could see in the image below that the **cohesion** and **grammar**  targets had the highest error rate, which explains why we gave more weight to these targets when computing the loss function.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2852238%2Fb24cd06928c67ca91f3e1cfd978a9c2d%2Fimage%20(1).png?generation=1669837101207255&alt=media)\n\nWe tried different combinaison of loss rates per target, and best combinaison was: `{'cohesion':0.21, 'syntax':0.16, 'vocabulary':0.10, 'phraseology':0.16, 'grammar':0.21, 'conventions':0.16}`",
          "votes": 14
        },
        {
          "id": 2050701,
          "postDate": "2022-11-30T23:41:44.470Z",
          "content": "<p>Thank you both for clarifying! Happy to learn these awesome tricks! 😄</p>\n<p>Congratulations again!</p>",
          "rawMarkdown": "Thank you both for clarifying! Happy to learn these awesome tricks! 😄\n\nCongratulations again!",
          "votes": 2
        }
      ]
    },
    {
      "id": 2050390,
      "postDate": "2022-11-30T17:44:09.490Z",
      "content": "<p>Congratulations on your gold medal!!<br>\nI'm always learning from you!💪</p>",
      "rawMarkdown": "Congratulations on your gold medal!!\nI'm always learning from you!💪",
      "votes": 4
    },
    {
      "id": 2054442,
      "postDate": "2022-12-04T06:35:16.590Z",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!",
      "votes": 1
    },
    {
      "id": 2071297,
      "postDate": "2022-12-20T21:03:15.223Z",
      "content": "<p>A huge Congratulations Amed for reaching the Most difficult Kaggle step: Competition GM.</p>\n<p>Also congratulations for the whole team (amedprof, crodoc, cdeotte) and the amazing solution. </p>",
      "rawMarkdown": "A huge Congratulations Amed for reaching the Most difficult Kaggle step: Competition GM.\n\nAlso congratulations for the whole team (amedprof, crodoc, cdeotte) and the amazing solution. ",
      "votes": 2,
      "replies": [
        {
          "id": 2071300,
          "postDate": "2022-12-20T21:09:20.547Z",
          "content": "<p>Thanks Marília!</p>",
          "rawMarkdown": "Thanks Marília!",
          "votes": 2
        }
      ]
    },
    {
      "id": 2058009,
      "postDate": "2022-12-07T14:39:40.940Z",
      "content": "<p>Congrats Amed! Do you have any idea as to why pseudo labels didn't work?</p>",
      "rawMarkdown": "Congrats Amed! Do you have any idea as to why pseudo labels didn't work?",
      "votes": 2,
      "replies": [
        {
          "id": 2058827,
          "postDate": "2022-12-08T08:40:53.773Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/perrypineapple\" target=\"_blank\">@perrypineapple</a>.<br>\nWhen doing pseudo-labels, you need to be sure that all the models used to generate the pseudo-labels are trained on the same cv split.<br>\nI suspect that one of our models was trained on a different split. This might have introduced a leak.</p>",
          "rawMarkdown": "Thanks @perrypineapple.\nWhen doing pseudo-labels, you need to be sure that all the models used to generate the pseudo-labels are trained on the same cv split.\nI suspect that one of our models was trained on a different split. This might have introduced a leak."
        }
      ]
    },
    {
      "id": 2057153,
      "postDate": "2022-12-06T19:57:49.173Z",
      "content": "<p>Amazing Work !!</p>",
      "rawMarkdown": "Amazing Work !!",
      "votes": 2
    },
    {
      "id": 2056487,
      "postDate": "2022-12-06T06:56:28.163Z",
      "content": "<p>Congratulations</p>",
      "rawMarkdown": "Congratulations",
      "votes": 2
    },
    {
      "id": 2056374,
      "postDate": "2022-12-06T04:28:51.760Z",
      "content": "<p>congratulations, this is very inspiring and a new insight for me. </p>",
      "rawMarkdown": "congratulations, this is very inspiring and a new insight for me. ",
      "votes": 2
    },
    {
      "id": 2055216,
      "postDate": "2022-12-04T22:04:54.400Z",
      "content": "<p>Nice Work!</p>",
      "rawMarkdown": "Nice Work!",
      "votes": 2
    },
    {
      "id": 2055151,
      "postDate": "2022-12-04T19:03:44.033Z",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!",
      "votes": 2
    },
    {
      "id": 2054765,
      "postDate": "2022-12-04T12:51:34.700Z",
      "content": "<p>congratulations!</p>",
      "rawMarkdown": "congratulations!",
      "votes": 2
    },
    {
      "id": 2054298,
      "postDate": "2022-12-04T02:25:07.540Z",
      "content": "<p>Congratulations! Really deserve it! i'm really interested in Hill Climbing. Could you please show some detailed infer about this part.</p>",
      "rawMarkdown": "Congratulations! Really deserve it! i'm really interested in Hill Climbing. Could you please show some detailed infer about this part.",
      "votes": 2
    },
    {
      "id": 2053884,
      "postDate": "2022-12-03T17:24:55.613Z",
      "content": "<p>Well done mate. </p>",
      "rawMarkdown": "Well done mate. ",
      "votes": 2
    },
    {
      "id": 2053344,
      "postDate": "2022-12-03T06:20:01.743Z",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!",
      "votes": 2
    },
    {
      "id": 2052158,
      "postDate": "2022-12-02T01:08:01.430Z",
      "content": "<p>Congratulations! I was able to experiment a lot with the notebook you shared.</p>\n<p>I have a question. I know that the larger the number of fold, the smaller the bias.<br>\nIs it possible to make an absolute comparison of CV scores calculated with different number of fold?<br>\nFor example, can we say that cv_score=0.45 with k=4 and cv_score=0.45 with k=5 are the same performance?</p>",
      "rawMarkdown": "Congratulations! I was able to experiment a lot with the notebook you shared.\n\nI have a question. I know that the larger the number of fold, the smaller the bias.\nIs it possible to make an absolute comparison of CV scores calculated with different number of fold?\nFor example, can we say that cv_score=0.45 with k=4 and cv_score=0.45 with k=5 are the same performance?",
      "votes": 2,
      "replies": [
        {
          "id": 2052161,
          "postDate": "2022-12-02T01:10:43.343Z",
          "content": "<p>When train data is small, then the more folds will achieve a better CV and LB. This is because 4 folds uses 75% train data to train a fold model whereas 5 folds, 10 folds, 25 folds, use 80%, 90%, and 95% respectively.</p>\n<p>When training data is very large like 100's of thousands of samples than even 75% is plenty. But when train data only has a few thousand then every extra sample to train with makes a noticeable difference.</p>",
          "rawMarkdown": "When train data is small, then the more folds will achieve a better CV and LB. This is because 4 folds uses 75% train data to train a fold model whereas 5 folds, 10 folds, 25 folds, use 80%, 90%, and 95% respectively.\n\nWhen training data is very large like 100's of thousands of samples than even 75% is plenty. But when train data only has a few thousand then every extra sample to train with makes a noticeable difference.",
          "votes": 6
        }
      ]
    },
    {
      "id": 2051946,
      "postDate": "2022-12-01T18:44:12.667Z",
      "content": "<p>I see that you tried the deberta versions finetuned with squad2. I also tried with other finetuned versions such as tner/deberta-v3-large-ontonotes5 and Elron/deberta-v3-large-sentiment.</p>\n<p>Those helped me a lot with the CV, however they performed really bad in the leaderboard. I achieved my medal with a model including only plain vanilla versions of roberta/deberta. Any idea why these finetuned versions didn't work?</p>\n<p>I realize in the case of tner, they are trained for NER, but still… they did well with the CV.</p>",
      "rawMarkdown": "I see that you tried the deberta versions finetuned with squad2. I also tried with other finetuned versions such as tner/deberta-v3-large-ontonotes5 and Elron/deberta-v3-large-sentiment.\n\nThose helped me a lot with the CV, however they performed really bad in the leaderboard. I achieved my medal with a model including only plain vanilla versions of roberta/deberta. Any idea why these finetuned versions didn't work?\n\nI realize in the case of tner, they are trained for NER, but still... they did well with the CV.",
      "votes": 2,
      "replies": [
        {
          "id": 2052394,
          "postDate": "2022-12-02T06:53:16.477Z",
          "content": "<p>With our setup, we could achieve a good CV score with <code>deberta-v3-large-squad2</code>, and it brings diversity to our ensemble.<br>\nAs you can see in the table above, <code>deberta-v3-large-squad2</code> was the second model selected by hill climbing.<br>\nI guess the LB score depends on how you ensemble your models.</p>",
          "rawMarkdown": "With our setup, we could achieve a good CV score with `deberta-v3-large-squad2`, and it brings diversity to our ensemble.\nAs you can see in the table above, `deberta-v3-large-squad2` was the second model selected by hill climbing.\nI guess the LB score depends on how you ensemble your models.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2051925,
      "postDate": "2022-12-01T18:14:10.153Z",
      "content": "<p>Nice work!</p>",
      "rawMarkdown": "Nice work!",
      "votes": 2
    },
    {
      "id": 2051344,
      "postDate": "2022-12-01T10:59:23.440Z",
      "content": "<p>Congratulations and thanks for sharing! i have learned something!</p>",
      "rawMarkdown": "Congratulations and thanks for sharing! i have learned something!",
      "votes": 2
    },
    {
      "id": 2051244,
      "postDate": "2022-12-01T09:47:40.593Z",
      "content": "<p>Thanks for the solution, congrats <a href=\"https://www.kaggle.com/amedprof\" target=\"_blank\">@amedprof</a> </p>",
      "rawMarkdown": "Thanks for the solution, congrats @amedprof ",
      "votes": 2
    },
    {
      "id": 2050482,
      "postDate": "2022-11-30T19:33:45.127Z",
      "content": "<p>I remember that you posted a strategy, when there was used a SVR over the pretrained models. Did it work well? Or the ensemble of finetuned models is still better for you?</p>",
      "rawMarkdown": "I remember that you posted a strategy, when there was used a SVR over the pretrained models. Did it work well? Or the ensemble of finetuned models is still better for you?",
      "votes": 2,
      "replies": [
        {
          "id": 2050503,
          "postDate": "2022-11-30T19:54:47.607Z",
          "content": "<p>We did not try to train one SVR from all our NLP transformer embeddings. Instead we trained a separate SVR and included it in our ensemble and it boosted CV LB. The 4th place solution <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369621\" target=\"_blank\">here</a> best single model was SVR</p>",
          "rawMarkdown": "We did not try to train one SVR from all our NLP transformer embeddings. Instead we trained a separate SVR and included it in our ensemble and it boosted CV LB. The 4th place solution [here][1] best single model was SVR\n\n[1]: https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369621",
          "votes": 3
        }
      ]
    },
    {
      "id": 2481456,
      "postDate": "2023-10-14T06:28:18.407Z",
      "content": "<p>Hill Climbing is really useful,thanks for sharing </p>",
      "rawMarkdown": "Hill Climbing is really useful,thanks for sharing "
    },
    {
      "id": 2407185,
      "postDate": "2023-08-24T23:20:39.343Z",
      "content": "<p><a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> When you say 'high impact', do you mean that it increased the score of the individual model, or that it increased the score of the ensemble by adding diversity?</p>",
      "rawMarkdown": "@cdeotte When you say 'high impact', do you mean that it increased the score of the individual model, or that it increased the score of the ensemble by adding diversity?"
    },
    {
      "id": 2113049,
      "postDate": "2023-01-24T03:52:00.367Z",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a>, quick question please - what's the benefit of using <code>CustomCollator</code> in the DataLoader as opposed to <code>pad_to_max_length=True</code> in the DataSet? </p>\n<p>Is this to save memory?</p>",
      "rawMarkdown": "Hey @cdeotte, quick question please - what's the benefit of using `CustomCollator` in the DataLoader as opposed to `pad_to_max_length=True` in the DataSet? \n\nIs this to save memory?"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:02:13Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9090487,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/now-you-see-me-3rd-place-solution-congratulations-"
}