{
  "id": 257302,
  "title": "Private 22nd Place solution",
  "content": "<h1>Acknowledgement</h1>\n<p>First of all, thanks to competition organizers for hosting this competition.<br><br>\nAnd I would like to thank you <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> for share a lot of great notebook and discussion. I couldn't have achieved this ranking without you, and I learned a lot method for fine-tune BERT models from you!<br></p>\n<h1>summary</h1>\n<p>I have three options(below), and in the end I chose the top two for final submission.</p>\n<ul>\n<li>LB best (9models, CV: 0.4449, LB: 0.444, Private LB: 0.452) </li>\n<li>CV best exclude base-model (9models, CV: 0.4430, LB: 0.447, Private LB: 0.452)</li>\n<li>CV best (12models, CV: 0.4420, LB: 0.453, Private LB: 0.452)</li>\n</ul>\n<p>The detail of LB best is below table:<br></p>\n<h2>LB-Best</h2>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV(5fold)</th>\n<th>weight</th>\n<th>Public</th>\n<th>Private</th>\n<th>other</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>luke-large</td>\n<td>0.4795</td>\n<td>0.0877</td>\n<td>0.467</td>\n<td>?</td>\n<td></td>\n</tr>\n<tr>\n<td>roberta-large</td>\n<td>0.4693</td>\n<td>0.1035</td>\n<td>0.464</td>\n<td>0.469</td>\n<td></td>\n</tr>\n<tr>\n<td>gpt2-medium</td>\n<td>0.481</td>\n<td>0.158</td>\n<td>0.469</td>\n<td>0.482</td>\n<td></td>\n</tr>\n<tr>\n<td>funnel-large-base</td>\n<td>0.4706</td>\n<td>0.1455</td>\n<td>0.461</td>\n<td>0.47</td>\n<td></td>\n</tr>\n<tr>\n<td>funnel-large</td>\n<td>0.4705</td>\n<td>0.0267</td>\n<td>0.46</td>\n<td>0.467</td>\n<td></td>\n</tr>\n<tr>\n<td>ernie-large</td>\n<td>0.4712</td>\n<td>0.093</td>\n<td>0.463</td>\n<td>0.477</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td>0.475</td>\n<td>0.1693</td>\n<td>0.464</td>\n<td>0.467</td>\n<td></td>\n</tr>\n<tr>\n<td>mpnet-base</td>\n<td>0.4782</td>\n<td>0.1415</td>\n<td>0.466</td>\n<td>0.474</td>\n<td></td>\n</tr>\n<tr>\n<td>gpt2-large</td>\n<td>0.4744</td>\n<td>0.1121</td>\n<td>0.468</td>\n<td>0.48</td>\n<td>batch_size=4</td>\n</tr>\n<tr>\n<td>coef</td>\n<td>-</td>\n<td>0.0235</td>\n<td>-</td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h2>CV-Best</h2>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV(5fold)</th>\n<th>weight</th>\n<th>Public</th>\n<th>Private</th>\n<th>other</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>electra-large</td>\n<td>0.4743</td>\n<td>0.0829</td>\n<td>0.474</td>\n<td>0.479</td>\n<td></td>\n</tr>\n<tr>\n<td>funnel-large</td>\n<td>0.4752</td>\n<td>0.1215</td>\n<td>0.467</td>\n<td>0.474</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-large(1)</td>\n<td>0.4738</td>\n<td>0.1516</td>\n<td>0.47</td>\n<td>0.463</td>\n<td></td>\n</tr>\n<tr>\n<td>gpt2-medium</td>\n<td>0.4775</td>\n<td>0.1219</td>\n<td>0.471</td>\n<td>0.483</td>\n<td></td>\n</tr>\n<tr>\n<td>luke-large</td>\n<td>0.474</td>\n<td>0.0937</td>\n<td>0.468</td>\n<td>0.465</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-large(2)</td>\n<td>0.475</td>\n<td>0.1179</td>\n<td>0.464</td>\n<td>0.467</td>\n<td></td>\n</tr>\n<tr>\n<td>mpnet-base</td>\n<td>0.4782</td>\n<td>0.0999</td>\n<td>0.466</td>\n<td>0.474</td>\n<td></td>\n</tr>\n<tr>\n<td>ernie-large</td>\n<td>0.4681</td>\n<td>0.0792</td>\n<td>0.47</td>\n<td>0.473</td>\n<td></td>\n</tr>\n<tr>\n<td>gpt2-large</td>\n<td>0.4744</td>\n<td>0.1648</td>\n<td>?</td>\n<td>?</td>\n<td>batch_size=4</td>\n</tr>\n<tr>\n<td>coef</td>\n<td>-</td>\n<td>0.009</td>\n<td>-</td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h1>configuration for almost all models</h1>\n<ul>\n<li>epochs = 4</li>\n<li>optimizer: AdamW</li>\n<li>scheduler: linear_schedule_with_warmup(warmup: 5%)</li>\n<li>lr_bert: 3e-5</li>\n<li>batch_size: 12</li>\n<li>gradient clipping: 0.2~0.5</li>\n<li>reinitialize layers: last 2~6 layers  </li>\n<li>ensemble: Nelder-Mead</li>\n<li>custom head(finally concat all)<ul>\n<li>averaging last 4 hidden layer</li>\n<li>LSTM head</li>\n<li>vocabulary dense</li></ul></li>\n</ul>\n<pre><code>  hidden_states: (batch_size, vocab_size, bert_hidden_size)\n  linear_vocab = nn.Sequential(\n      nn.Linear(bert_hidden_size, 128),\n      nn.GELU(),\n      nn.Linear(128, 64),\n      nn.GELU()\n  )\n  linear_final = nn.Linear(vocab_size * 64, 128)\n  out = linear_vocab(hidden_states).view(len(input_ids), -1)) # final shape: (batch_size, vocab_size * 64)\n  out = linear_final(out) # out shape: (batch_size, 128)\n</code></pre>\n<ul>\n<li>17 hand-made features <ul>\n<li>sentence count</li>\n<li>average character count in documents</li></ul></li>\n</ul>\n<h1>worked for me</h1>\n<ul>\n<li>model ensemble: I thought diversity is the most important thing in this competition.<ul>\n<li>At the beginning of the competition, I tested the effectiveness of the ensemble.</li>\n<li>Up to the middle stage, I fixed the model to roberta-large and tried to improve the score. </li>\n<li>At the end, I applied the method to another models. I found that key parameters for this task are {learning_rate, N layers to re-initialize}, so I tuned those parameters for each models.</li></ul></li>\n<li>re-initialization<ul>\n<li>This paper (<a href=\"https://arxiv.org/pdf/2006.05987.pdf\" target=\"_blank\">https://arxiv.org/pdf/2006.05987.pdf</a>) shows that fine-tuning with reinitialization last N layers works well.</li>\n<li>Different models have different optimal N. Almost models set N=4~5, gpt2-models set N=6.</li></ul></li>\n<li>LSTM head<ul>\n<li>Input BERT's first and last hidden layer into LSTM layer worked well.</li>\n<li>I think first layer represent vocabulary difficulty and last layer represent sentence difficulty. Both are important for inference readbility.</li></ul></li>\n<li>Remove dropout. Improve 0.01~0.02 CV.</li>\n<li>gradient clipping. (0.2 or 0.5 works well for me, improve about 0.005 CV)</li>\n</ul>\n<h1>not worked for me</h1>\n<ul>\n<li>Input attention matrix to 2D-CNN(like ResNet18 or simple 2DCNN)<ul>\n<li>I thought this could represent the complexity of sentences with relative pronouns.</li></ul></li>\n<li>masked 5%~10% vocabulary.</li>\n<li>Minimize KLDiv loss to fit distribution.</li>\n<li>Scale target to 0~1 and minimize crossentropy loss</li>\n<li>\"base\" models excluding mpnet. I got 0.47x CV but Public LB: 0.48x ~ 0.49x.</li>\n<li>Stacking using LightGBM.</li>\n<li>another models.(result is below table. single CV is well but zero weight for ensemble)</li>\n<li>T5. Below notebook achieve 0.47 LB using T5, so I tried but failed.<br>\nI got only 0.49x(fold 0 only) with learning_rate=1.5e-4<br>\n<a href=\"https://www.kaggle.com/gilfernandes/commonlit-pytorch-t5-large-svm/comments\" target=\"_blank\">https://www.kaggle.com/gilfernandes/commonlit-pytorch-t5-large-svm/comments</a></li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV(5fold)</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>xlnet-large-cased</td>\n<td>0.471</td>\n<td>0.473</td>\n</tr>\n<tr>\n<td>bert-large-cased</td>\n<td>0.4828</td>\n<td>0.488</td>\n</tr>\n</tbody>\n</table>\n<h1>GPU</h1>\n<ul>\n<li>GeForce RTX3090 * 1.</li>\n</ul>\n<h1>Inference notebook</h1>\n<p><a href=\"https://www.kaggle.com/kurupical/191-192-202-228-251-253-268-288-278-final?scriptVersionId=69642056\" target=\"_blank\">https://www.kaggle.com/kurupical/191-192-202-228-251-253-268-288-278-final?scriptVersionId=69642056</a><br>\n(sorry for almost of dataset is private.)</p>\n<h1>repository</h1>\n<p><a href=\"https://github.com/kurupical/commonlit\" target=\"_blank\">https://github.com/kurupical/commonlit</a></p>",
  "comments": [
    {
      "id": 1418152,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2021-08-03T01:06:52.313000",
      "content": "<p>congrats on results <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> :D</p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 1427361,
      "author_name": "Mr_KnowNothing",
      "author_url": "",
      "post_date": "2021-08-03T08:22:05.823000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> , I also tried LSTM and other heads but it didn't work out for me . Thanks for sharing the code</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1430156,
          "author_name": "kurupical",
          "author_url": "",
          "post_date": "2021-08-03T11:21:29.253000",
          "content": "<p>Thanks! Normal LSTM didn't work, but input first/last hidden_layer to LSTM worked for me :D</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1419956,
      "author_name": "Rakesh Jarupula",
      "author_url": "",
      "post_date": "2021-08-03T02:52:21.310000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> 🎉</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1419081,
      "author_name": "HinePo",
      "author_url": "",
      "post_date": "2021-08-03T02:03:50.267000",
      "content": "<p>Very impressed by your ensemble. Congrats! Amazing result and solution!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1418156,
      "author_name": "Salim Khazem",
      "author_url": "",
      "post_date": "2021-08-03T01:07:10.403000",
      "content": "<p>Great job ! Thanks for sharing, did you plan to share your final notebook ? inference or train one ? </p>",
      "votes": 1,
      "replies": [
        {
          "id": 1418854,
          "author_name": "kurupical",
          "author_url": "",
          "post_date": "2021-08-03T01:52:09.733000",
          "content": "<p>Thank you! <br>\nI add the inference notebook and github repository in this post.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1426609,
          "author_name": "Salim Khazem",
          "author_url": "",
          "post_date": "2021-08-03T07:43:48.340000",
          "content": "<p>Thanks a lot for sharing, a real pleasure to read your approach </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1418075,
      "author_name": "Vincent Wang",
      "author_url": "",
      "post_date": "2021-08-03T01:02:10.230000",
      "content": "<p>Thank you for sharing! Actually our team in the end found your public dataset quite helpful and include some new models into the final submission, which helps a lot! Thank you!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1428516,
          "author_name": "kurupical",
          "author_url": "",
          "post_date": "2021-08-03T09:33:15.647000",
          "content": "<p>I didn't think my model weights would be useful for the other team :D<br>\nGood job! Congratulations on your medal.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1423423,
      "author_name": "Yiemon773",
      "author_url": "",
      "post_date": "2021-08-03T05:39:55.873000",
      "content": "<p>Hi, <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> . <br>\nThank you for sharing your solution.  </p>\n<pre><code>Remove dropout. Improve 0.01~0.02 CV.\ngradient clipping. (0.2 or 0.5 works well for me, improve about 0.005 CV)\n</code></pre>\n<p>Are removing dropout and gradient clipping effective for improving public / private scores ?  </p>",
      "votes": 2,
      "replies": [
        {
          "id": 1430145,
          "author_name": "kurupical",
          "author_url": "",
          "post_date": "2021-08-03T11:20:18.897000",
          "content": "<p>Hi! Both worked for public/private.</p>",
          "votes": -1,
          "replies": []
        },
        {
          "id": 1430723,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-08-03T11:54:21.133000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1430728,
          "author_name": "Yiemon773",
          "author_url": "",
          "post_date": "2021-08-03T11:54:21.133000",
          "content": "<p>I see, Thank you !  <br>\nI look forward to seeing your work in next competition again. </p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1422282,
      "author_name": "over watch",
      "author_url": "",
      "post_date": "2021-08-03T04:47:31.467000",
      "content": "<p>Congratulations to your solo silver metal!</p>\n<p>Just a quick question- did you train all of your models in the same fold and optimize the ensemble within each fold or did you optimize based on all of the out of fold predictions? </p>\n<p>Thanks a lot!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1430080,
          "author_name": "kurupical",
          "author_url": "",
          "post_date": "2021-08-03T11:16:49.910000",
          "content": "<p>Thanks!<br>\nAll model is same fold and optimize all of the out of fold predictions!<br>\nI tried to optimize each fold, but didn't work for me.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1418586,
      "author_name": "torch",
      "author_url": "",
      "post_date": "2021-08-03T01:35:26.317000",
      "content": "<p>Many congratulations <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> on your solo silver and thanks for sharing. Quite a clean and well thought solution. </p>\n<p>Here are my queries,</p>\n<ul>\n<li>Which strategy did you use for combining the text and tabular features?</li>\n<li>Can you highlight more on the Nelder Mead Ensembling technique.</li>\n</ul>\n<p>Thanks. </p>",
      "votes": 2,
      "replies": [
        {
          "id": 1428665,
          "author_name": "kurupical",
          "author_url": "",
          "post_date": "2021-08-03T09:41:46.693000",
          "content": "<p>Hi, <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a>! (sorry for wrong mention.) I think your contribution to this competition is so great!</p>\n<ul>\n<li>Which strategy did you use for combining the text and tabular features?</li>\n</ul>\n<p>The code is below. and concatenate other features like LSTM head.<br>\nnn.Linear's input is normalized by train dataset.</p>\n<pre><code>self.linear_feature = nn.Sequential(\n    nn.Linear(17, 128),\n    nn.GELU(),\n)\n</code></pre>\n<ul>\n<li>Can you highlight more on the Nelder Mead Ensembling technique.</li>\n</ul>\n<p>The code is below. </p>\n<pre><code>from scipy.optimize import minimize\n\ndef rmse_func(weights):\n    # predictions.shape = (n_data, n_models_for_ensemble)\n    pred = (predictions * weights).sum(axis=1)\n    rmse = np.sqrt(1 / len(pred) * ((target - pred)**2).sum())\n    return rmse\n\nn_models = predictions.shape[1] \ninitial_weights = np.ones(n_models) / len(n_models)\nres = minimize(rmse_func, initial_weights, method='Nelder-Mead')\n\nweights = res[\"x\"]\nrmse = res[\"fun\"]\n</code></pre>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 1443756,
          "author_name": "torch",
          "author_url": "",
          "post_date": "2021-08-04T04:05:40.230000",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> for the explanations and code. Congratulations once more. 😊</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1418466,
      "author_name": "Julián Peller (dataista0)",
      "author_url": "",
      "post_date": "2021-08-03T01:27:09.360000",
      "content": "<p>Congrats on your great results <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a>!</p>\n<p>The great and beloved - to whom all of us are in debt - <strong>torch</strong> is actually <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> in case you want to correct your mention. I want to use the occasion to thank him/her too.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1418615,
          "author_name": "torch",
          "author_url": "",
          "post_date": "2021-08-03T01:37:05.290000",
          "content": "<p>Thank you so much <a href=\"https://www.kaggle.com/julian3833\" target=\"_blank\">@julian3833</a> 😊</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1418629,
          "author_name": "kurupical",
          "author_url": "",
          "post_date": "2021-08-03T01:38:02.617000",
          "content": "<p>Thank you for pointing that out. I fix mention! <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a>'s work is very wonderful :D I was helped by him/her a lot!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1426055,
      "author_name": "Vu",
      "author_url": "",
      "post_date": "2021-08-03T07:16:04.713000",
      "content": "<p>Thanks for sharing your solution and <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> too for great notebooks</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1420983,
      "author_name": "Clear n' Simple",
      "author_url": "",
      "post_date": "2021-08-03T03:43:22.540000",
      "content": "<p>Congratulations!</p>\n<p>That's a lot of modeling…</p>\n<p>When you chose parameters, did you run the training fully each time?<br>\nOr did you get training stable enough that one fold/less epochs was enough to tell?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1430077,
          "author_name": "kurupical",
          "author_url": "",
          "post_date": "2021-08-03T11:16:27.887000",
          "content": "<p>Thanks!<br>\nIf I want to know if the model improve ~0.01, train 1 fold, else train 5 fold.<br>\nAnd to increase the efficiency of the experiment, I defined a minimum line for fold1 for each model. If the minimum line was not met, the experiment was considered a failure.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1420015,
      "author_name": "Khuong T G Hieu",
      "author_url": "",
      "post_date": "2021-08-03T02:55:09.093000",
      "content": "<p>congrats on 22nd place <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> 💪, just one question. How are you able to make Funnel-large converge, because I can't even train its CV lower than 0.5 😄?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1428713,
          "author_name": "kurupical",
          "author_url": "",
          "post_date": "2021-08-03T09:58:45.907000",
          "content": "<p>Thanks! I'm not doing anything special. <br>\nThe main hyperparameters are as follows: </p>\n<table>\n<thead>\n<tr>\n<th>nlp_model_name</th>\n<th>funnel-large-base</th>\n<th>funnel-large</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>dropout</td>\n<td>0</td>\n<td>0</td>\n</tr>\n<tr>\n<td>batch_size</td>\n<td>12</td>\n<td>12</td>\n</tr>\n<tr>\n<td>lr_bert</td>\n<td>2E-05</td>\n<td>2E-05</td>\n</tr>\n<tr>\n<td>lr_fc</td>\n<td>5E-05</td>\n<td>5E-05</td>\n</tr>\n<tr>\n<td>warmup_ratio</td>\n<td>0.05</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>epochs</td>\n<td>6</td>\n<td>6</td>\n</tr>\n<tr>\n<td>activation</td>\n<td>GELU</td>\n<td>GELU</td>\n</tr>\n<tr>\n<td>optimizer</td>\n<td>AdamW</td>\n<td>AdamW</td>\n</tr>\n<tr>\n<td>weight_decay</td>\n<td>0.1</td>\n<td>0.1</td>\n</tr>\n<tr>\n<td>rnn_module</td>\n<td>LSTM</td>\n<td>LSTM</td>\n</tr>\n<tr>\n<td>rnn_module_num</td>\n<td>0</td>\n<td>1</td>\n</tr>\n<tr>\n<td>rnn_hidden_indice</td>\n<td>(-1, 0)</td>\n<td>(-1, 0)</td>\n</tr>\n<tr>\n<td>linear_vocab_enable</td>\n<td>False</td>\n<td>True</td>\n</tr>\n<tr>\n<td>multi_dropout_ratio</td>\n<td>0.3</td>\n<td>0.3</td>\n</tr>\n<tr>\n<td>multi_dropout_num</td>\n<td>10</td>\n<td>10</td>\n</tr>\n<tr>\n<td>max_length</td>\n<td>256</td>\n<td>256</td>\n</tr>\n<tr>\n<td>hidden_stack_enable</td>\n<td>True</td>\n<td>True</td>\n</tr>\n<tr>\n<td>reinit_layers</td>\n<td>4</td>\n<td>4</td>\n</tr>\n<tr>\n<td>gradient_clipping</td>\n<td>0.2</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>feature_enable</td>\n<td>False</td>\n<td>True</td>\n</tr>\n<tr>\n<td>stochastic_weight_avg</td>\n<td>False</td>\n<td>False</td>\n</tr>\n<tr>\n<td>val_check_interval</td>\n<td>0.05</td>\n<td>0.05</td>\n</tr>\n</tbody>\n</table>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 27,
  "messages": [
    {
      "id": 1417882,
      "postDate": "2021-08-03T00:49:10.180Z",
      "content": "<h1>Acknowledgement</h1>\n<p>First of all, thanks to competition organizers for hosting this competition.<br><br>\nAnd I would like to thank you <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> for share a lot of great notebook and discussion. I couldn't have achieved this ranking without you, and I learned a lot method for fine-tune BERT models from you!<br></p>\n<h1>summary</h1>\n<p>I have three options(below), and in the end I chose the top two for final submission.</p>\n<ul>\n<li>LB best (9models, CV: 0.4449, LB: 0.444, Private LB: 0.452) </li>\n<li>CV best exclude base-model (9models, CV: 0.4430, LB: 0.447, Private LB: 0.452)</li>\n<li>CV best (12models, CV: 0.4420, LB: 0.453, Private LB: 0.452)</li>\n</ul>\n<p>The detail of LB best is below table:<br></p>\n<h2>LB-Best</h2>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV(5fold)</th>\n<th>weight</th>\n<th>Public</th>\n<th>Private</th>\n<th>other</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>luke-large</td>\n<td>0.4795</td>\n<td>0.0877</td>\n<td>0.467</td>\n<td>?</td>\n<td></td>\n</tr>\n<tr>\n<td>roberta-large</td>\n<td>0.4693</td>\n<td>0.1035</td>\n<td>0.464</td>\n<td>0.469</td>\n<td></td>\n</tr>\n<tr>\n<td>gpt2-medium</td>\n<td>0.481</td>\n<td>0.158</td>\n<td>0.469</td>\n<td>0.482</td>\n<td></td>\n</tr>\n<tr>\n<td>funnel-large-base</td>\n<td>0.4706</td>\n<td>0.1455</td>\n<td>0.461</td>\n<td>0.47</td>\n<td></td>\n</tr>\n<tr>\n<td>funnel-large</td>\n<td>0.4705</td>\n<td>0.0267</td>\n<td>0.46</td>\n<td>0.467</td>\n<td></td>\n</tr>\n<tr>\n<td>ernie-large</td>\n<td>0.4712</td>\n<td>0.093</td>\n<td>0.463</td>\n<td>0.477</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td>0.475</td>\n<td>0.1693</td>\n<td>0.464</td>\n<td>0.467</td>\n<td></td>\n</tr>\n<tr>\n<td>mpnet-base</td>\n<td>0.4782</td>\n<td>0.1415</td>\n<td>0.466</td>\n<td>0.474</td>\n<td></td>\n</tr>\n<tr>\n<td>gpt2-large</td>\n<td>0.4744</td>\n<td>0.1121</td>\n<td>0.468</td>\n<td>0.48</td>\n<td>batch_size=4</td>\n</tr>\n<tr>\n<td>coef</td>\n<td>-</td>\n<td>0.0235</td>\n<td>-</td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h2>CV-Best</h2>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV(5fold)</th>\n<th>weight</th>\n<th>Public</th>\n<th>Private</th>\n<th>other</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>electra-large</td>\n<td>0.4743</td>\n<td>0.0829</td>\n<td>0.474</td>\n<td>0.479</td>\n<td></td>\n</tr>\n<tr>\n<td>funnel-large</td>\n<td>0.4752</td>\n<td>0.1215</td>\n<td>0.467</td>\n<td>0.474</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-large(1)</td>\n<td>0.4738</td>\n<td>0.1516</td>\n<td>0.47</td>\n<td>0.463</td>\n<td></td>\n</tr>\n<tr>\n<td>gpt2-medium</td>\n<td>0.4775</td>\n<td>0.1219</td>\n<td>0.471</td>\n<td>0.483</td>\n<td></td>\n</tr>\n<tr>\n<td>luke-large</td>\n<td>0.474</td>\n<td>0.0937</td>\n<td>0.468</td>\n<td>0.465</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-large(2)</td>\n<td>0.475</td>\n<td>0.1179</td>\n<td>0.464</td>\n<td>0.467</td>\n<td></td>\n</tr>\n<tr>\n<td>mpnet-base</td>\n<td>0.4782</td>\n<td>0.0999</td>\n<td>0.466</td>\n<td>0.474</td>\n<td></td>\n</tr>\n<tr>\n<td>ernie-large</td>\n<td>0.4681</td>\n<td>0.0792</td>\n<td>0.47</td>\n<td>0.473</td>\n<td></td>\n</tr>\n<tr>\n<td>gpt2-large</td>\n<td>0.4744</td>\n<td>0.1648</td>\n<td>?</td>\n<td>?</td>\n<td>batch_size=4</td>\n</tr>\n<tr>\n<td>coef</td>\n<td>-</td>\n<td>0.009</td>\n<td>-</td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h1>configuration for almost all models</h1>\n<ul>\n<li>epochs = 4</li>\n<li>optimizer: AdamW</li>\n<li>scheduler: linear_schedule_with_warmup(warmup: 5%)</li>\n<li>lr_bert: 3e-5</li>\n<li>batch_size: 12</li>\n<li>gradient clipping: 0.2~0.5</li>\n<li>reinitialize layers: last 2~6 layers  </li>\n<li>ensemble: Nelder-Mead</li>\n<li>custom head(finally concat all)<ul>\n<li>averaging last 4 hidden layer</li>\n<li>LSTM head</li>\n<li>vocabulary dense</li></ul></li>\n</ul>\n<pre><code>  hidden_states: (batch_size, vocab_size, bert_hidden_size)\n  linear_vocab = nn.Sequential(\n      nn.Linear(bert_hidden_size, 128),\n      nn.GELU(),\n      nn.Linear(128, 64),\n      nn.GELU()\n  )\n  linear_final = nn.Linear(vocab_size * 64, 128)\n  out = linear_vocab(hidden_states).view(len(input_ids), -1)) # final shape: (batch_size, vocab_size * 64)\n  out = linear_final(out) # out shape: (batch_size, 128)\n</code></pre>\n<ul>\n<li>17 hand-made features <ul>\n<li>sentence count</li>\n<li>average character count in documents</li></ul></li>\n</ul>\n<h1>worked for me</h1>\n<ul>\n<li>model ensemble: I thought diversity is the most important thing in this competition.<ul>\n<li>At the beginning of the competition, I tested the effectiveness of the ensemble.</li>\n<li>Up to the middle stage, I fixed the model to roberta-large and tried to improve the score. </li>\n<li>At the end, I applied the method to another models. I found that key parameters for this task are {learning_rate, N layers to re-initialize}, so I tuned those parameters for each models.</li></ul></li>\n<li>re-initialization<ul>\n<li>This paper (<a href=\"https://arxiv.org/pdf/2006.05987.pdf\" target=\"_blank\">https://arxiv.org/pdf/2006.05987.pdf</a>) shows that fine-tuning with reinitialization last N layers works well.</li>\n<li>Different models have different optimal N. Almost models set N=4~5, gpt2-models set N=6.</li></ul></li>\n<li>LSTM head<ul>\n<li>Input BERT's first and last hidden layer into LSTM layer worked well.</li>\n<li>I think first layer represent vocabulary difficulty and last layer represent sentence difficulty. Both are important for inference readbility.</li></ul></li>\n<li>Remove dropout. Improve 0.01~0.02 CV.</li>\n<li>gradient clipping. (0.2 or 0.5 works well for me, improve about 0.005 CV)</li>\n</ul>\n<h1>not worked for me</h1>\n<ul>\n<li>Input attention matrix to 2D-CNN(like ResNet18 or simple 2DCNN)<ul>\n<li>I thought this could represent the complexity of sentences with relative pronouns.</li></ul></li>\n<li>masked 5%~10% vocabulary.</li>\n<li>Minimize KLDiv loss to fit distribution.</li>\n<li>Scale target to 0~1 and minimize crossentropy loss</li>\n<li>\"base\" models excluding mpnet. I got 0.47x CV but Public LB: 0.48x ~ 0.49x.</li>\n<li>Stacking using LightGBM.</li>\n<li>another models.(result is below table. single CV is well but zero weight for ensemble)</li>\n<li>T5. Below notebook achieve 0.47 LB using T5, so I tried but failed.<br>\nI got only 0.49x(fold 0 only) with learning_rate=1.5e-4<br>\n<a href=\"https://www.kaggle.com/gilfernandes/commonlit-pytorch-t5-large-svm/comments\" target=\"_blank\">https://www.kaggle.com/gilfernandes/commonlit-pytorch-t5-large-svm/comments</a></li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV(5fold)</th>\n<th>Public</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>xlnet-large-cased</td>\n<td>0.471</td>\n<td>0.473</td>\n</tr>\n<tr>\n<td>bert-large-cased</td>\n<td>0.4828</td>\n<td>0.488</td>\n</tr>\n</tbody>\n</table>\n<h1>GPU</h1>\n<ul>\n<li>GeForce RTX3090 * 1.</li>\n</ul>\n<h1>Inference notebook</h1>\n<p><a href=\"https://www.kaggle.com/kurupical/191-192-202-228-251-253-268-288-278-final?scriptVersionId=69642056\" target=\"_blank\">https://www.kaggle.com/kurupical/191-192-202-228-251-253-268-288-278-final?scriptVersionId=69642056</a><br>\n(sorry for almost of dataset is private.)</p>\n<h1>repository</h1>\n<p><a href=\"https://github.com/kurupical/commonlit\" target=\"_blank\">https://github.com/kurupical/commonlit</a></p>",
      "rawMarkdown": "# Acknowledgement\nFirst of all, thanks to competition organizers for hosting this competition.<br>\nAnd I would like to thank you @rhtsingh for share a lot of great notebook and discussion. I couldn't have achieved this ranking without you, and I learned a lot method for fine-tune BERT models from you!<br>\n\n# summary\nI have three options(below), and in the end I chose the top two for final submission.\n* LB best (9models, CV: 0.4449, LB: 0.444, Private LB: 0.452) \n* CV best exclude base-model (9models, CV: 0.4430, LB: 0.447, Private LB: 0.452)\n* CV best (12models, CV: 0.4420, LB: 0.453, Private LB: 0.452)\n\nThe detail of LB best is below table:<br>\n\n## LB-Best\n\n|                   | CV(5fold) | weight | Public | Private | other        |\n|-------------------|-----------|--------|--------|---------|--------------|\n| luke-large        | 0.4795    | 0.0877 | 0.467  | ?       |              |\n| roberta-large     | 0.4693    | 0.1035 | 0.464  | 0.469   |              |\n| gpt2-medium       | 0.481     | 0.158  | 0.469  | 0.482   |              |\n| funnel-large-base | 0.4706    | 0.1455 | 0.461  | 0.47    |              |\n| funnel-large      | 0.4705    | 0.0267 | 0.46   | 0.467   |              |\n| ernie-large       | 0.4712    | 0.093  | 0.463  | 0.477   |              |\n| deberta-large     | 0.475     | 0.1693 | 0.464  | 0.467   |              |\n| mpnet-base        | 0.4782    | 0.1415 | 0.466  | 0.474   |              |\n| gpt2-large        | 0.4744    | 0.1121 | 0.468  | 0.48    | batch_size=4 |\n| coef              | -         | 0.0235 | -      |         |              |\n\n## CV-Best\n\n|                  | CV(5fold) | weight | Public | Private | other        |\n|------------------|-----------|--------|--------|---------|--------------|\n| electra-large    | 0.4743    | 0.0829 | 0.474  | 0.479   |              |\n| funnel-large     | 0.4752    | 0.1215 | 0.467  | 0.474   |              |\n| deberta-large(1) | 0.4738    | 0.1516 | 0.47   | 0.463   |              |\n| gpt2-medium      | 0.4775    | 0.1219 | 0.471  | 0.483   |              |\n| luke-large       | 0.474     | 0.0937 | 0.468  | 0.465   |              |\n| deberta-large(2) | 0.475     | 0.1179 | 0.464  | 0.467   |              |\n| mpnet-base       | 0.4782    | 0.0999 | 0.466  | 0.474   |              |\n| ernie-large      | 0.4681    | 0.0792 | 0.47   | 0.473   |              |\n| gpt2-large       | 0.4744    | 0.1648 | ?      | ?       | batch_size=4 |\n| coef             | -         | 0.009  | -      |         |              |\n\n\n# configuration for almost all models\n* epochs = 4\n* optimizer: AdamW\n* scheduler: linear_schedule_with_warmup(warmup: 5%)\n* lr_bert: 3e-5\n* batch_size: 12\n* gradient clipping: 0.2~0.5\n* reinitialize layers: last 2~6 layers  \n* ensemble: Nelder-Mead\n* custom head(finally concat all)\n  * averaging last 4 hidden layer\n  * LSTM head\n  * vocabulary dense\n  ```\n  hidden_states: (batch_size, vocab_size, bert_hidden_size)\n  linear_vocab = nn.Sequential(\n      nn.Linear(bert_hidden_size, 128),\n      nn.GELU(),\n      nn.Linear(128, 64),\n      nn.GELU()\n  )\n  linear_final = nn.Linear(vocab_size * 64, 128)\n  out = linear_vocab(hidden_states).view(len(input_ids), -1)) # final shape: (batch_size, vocab_size * 64)\n  out = linear_final(out) # out shape: (batch_size, 128)\n  ```\n* 17 hand-made features \n  * sentence count\n  * average character count in documents\n\n  \n# worked for me\n* model ensemble: I thought diversity is the most important thing in this competition.\n  * At the beginning of the competition, I tested the effectiveness of the ensemble.\n  * Up to the middle stage, I fixed the model to roberta-large and tried to improve the score. \n  * At the end, I applied the method to another models. I found that key parameters for this task are {learning_rate, N layers to re-initialize}, so I tuned those parameters for each models.\n* re-initialization\n  * This paper (https://arxiv.org/pdf/2006.05987.pdf) shows that fine-tuning with reinitialization last N layers works well.\n  * Different models have different optimal N. Almost models set N=4~5, gpt2-models set N=6.\n* LSTM head\n  * Input BERT's first and last hidden layer into LSTM layer worked well.\n  * I think first layer represent vocabulary difficulty and last layer represent sentence difficulty. Both are important for inference readbility.\n* Remove dropout. Improve 0.01~0.02 CV.\n* gradient clipping. (0.2 or 0.5 works well for me, improve about 0.005 CV)\n\n# not worked for me\n* Input attention matrix to 2D-CNN(like ResNet18 or simple 2DCNN)\n  * I thought this could represent the complexity of sentences with relative pronouns.\n* masked 5%~10% vocabulary.\n* Minimize KLDiv loss to fit distribution.\n* Scale target to 0~1 and minimize crossentropy loss\n* \"base\" models excluding mpnet. I got 0.47x CV but Public LB: 0.48x ~ 0.49x.\n* Stacking using LightGBM.\n* another models.(result is below table. single CV is well but zero weight for ensemble)\n* T5. Below notebook achieve 0.47 LB using T5, so I tried but failed.\n  I got only 0.49x(fold 0 only) with learning_rate=1.5e-4\n  https://www.kaggle.com/gilfernandes/commonlit-pytorch-t5-large-svm/comments\n  \n  \n| model             | CV(5fold) | Public |\n|-------------------|-----------|--------|\n| xlnet-large-cased | 0.471     | 0.473  |\n| bert-large-cased  | 0.4828    | 0.488  |\n\n\n# GPU\n* GeForce RTX3090 * 1.\n\n# Inference notebook\nhttps://www.kaggle.com/kurupical/191-192-202-228-251-253-268-288-278-final?scriptVersionId=69642056\n(sorry for almost of dataset is private.)\n\n# repository\nhttps://github.com/kurupical/commonlit",
      "votes": 87
    },
    {
      "id": 1418152,
      "postDate": "2021-08-03T01:06:52.313Z",
      "content": "<p>congrats on results <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> :D</p>",
      "rawMarkdown": "congrats on results @kurupical :D",
      "votes": 4
    },
    {
      "id": 1427361,
      "postDate": "2021-08-03T08:22:05.823Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> , I also tried LSTM and other heads but it didn't work out for me . Thanks for sharing the code</p>",
      "rawMarkdown": "Congratulations @kurupical , I also tried LSTM and other heads but it didn't work out for me . Thanks for sharing the code",
      "votes": 1,
      "replies": [
        {
          "id": 1430156,
          "postDate": "2021-08-03T11:21:29.253Z",
          "content": "<p>Thanks! Normal LSTM didn't work, but input first/last hidden_layer to LSTM worked for me :D</p>",
          "rawMarkdown": "Thanks! Normal LSTM didn't work, but input first/last hidden_layer to LSTM worked for me :D",
          "votes": 1
        }
      ]
    },
    {
      "id": 1419956,
      "postDate": "2021-08-03T02:52:21.310Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> 🎉</p>",
      "rawMarkdown": "Congratulations @kurupical 🎉",
      "votes": 1
    },
    {
      "id": 1419081,
      "postDate": "2021-08-03T02:03:50.267Z",
      "content": "<p>Very impressed by your ensemble. Congrats! Amazing result and solution!</p>",
      "rawMarkdown": "Very impressed by your ensemble. Congrats! Amazing result and solution!",
      "votes": 1
    },
    {
      "id": 1418156,
      "postDate": "2021-08-03T01:07:10.403Z",
      "content": "<p>Great job ! Thanks for sharing, did you plan to share your final notebook ? inference or train one ? </p>",
      "rawMarkdown": "Great job ! Thanks for sharing, did you plan to share your final notebook ? inference or train one ? ",
      "votes": 1,
      "replies": [
        {
          "id": 1418854,
          "postDate": "2021-08-03T01:52:09.733Z",
          "content": "<p>Thank you! <br>\nI add the inference notebook and github repository in this post.</p>",
          "rawMarkdown": "Thank you! \nI add the inference notebook and github repository in this post."
        },
        {
          "id": 1426609,
          "postDate": "2021-08-03T07:43:48.340Z",
          "content": "<p>Thanks a lot for sharing, a real pleasure to read your approach </p>",
          "rawMarkdown": "Thanks a lot for sharing, a real pleasure to read your approach "
        }
      ]
    },
    {
      "id": 1418075,
      "postDate": "2021-08-03T01:02:10.230Z",
      "content": "<p>Thank you for sharing! Actually our team in the end found your public dataset quite helpful and include some new models into the final submission, which helps a lot! Thank you!</p>",
      "rawMarkdown": "Thank you for sharing! Actually our team in the end found your public dataset quite helpful and include some new models into the final submission, which helps a lot! Thank you!",
      "votes": 1,
      "replies": [
        {
          "id": 1428516,
          "postDate": "2021-08-03T09:33:15.647Z",
          "content": "<p>I didn't think my model weights would be useful for the other team :D<br>\nGood job! Congratulations on your medal.</p>",
          "rawMarkdown": "I didn't think my model weights would be useful for the other team :D\nGood job! Congratulations on your medal."
        }
      ]
    },
    {
      "id": 1423423,
      "postDate": "2021-08-03T05:39:55.873Z",
      "content": "<p>Hi, <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> . <br>\nThank you for sharing your solution.  </p>\n<pre><code>Remove dropout. Improve 0.01~0.02 CV.\ngradient clipping. (0.2 or 0.5 works well for me, improve about 0.005 CV)\n</code></pre>\n<p>Are removing dropout and gradient clipping effective for improving public / private scores ?  </p>",
      "rawMarkdown": "Hi, @kurupical . \nThank you for sharing your solution.  \n\n```\nRemove dropout. Improve 0.01~0.02 CV.\ngradient clipping. (0.2 or 0.5 works well for me, improve about 0.005 CV)\n```\n\nAre removing dropout and gradient clipping effective for improving public / private scores ?  \n \n",
      "votes": 2,
      "replies": [
        {
          "id": 1430145,
          "postDate": "2021-08-03T11:20:18.897Z",
          "content": "<p>Hi! Both worked for public/private.</p>",
          "rawMarkdown": "Hi! Both worked for public/private.",
          "votes": -1
        },
        {
          "id": 1430723,
          "postDate": "2021-08-03T11:54:21.133Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 1430728,
          "postDate": "2021-08-03T11:54:21.133Z",
          "content": "<p>I see, Thank you !  <br>\nI look forward to seeing your work in next competition again. </p>",
          "rawMarkdown": "I see, Thank you !  \nI look forward to seeing your work in next competition again. ",
          "votes": 2
        }
      ]
    },
    {
      "id": 1422282,
      "postDate": "2021-08-03T04:47:31.467Z",
      "content": "<p>Congratulations to your solo silver metal!</p>\n<p>Just a quick question- did you train all of your models in the same fold and optimize the ensemble within each fold or did you optimize based on all of the out of fold predictions? </p>\n<p>Thanks a lot!</p>",
      "rawMarkdown": "Congratulations to your solo silver metal!\n\nJust a quick question- did you train all of your models in the same fold and optimize the ensemble within each fold or did you optimize based on all of the out of fold predictions? \n\nThanks a lot!",
      "votes": 2,
      "replies": [
        {
          "id": 1430080,
          "postDate": "2021-08-03T11:16:49.910Z",
          "content": "<p>Thanks!<br>\nAll model is same fold and optimize all of the out of fold predictions!<br>\nI tried to optimize each fold, but didn't work for me.</p>",
          "rawMarkdown": "Thanks!\nAll model is same fold and optimize all of the out of fold predictions!\nI tried to optimize each fold, but didn't work for me."
        }
      ]
    },
    {
      "id": 1418586,
      "postDate": "2021-08-03T01:35:26.317Z",
      "content": "<p>Many congratulations <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> on your solo silver and thanks for sharing. Quite a clean and well thought solution. </p>\n<p>Here are my queries,</p>\n<ul>\n<li>Which strategy did you use for combining the text and tabular features?</li>\n<li>Can you highlight more on the Nelder Mead Ensembling technique.</li>\n</ul>\n<p>Thanks. </p>",
      "rawMarkdown": "Many congratulations @kurupical on your solo silver and thanks for sharing. Quite a clean and well thought solution. \n\nHere are my queries,\n - Which strategy did you use for combining the text and tabular features?\n - Can you highlight more on the Nelder Mead Ensembling technique.\n\nThanks. ",
      "votes": 2,
      "replies": [
        {
          "id": 1428665,
          "postDate": "2021-08-03T09:41:46.693Z",
          "content": "<p>Hi, <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a>! (sorry for wrong mention.) I think your contribution to this competition is so great!</p>\n<ul>\n<li>Which strategy did you use for combining the text and tabular features?</li>\n</ul>\n<p>The code is below. and concatenate other features like LSTM head.<br>\nnn.Linear's input is normalized by train dataset.</p>\n<pre><code>self.linear_feature = nn.Sequential(\n    nn.Linear(17, 128),\n    nn.GELU(),\n)\n</code></pre>\n<ul>\n<li>Can you highlight more on the Nelder Mead Ensembling technique.</li>\n</ul>\n<p>The code is below. </p>\n<pre><code>from scipy.optimize import minimize\n\ndef rmse_func(weights):\n    # predictions.shape = (n_data, n_models_for_ensemble)\n    pred = (predictions * weights).sum(axis=1)\n    rmse = np.sqrt(1 / len(pred) * ((target - pred)**2).sum())\n    return rmse\n\nn_models = predictions.shape[1] \ninitial_weights = np.ones(n_models) / len(n_models)\nres = minimize(rmse_func, initial_weights, method='Nelder-Mead')\n\nweights = res[\"x\"]\nrmse = res[\"fun\"]\n</code></pre>",
          "rawMarkdown": "Hi, @rhtsingh! (sorry for wrong mention.) I think your contribution to this competition is so great!\n\n* Which strategy did you use for combining the text and tabular features?\n\nThe code is below. and concatenate other features like LSTM head.\nnn.Linear's input is normalized by train dataset.\n```\nself.linear_feature = nn.Sequential(\n    nn.Linear(17, 128),\n    nn.GELU(),\n)\n```\n\n* Can you highlight more on the Nelder Mead Ensembling technique.\n\nThe code is below. \n\n```python\nfrom scipy.optimize import minimize\n\ndef rmse_func(weights):\n    # predictions.shape = (n_data, n_models_for_ensemble)\n    pred = (predictions * weights).sum(axis=1)\n    rmse = np.sqrt(1 / len(pred) * ((target - pred)**2).sum())\n    return rmse\n\nn_models = predictions.shape[1] \ninitial_weights = np.ones(n_models) / len(n_models)\nres = minimize(rmse_func, initial_weights, method='Nelder-Mead')\n\nweights = res[\"x\"]\nrmse = res[\"fun\"]\n```",
          "votes": 4
        },
        {
          "id": 1443756,
          "postDate": "2021-08-04T04:05:40.230Z",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> for the explanations and code. Congratulations once more. 😊</p>",
          "rawMarkdown": "Thanks @kurupical for the explanations and code. Congratulations once more. 😊",
          "votes": 2
        }
      ]
    },
    {
      "id": 1418466,
      "postDate": "2021-08-03T01:27:09.360Z",
      "content": "<p>Congrats on your great results <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a>!</p>\n<p>The great and beloved - to whom all of us are in debt - <strong>torch</strong> is actually <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> in case you want to correct your mention. I want to use the occasion to thank him/her too.</p>",
      "rawMarkdown": "Congrats on your great results @kurupical!\n\nThe great and beloved - to whom all of us are in debt - **torch** is actually @rhtsingh in case you want to correct your mention. I want to use the occasion to thank him/her too.",
      "votes": 2,
      "replies": [
        {
          "id": 1418615,
          "postDate": "2021-08-03T01:37:05.290Z",
          "content": "<p>Thank you so much <a href=\"https://www.kaggle.com/julian3833\" target=\"_blank\">@julian3833</a> 😊</p>",
          "rawMarkdown": "Thank you so much @julian3833 😊",
          "votes": 2
        },
        {
          "id": 1418629,
          "postDate": "2021-08-03T01:38:02.617Z",
          "content": "<p>Thank you for pointing that out. I fix mention! <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a>'s work is very wonderful :D I was helped by him/her a lot!</p>",
          "rawMarkdown": "Thank you for pointing that out. I fix mention! @rhtsingh's work is very wonderful :D I was helped by him/her a lot!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1426055,
      "postDate": "2021-08-03T07:16:04.713Z",
      "content": "<p>Thanks for sharing your solution and <a href=\"https://www.kaggle.com/rhtsingh\" target=\"_blank\">@rhtsingh</a> too for great notebooks</p>",
      "rawMarkdown": "Thanks for sharing your solution and @rhtsingh too for great notebooks"
    },
    {
      "id": 1420983,
      "postDate": "2021-08-03T03:43:22.540Z",
      "content": "<p>Congratulations!</p>\n<p>That's a lot of modeling…</p>\n<p>When you chose parameters, did you run the training fully each time?<br>\nOr did you get training stable enough that one fold/less epochs was enough to tell?</p>",
      "rawMarkdown": "Congratulations!\n\nThat's a lot of modeling...\n\nWhen you chose parameters, did you run the training fully each time?\nOr did you get training stable enough that one fold/less epochs was enough to tell?",
      "replies": [
        {
          "id": 1430077,
          "postDate": "2021-08-03T11:16:27.887Z",
          "content": "<p>Thanks!<br>\nIf I want to know if the model improve ~0.01, train 1 fold, else train 5 fold.<br>\nAnd to increase the efficiency of the experiment, I defined a minimum line for fold1 for each model. If the minimum line was not met, the experiment was considered a failure.</p>",
          "rawMarkdown": "Thanks!\nIf I want to know if the model improve ~0.01, train 1 fold, else train 5 fold.\nAnd to increase the efficiency of the experiment, I defined a minimum line for fold1 for each model. If the minimum line was not met, the experiment was considered a failure."
        }
      ]
    },
    {
      "id": 1420015,
      "postDate": "2021-08-03T02:55:09.093Z",
      "content": "<p>congrats on 22nd place <a href=\"https://www.kaggle.com/kurupical\" target=\"_blank\">@kurupical</a> 💪, just one question. How are you able to make Funnel-large converge, because I can't even train its CV lower than 0.5 😄?</p>",
      "rawMarkdown": "congrats on 22nd place @kurupical 💪, just one question. How are you able to make Funnel-large converge, because I can't even train its CV lower than 0.5 😄?",
      "replies": [
        {
          "id": 1428713,
          "postDate": "2021-08-03T09:58:45.907Z",
          "content": "<p>Thanks! I'm not doing anything special. <br>\nThe main hyperparameters are as follows: </p>\n<table>\n<thead>\n<tr>\n<th>nlp_model_name</th>\n<th>funnel-large-base</th>\n<th>funnel-large</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>dropout</td>\n<td>0</td>\n<td>0</td>\n</tr>\n<tr>\n<td>batch_size</td>\n<td>12</td>\n<td>12</td>\n</tr>\n<tr>\n<td>lr_bert</td>\n<td>2E-05</td>\n<td>2E-05</td>\n</tr>\n<tr>\n<td>lr_fc</td>\n<td>5E-05</td>\n<td>5E-05</td>\n</tr>\n<tr>\n<td>warmup_ratio</td>\n<td>0.05</td>\n<td>0.05</td>\n</tr>\n<tr>\n<td>epochs</td>\n<td>6</td>\n<td>6</td>\n</tr>\n<tr>\n<td>activation</td>\n<td>GELU</td>\n<td>GELU</td>\n</tr>\n<tr>\n<td>optimizer</td>\n<td>AdamW</td>\n<td>AdamW</td>\n</tr>\n<tr>\n<td>weight_decay</td>\n<td>0.1</td>\n<td>0.1</td>\n</tr>\n<tr>\n<td>rnn_module</td>\n<td>LSTM</td>\n<td>LSTM</td>\n</tr>\n<tr>\n<td>rnn_module_num</td>\n<td>0</td>\n<td>1</td>\n</tr>\n<tr>\n<td>rnn_hidden_indice</td>\n<td>(-1, 0)</td>\n<td>(-1, 0)</td>\n</tr>\n<tr>\n<td>linear_vocab_enable</td>\n<td>False</td>\n<td>True</td>\n</tr>\n<tr>\n<td>multi_dropout_ratio</td>\n<td>0.3</td>\n<td>0.3</td>\n</tr>\n<tr>\n<td>multi_dropout_num</td>\n<td>10</td>\n<td>10</td>\n</tr>\n<tr>\n<td>max_length</td>\n<td>256</td>\n<td>256</td>\n</tr>\n<tr>\n<td>hidden_stack_enable</td>\n<td>True</td>\n<td>True</td>\n</tr>\n<tr>\n<td>reinit_layers</td>\n<td>4</td>\n<td>4</td>\n</tr>\n<tr>\n<td>gradient_clipping</td>\n<td>0.2</td>\n<td>0.2</td>\n</tr>\n<tr>\n<td>feature_enable</td>\n<td>False</td>\n<td>True</td>\n</tr>\n<tr>\n<td>stochastic_weight_avg</td>\n<td>False</td>\n<td>False</td>\n</tr>\n<tr>\n<td>val_check_interval</td>\n<td>0.05</td>\n<td>0.05</td>\n</tr>\n</tbody>\n</table>",
          "rawMarkdown": "Thanks! I'm not doing anything special. \nThe main hyperparameters are as follows: \n\n| nlp_model_name        | funnel-large-base | funnel-large |\n|-----------------------|-------------------|--------------|\n| dropout               | 0                 | 0            |\n| batch_size            | 12                | 12           |\n| lr_bert               | 2E-05             | 2E-05        |\n| lr_fc                 | 5E-05             | 5E-05        |\n| warmup_ratio          | 0.05              | 0.05         |\n| epochs                | 6                 | 6            |\n| activation            | GELU              | GELU         |\n| optimizer             | AdamW             | AdamW        |\n| weight_decay          | 0.1               | 0.1          |\n| rnn_module            | LSTM              | LSTM         |\n| rnn_module_num        | 0                 | 1            |\n| rnn_hidden_indice     | (-1, 0)           | (-1, 0)      |\n| linear_vocab_enable   | False             | True         |\n| multi_dropout_ratio   | 0.3               | 0.3          |\n| multi_dropout_num     | 10                | 10           |\n| max_length            | 256               | 256          |\n| hidden_stack_enable   | True              | True         |\n| reinit_layers         | 4                 | 4            |\n| gradient_clipping     | 0.2               | 0.2          |\n| feature_enable        | False             | True         |\n| stochastic_weight_avg | False             | False        |\n| val_check_interval    | 0.05              | 0.05         |",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:44:00Z",
  "competition_id": 25914,
  "competition_slug": "commonlitreadabilityprize",
  "team_id": 6735625,
  "solution_url": "/competitions/commonlitreadabilityprize/writeups/kurupical-private-22nd-place-solution"
}