{
  "id": 258328,
  "title": "2nd place solution",
  "content": "<p>First of all, I would like to thank Kaggle and host for hosting such an interesting competition! </p>\n<h2>Summary</h2>\n<p>I chose the final sub the one with the best Public and the one with the best CV.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>best Public</td>\n<td>0.4503</td>\n<td>0.444</td>\n<td>0.446</td>\n</tr>\n<tr>\n<td>best CV</td>\n<td>0.4449</td>\n<td>0.447</td>\n<td>0.447</td>\n</tr>\n</tbody>\n</table>\n<p>The following is about best Public.(In best CV, the weights were determined by nelder-mead and the restriction of weight summing to 1 has been removed. And the models were partially changed from best Public. )<br>\nI ensembled 19 models and did a post process. I adjusted the weights of the model by looking at the LB and CV. I've included negative values for weights as well as positive ones. In the post process, I multiplied different coefficients depending on the predicted value.</p>\n<h2>Cross Validatioin</h2>\n<p>I used the following.<br>\n<a href=\"https://www.kaggle.com/abhishek/step-1-create-folds\" target=\"_blank\">https://www.kaggle.com/abhishek/step-1-create-folds</a></p>\n<h2>Model &amp; Weight</h2>\n<p>I trained the model with dropout set to 0 except for 1 and 2. And, I did mlm pretraining only for model 3. Weights were calculated by nelder-mead and then tuned for higher LB</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV</th>\n<th>Public</th>\n<th>weight</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1.  roberta-base -&gt; svr</td>\n<td>0.500</td>\n<td>0.476</td>\n<td>0.020</td>\n</tr>\n<tr>\n<td>2. roberta-base -&gt; ridge</td>\n<td>0.500</td>\n<td></td>\n<td>0.020</td>\n</tr>\n<tr>\n<td>3. roberta-base</td>\n<td>0.485</td>\n<td>0.476</td>\n<td>0.040</td>\n</tr>\n<tr>\n<td>4. roberta-large</td>\n<td>0.483</td>\n<td>0.463</td>\n<td>0.088</td>\n</tr>\n<tr>\n<td>5. muppet-roberta-large</td>\n<td>0.480</td>\n<td>0.466</td>\n<td>0.022</td>\n</tr>\n<tr>\n<td>6.  bart-large</td>\n<td>0.476</td>\n<td>0.469</td>\n<td>0.090</td>\n</tr>\n<tr>\n<td>7.  electra-large</td>\n<td>0.483</td>\n<td>0.470</td>\n<td>0.050</td>\n</tr>\n<tr>\n<td>8.  funnel-large-base</td>\n<td>0.479</td>\n<td>0.471</td>\n<td>0.050</td>\n</tr>\n<tr>\n<td>9.  deberta-large</td>\n<td>0.481</td>\n<td>0.460</td>\n<td>0.230</td>\n</tr>\n<tr>\n<td>10.  deberta-v2-xlarge</td>\n<td>0.486</td>\n<td>0.466</td>\n<td>0.050</td>\n</tr>\n<tr>\n<td>11.  mpnet-base</td>\n<td>0.482</td>\n<td>0.470</td>\n<td>0.130</td>\n</tr>\n<tr>\n<td>12.  deberta-v2-xxlarge</td>\n<td>0.482</td>\n<td>0.465</td>\n<td>0.140</td>\n</tr>\n<tr>\n<td>13.  funnel-large</td>\n<td>0.475</td>\n<td>0.464</td>\n<td>0.110</td>\n</tr>\n<tr>\n<td>14.  gpt2-medium</td>\n<td>0.498</td>\n<td>0.478</td>\n<td>0.170</td>\n</tr>\n<tr>\n<td>15.  albert-v2-xxlarge</td>\n<td>0.486</td>\n<td>0.467</td>\n<td>0.120</td>\n</tr>\n<tr>\n<td>16.  electra-base</td>\n<td>0.493</td>\n<td></td>\n<td>-0.170</td>\n</tr>\n<tr>\n<td>17.  bert-base-uncased</td>\n<td>0.507</td>\n<td></td>\n<td>-0.140</td>\n</tr>\n<tr>\n<td>18.  t5-large</td>\n<td>0.504</td>\n<td></td>\n<td>-0.110</td>\n</tr>\n<tr>\n<td>19.  distilbart-cnn-12-6</td>\n<td>0.489</td>\n<td>0.479</td>\n<td>0.090</td>\n</tr>\n</tbody>\n</table>\n<h2>Post process</h2>\n<p>Post process improved the score by about 0.001 ~ 0.002. The coefficients were calculated by nelder-mead and then tuned by looking at Public. The threshold was determined by looking at CV and Public</p>\n<ul>\n<li>pred &gt;= 0.3  -&gt;  pred * 1.07</li>\n<li>0.3 &gt; pred &gt;= 0 -&gt;  pred * 1.2</li>\n<li>0 &gt; pred &gt;= -0.7 -&gt;  pred * 0.974</li>\n<li>-0.7 &gt; pred &gt;= -0.9 -&gt;  pred * 1.01</li>\n<li>-0.9 &gt; pred &gt; =-2 -&gt;  pred * 1.021</li>\n<li>-2  &gt; pred -&gt;  pred * 1.027</li>\n</ul>\n<h2>GPU</h2>\n<ul>\n<li>A100(GCP)</li>\n<li>V100(GCP)</li>\n<li>P100(kaggle)</li>\n</ul>\n<h2>Edit</h2>\n<p>Inference notebook(final sub)<br>\n<a href=\"https://www.kaggle.com/takoihiraokazu/lb-ensemble-add-electra-base-bert-base2-t5-diba2?scriptVersionId=69659799\" target=\"_blank\">https://www.kaggle.com/takoihiraokazu/lb-ensemble-add-electra-base-bert-base2-t5-diba2?scriptVersionId=69659799</a><br>\nInference notebook(Partially modified version)<br>\n<a href=\"https://www.kaggle.com/takoihiraokazu/final-sub1\" target=\"_blank\">https://www.kaggle.com/takoihiraokazu/final-sub1</a><br>\ncode<br>\n<a href=\"https://github.com/TakoiHirokazu/kaggle_commonLit_readability_prize\" target=\"_blank\">https://github.com/TakoiHirokazu/kaggle_commonLit_readability_prize</a></p>",
  "comments": [
    {
      "id": 1436096,
      "author_name": "Muktan Patel",
      "author_url": "",
      "post_date": "2021-08-03T14:57:50.857000",
      "content": "<p>Congratulations! 🎉<br>\nHad tried the mentioned post-processing approach but it didn't seem to work for me. I wanted to know how did you find those postprocessing weights.</p>",
      "votes": 5,
      "replies": [
        {
          "id": 1443807,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-04T04:13:03.320000",
          "content": "<p>Thanks!<br>\nFirst, I  calculated the coefficients with the following. The threshold value was determined by looking at the distribution of the target of the train and LB while changing the threshold value.<br>\nBecause of the slight difference in the distribution of the target between train and test, it was better to increase or decrease the coefficients to get a better LB than to use the results as they are, so I adjusted them while watching the LB.</p>\n<pre><code>from scipy.optimize import minimize\ndef f(x):\n    pred1 = y_test.copy()\n    pred1[y_test &gt;= 0.3] = y_test[y_test&gt;= 0.3] * x[0]\n    pred1[(y_test &lt; 0.3) &amp; (y_test &gt;= 0)] = y_test[(y_test&lt; 0.3) &amp; (y_test &gt;= 0)] * x[1]\n    pred1[(y_test &lt; 0) &amp; (y_test &gt;= -0.7)] = y_test[(y_test &lt; 0) &amp; (y_test &gt;= -0.7)] * x[2]\n    pred1[(y_test &lt; -0.7) &amp; (y_test &gt;= -0.9)] = y_test[(y_test&lt; -0.7) &amp; (y_test&gt;= -0.9)] * x[3]\n    pred1[(y_test &lt; -0.9) &amp; (y_test &gt;= -2.0)] = y_test[(y_test&lt; -0.9) &amp; (y_test&gt;= -2.0)] * x[4]\n    pred1[(y_test &lt; -2.0)] = y_test[(y_test &lt; -2.0)] * x[5]\n    score = np.sqrt(mean_squared_error(y, pred1))\n    return score\nresult = minimize(f, [1,1,1,1,1,1], method=\"Nelder-Mead\")\nresult\n</code></pre>",
          "votes": 7,
          "replies": [
            {
              "id": 2459897,
              "author_name": "",
              "author_url": "",
              "post_date": "2023-09-28T12:57:39.363000",
              "content": "<p><a href=\"https://www.kaggle.com/takoihiraokazu\" target=\"_blank\">@takoihiraokazu</a> , hi :)</p>\n<p>why you use this number for name files:)?</p>\n<p>ex194.py</p>\n<p>ex216.py</p>\n<p>ex237.py</p>\n<p>ex272.py</p>\n<p>ex292.py</p>\n<p>ex384.py</p>\n<p>ex407.py</p>\n<p>ex429.py</p>\n<p>ex434.py</p>\n<p>ex4..</p>",
              "votes": 0,
              "replies": []
            }
          ]
        },
        {
          "id": 1444902,
          "author_name": "Muktan Patel",
          "author_url": "",
          "post_date": "2021-08-04T07:17:58.173000",
          "content": "<p>Awesome, Did it took lots of submissions to figure out that?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1444935,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-04T07:23:00.310000",
          "content": "<p>Yes, roughly 15 subs were used to tune the post process.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1448929,
          "author_name": "Vinay Ethiraj",
          "author_url": "",
          "post_date": "2021-08-04T19:14:50.153000",
          "content": "<p>Congratulations! and I am just wowed with the post processing. I am relatively new to post processing, do you have any good material where I can learn about the process you have described above? </p>\n<p>Thank you for sharing!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1449798,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-05T00:50:59.160000",
          "content": "<p>I'm sorry, but I don't know of any material to learn about post process. In this case, I decided to post process by comparing the target and predicted values, or by looking at how many weights sum up when optimizing the ensemble weights. The total optimized weight was about 1.02.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1455725,
          "author_name": "Vinay Ethiraj",
          "author_url": "",
          "post_date": "2021-08-06T17:00:03.550000",
          "content": "<p>No worries, thank you for the reply!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1433509,
      "author_name": "عثمان",
      "author_url": "",
      "post_date": "2021-08-03T14:03:55.497000",
      "content": "<p>おめでとう takoi-san on the strong finish<br>\nwould it be possible to publicize the kernel you used to do the ensemble optimization?<br>\nthank you!</p>",
      "votes": 3,
      "replies": [
        {
          "id": 1443414,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-04T02:55:49.803000",
          "content": "<p>ありがとうございます！<br>\nFirst, I set the initial value of weight as shown below, and then I adjusted the weight so that the LB would be high by increasing the weight of models with high LB.</p>\n<pre><code>from scipy.optimize import minimize \ndef f(x):\n    pred1 = (ex15 + ex237)/2 * x[0] + ex64 * x[1] + \\\n            (ex72 * 0.8 + ex384 * 0.2)* x[2] +  ex107 * x[3] + \\\n            (ex190 + ex272) / 2 * x[4] + ex182 * x[5] + ex194 * x[6] +\\\n             ex292 * x[7] + ex216 * x[8] + ex407 * x[9] +  ex429 * x[10] +\\\n             ex450 * x[11] + ex465 * x[12] + ex434 * x[13] + \\\n             ex497 * x[14] + ex507 * x[15]\n    score = np.sqrt(mean_squared_error(y, pred1))\n    return score\nweight_init = [ 1 / 16 for _ in range(16)]\nresult = minimize(f, weight_init , method=\"Nelder-Mead\")\nresult\n</code></pre>",
          "votes": 7,
          "replies": []
        }
      ]
    },
    {
      "id": 1427938,
      "author_name": "Mengfei Li",
      "author_url": "",
      "post_date": "2021-08-03T08:57:29.973000",
      "content": "<p>Thanks, wow you got an A100 !</p>",
      "votes": 4,
      "replies": [
        {
          "id": 1451197,
          "author_name": "Yassine Alouini",
          "author_url": "",
          "post_date": "2021-08-05T09:27:38.793000",
          "content": "<p>It was through GCP so anyone can get it though it is expensive. 😄</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1460917,
      "author_name": "ykskks",
      "author_url": "",
      "post_date": "2021-08-09T06:11:14.883000",
      "content": "<p>Congrats on your 2nd place!</p>\n<p>You said you tuned model weights for higher LB, however I thought the public data size was too small to<br>\ndo that. Why do you think this approach (tuning to LB) worked in this comp, even though public data size is small?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1460943,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-09T06:36:43.140000",
          "content": "<p>Thanks!<br>\nI did not know during the competition if LB tuning would work in this competition. Since the socre I could refer to was LB and CV, I tried to make one sub higher LB, although the data was small.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1453131,
      "author_name": "Michal Bogacz",
      "author_url": "",
      "post_date": "2021-08-05T19:13:08.083000",
      "content": "<p>Congratulations, thanks for sharing your approach! <br>\nVery interesting idea with negative weights!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1587560,
          "author_name": "Sarthak Bhatt",
          "author_url": "",
          "post_date": "2021-11-18T19:38:15.003000",
          "content": "<p>How could negative weights possibly work?</p>",
          "votes": -1,
          "replies": []
        }
      ]
    },
    {
      "id": 1452096,
      "author_name": "Simon Meoni",
      "author_url": "",
      "post_date": "2021-08-05T14:16:30.643000",
      "content": "<p>Congratulations and very interesting solution! I have questions concerning the nelder-mead method. </p>\n<p>Do you have any material to understand the principle of this optimization? Which python library are you used to apply it ?<br>\nAnd another one, which head are you used for your models? </p>",
      "votes": 1,
      "replies": [
        {
          "id": 1452520,
          "author_name": "Yassine Alouini",
          "author_url": "",
          "post_date": "2021-08-05T16:17:03.053000",
          "content": "<p>The numpy documentation is a good starting point: <a href=\"https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html\" target=\"_blank\">https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html</a></p>\n<p>In the reference, there is a link to a paper but it is behind a paywall: <a href=\"https://link.springer.com/article/10.1007/s10589-010-9329-3\" target=\"_blank\">https://link.springer.com/article/10.1007/s10589-010-9329-3</a></p>\n<p>There is another one here accessible: <a href=\"https://www.webpages.uidaho.edu/~fuchang/res/ANMS.pdf\" target=\"_blank\">https://www.webpages.uidaho.edu/~fuchang/res/ANMS.pdf</a> </p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1460714,
          "author_name": "Chinta",
          "author_url": "",
          "post_date": "2021-08-09T03:36:58.837000",
          "content": "<p><a href=\"https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method\" target=\"_blank\">https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method</a></p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1447467,
      "author_name": "sunathr",
      "author_url": "",
      "post_date": "2021-08-04T15:00:00.923000",
      "content": "<p>Congratulations on the prize!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1445621,
      "author_name": "Yassine Alouini",
      "author_url": "",
      "post_date": "2021-08-04T09:32:40.323000",
      "content": "<p>That's a lot of models. 😄<br>\nCongratulations on the second place and sharing your insights!<br>\nHow much did it cost you to have these GPU instances? 👀</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1445903,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-04T10:25:13.643000",
          "content": "<p>Thank you very much. The cost of GPU instances was about $1500.</p>",
          "votes": 5,
          "replies": []
        },
        {
          "id": 1446557,
          "author_name": "Yassine Alouini",
          "author_url": "",
          "post_date": "2021-08-04T12:33:53.380000",
          "content": "<p>You are welcome. That's relatively a lot of money but worth it considering you got second place. Again, well done!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1445219,
      "author_name": "Old Monk",
      "author_url": "",
      "post_date": "2021-08-04T08:09:52.893000",
      "content": "<p>Awesome work, thanks for sharing! Congratulations for having A100 as well !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1442413,
      "author_name": "Salim Khazem",
      "author_url": "",
      "post_date": "2021-08-03T22:41:55.037000",
      "content": "<p>Congrats for the great performance, and thanks for sharing your approach </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1439123,
      "author_name": "Sergey Zlobin",
      "author_url": "",
      "post_date": "2021-08-03T16:45:54.777000",
      "content": "<p>Did you use any external dateset? </p>",
      "votes": 1,
      "replies": [
        {
          "id": 1443664,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-04T03:47:50.850000",
          "content": "<p>I did not use any external dataset.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1434960,
      "author_name": "spy1234567",
      "author_url": "",
      "post_date": "2021-08-03T14:37:39.127000",
      "content": "<p>Congrats! Thanks for the sharing, you mentioned that your used dropout in the 1st and 2 end models, did you use dropout in the Bert part or regression part?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1443428,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-04T02:58:51.767000",
          "content": "<p>Thanks! I used drop out in bert part.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1443500,
          "author_name": "spy1234567",
          "author_url": "",
          "post_date": "2021-08-04T03:11:13.163000",
          "content": "<p>Hi, Takoi, I also tried to use dropout in the bert part, but I got really bad result after adding dropout, this confused me for a long time to figure out why dropout hurt the performance of bert based models, do you have some idea about that? Or do I need some other additional processing after adding the dropout in the bert part? Thank you!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1449801,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-05T00:52:39.677000",
          "content": "<p>I'm sorry for the late reply, but the score of the one with drop out was low as it is, so I output embedding and used it as a feature to train with svr and ridge.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1449869,
          "author_name": "spy1234567",
          "author_url": "",
          "post_date": "2021-08-05T01:41:00.743000",
          "content": "<p>Thanks a lot for your reply!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1429230,
      "author_name": "hxcc",
      "author_url": "",
      "post_date": "2021-08-03T10:26:50.503000",
      "content": "<p>Congratulations on becoming a Grandmaster! What head did you use for your models? Did you get a boost in private LB from deberta large to xxlarge?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1443289,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-04T02:29:00.413000",
          "content": "<p>Thanks! I mainly used cls embedding and mean pooling. The privete score of deberta large is 0.463 and the private score of deberta xxlarge is 0.466. deberta large was better.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1427829,
      "author_name": "SSinyu",
      "author_url": "",
      "post_date": "2021-08-03T08:50:40.423000",
      "content": "<p>Congratulations!, Thank you for sharing your experience</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1450521,
      "author_name": "Sergey Zlobin",
      "author_url": "",
      "post_date": "2021-08-05T05:57:34.487000",
      "content": "<p>One more question. Why did you use models with negative weights? Maybe it's better to remove them completely.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1451018,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-05T08:37:07.577000",
          "content": "<p>Thanks for the question. I used the model with negative weights because it improved my score. In fact, it improved by 0.003 for public and 0.004 for private. </p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1451062,
          "author_name": "hxcc",
          "author_url": "",
          "post_date": "2021-08-05T08:50:05.337000",
          "content": "<p>😂I find it really funny when a model is so bad taking a negative of its prediction improves score.</p>",
          "votes": 6,
          "replies": []
        },
        {
          "id": 1451071,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-05T08:53:58.907000",
          "content": "<p><a href=\"https://www.kaggle.com/potatoc5\" target=\"_blank\">@potatoc5</a> I think so too. This is the first time I've used negative weights in an ensemble.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1451173,
          "author_name": "Yassine Alouini",
          "author_url": "",
          "post_date": "2021-08-05T09:22:43.427000",
          "content": "<p>That's interesting indeed. It seems weights aren't correlated much to your CV score (i.e. some models with better CV have lower weights and even negative as noted above). Do you know what could explain this discrepancy? </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1451258,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-05T09:50:58.230000",
          "content": "<p>In Best public, weight is manually adjusted by looking at LB, so there is not much correlation between CV and weight. As for the negative weight, I was referring to the value when I did the weight optimization.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1453355,
          "author_name": "Sergey Zlobin",
          "author_url": "",
          "post_date": "2021-08-05T20:39:52.703000",
          "content": "<blockquote>\n  <p>In fact, it improved by 0.003 for public and 0.004 for private. </p>\n</blockquote>\n<p>Wow, great! Thanks for the answer!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1428453,
      "author_name": "kurupical",
      "author_url": "",
      "post_date": "2021-08-03T09:29:12.567000",
      "content": "<p>Congratulations on your promotion to Grand Master!<br>\n I'll try to catch up with you!!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1428910,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2021-08-03T10:12:48.037000",
          "content": "<p>Thanks! I expect you to be GM in next competition!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 44,
  "messages": [
    {
      "id": 1427723,
      "postDate": "2021-08-03T08:44:17.573Z",
      "content": "<p>First of all, I would like to thank Kaggle and host for hosting such an interesting competition! </p>\n<h2>Summary</h2>\n<p>I chose the final sub the one with the best Public and the one with the best CV.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>CV</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>best Public</td>\n<td>0.4503</td>\n<td>0.444</td>\n<td>0.446</td>\n</tr>\n<tr>\n<td>best CV</td>\n<td>0.4449</td>\n<td>0.447</td>\n<td>0.447</td>\n</tr>\n</tbody>\n</table>\n<p>The following is about best Public.(In best CV, the weights were determined by nelder-mead and the restriction of weight summing to 1 has been removed. And the models were partially changed from best Public. )<br>\nI ensembled 19 models and did a post process. I adjusted the weights of the model by looking at the LB and CV. I've included negative values for weights as well as positive ones. In the post process, I multiplied different coefficients depending on the predicted value.</p>\n<h2>Cross Validatioin</h2>\n<p>I used the following.<br>\n<a href=\"https://www.kaggle.com/abhishek/step-1-create-folds\" target=\"_blank\">https://www.kaggle.com/abhishek/step-1-create-folds</a></p>\n<h2>Model &amp; Weight</h2>\n<p>I trained the model with dropout set to 0 except for 1 and 2. And, I did mlm pretraining only for model 3. Weights were calculated by nelder-mead and then tuned for higher LB</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV</th>\n<th>Public</th>\n<th>weight</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1.  roberta-base -&gt; svr</td>\n<td>0.500</td>\n<td>0.476</td>\n<td>0.020</td>\n</tr>\n<tr>\n<td>2. roberta-base -&gt; ridge</td>\n<td>0.500</td>\n<td></td>\n<td>0.020</td>\n</tr>\n<tr>\n<td>3. roberta-base</td>\n<td>0.485</td>\n<td>0.476</td>\n<td>0.040</td>\n</tr>\n<tr>\n<td>4. roberta-large</td>\n<td>0.483</td>\n<td>0.463</td>\n<td>0.088</td>\n</tr>\n<tr>\n<td>5. muppet-roberta-large</td>\n<td>0.480</td>\n<td>0.466</td>\n<td>0.022</td>\n</tr>\n<tr>\n<td>6.  bart-large</td>\n<td>0.476</td>\n<td>0.469</td>\n<td>0.090</td>\n</tr>\n<tr>\n<td>7.  electra-large</td>\n<td>0.483</td>\n<td>0.470</td>\n<td>0.050</td>\n</tr>\n<tr>\n<td>8.  funnel-large-base</td>\n<td>0.479</td>\n<td>0.471</td>\n<td>0.050</td>\n</tr>\n<tr>\n<td>9.  deberta-large</td>\n<td>0.481</td>\n<td>0.460</td>\n<td>0.230</td>\n</tr>\n<tr>\n<td>10.  deberta-v2-xlarge</td>\n<td>0.486</td>\n<td>0.466</td>\n<td>0.050</td>\n</tr>\n<tr>\n<td>11.  mpnet-base</td>\n<td>0.482</td>\n<td>0.470</td>\n<td>0.130</td>\n</tr>\n<tr>\n<td>12.  deberta-v2-xxlarge</td>\n<td>0.482</td>\n<td>0.465</td>\n<td>0.140</td>\n</tr>\n<tr>\n<td>13.  funnel-large</td>\n<td>0.475</td>\n<td>0.464</td>\n<td>0.110</td>\n</tr>\n<tr>\n<td>14.  gpt2-medium</td>\n<td>0.498</td>\n<td>0.478</td>\n<td>0.170</td>\n</tr>\n<tr>\n<td>15.  albert-v2-xxlarge</td>\n<td>0.486</td>\n<td>0.467</td>\n<td>0.120</td>\n</tr>\n<tr>\n<td>16.  electra-base</td>\n<td>0.493</td>\n<td></td>\n<td>-0.170</td>\n</tr>\n<tr>\n<td>17.  bert-base-uncased</td>\n<td>0.507</td>\n<td></td>\n<td>-0.140</td>\n</tr>\n<tr>\n<td>18.  t5-large</td>\n<td>0.504</td>\n<td></td>\n<td>-0.110</td>\n</tr>\n<tr>\n<td>19.  distilbart-cnn-12-6</td>\n<td>0.489</td>\n<td>0.479</td>\n<td>0.090</td>\n</tr>\n</tbody>\n</table>\n<h2>Post process</h2>\n<p>Post process improved the score by about 0.001 ~ 0.002. The coefficients were calculated by nelder-mead and then tuned by looking at Public. The threshold was determined by looking at CV and Public</p>\n<ul>\n<li>pred &gt;= 0.3  -&gt;  pred * 1.07</li>\n<li>0.3 &gt; pred &gt;= 0 -&gt;  pred * 1.2</li>\n<li>0 &gt; pred &gt;= -0.7 -&gt;  pred * 0.974</li>\n<li>-0.7 &gt; pred &gt;= -0.9 -&gt;  pred * 1.01</li>\n<li>-0.9 &gt; pred &gt; =-2 -&gt;  pred * 1.021</li>\n<li>-2  &gt; pred -&gt;  pred * 1.027</li>\n</ul>\n<h2>GPU</h2>\n<ul>\n<li>A100(GCP)</li>\n<li>V100(GCP)</li>\n<li>P100(kaggle)</li>\n</ul>\n<h2>Edit</h2>\n<p>Inference notebook(final sub)<br>\n<a href=\"https://www.kaggle.com/takoihiraokazu/lb-ensemble-add-electra-base-bert-base2-t5-diba2?scriptVersionId=69659799\" target=\"_blank\">https://www.kaggle.com/takoihiraokazu/lb-ensemble-add-electra-base-bert-base2-t5-diba2?scriptVersionId=69659799</a><br>\nInference notebook(Partially modified version)<br>\n<a href=\"https://www.kaggle.com/takoihiraokazu/final-sub1\" target=\"_blank\">https://www.kaggle.com/takoihiraokazu/final-sub1</a><br>\ncode<br>\n<a href=\"https://github.com/TakoiHirokazu/kaggle_commonLit_readability_prize\" target=\"_blank\">https://github.com/TakoiHirokazu/kaggle_commonLit_readability_prize</a></p>",
      "rawMarkdown": "First of all, I would like to thank Kaggle and host for hosting such an interesting competition! \n\n## Summary\nI chose the final sub the one with the best Public and the one with the best CV.\n|  | CV|Public|Private|\n| --- | --- |--- |\n |best Public|0.4503 | 0.444 |0.446|\n |best CV|0.4449 | 0.447 |0.447|\n\nThe following is about best Public.(In best CV, the weights were determined by nelder-mead and the restriction of weight summing to 1 has been removed. And the models were partially changed from best Public. )\nI ensembled 19 models and did a post process. I adjusted the weights of the model by looking at the LB and CV. I've included negative values for weights as well as positive ones. In the post process, I multiplied different coefficients depending on the predicted value.\n\n\n## Cross Validatioin\nI used the following.\nhttps://www.kaggle.com/abhishek/step-1-create-folds\n\n\n## Model & Weight\nI trained the model with dropout set to 0 except for 1 and 2. And, I did mlm pretraining only for model 3. Weights were calculated by nelder-mead and then tuned for higher LB\n| model | CV | Public | weight |\n| --- | --- |\n|1.  roberta-base -> svr | 0.500 |0.476|0.020|\n|2. roberta-base -> ridge | 0.500 ||0.020|\n|3. roberta-base| 0.485 |0.476|0.040|\n|4. roberta-large| 0.483 |0.463|0.088|\n|5. muppet-roberta-large| 0.480 |0.466|0.022|\n|6.  bart-large | 0.476 |0.469|0.090|\n|7.  electra-large | 0.483 |0.470|0.050|\n|8.  funnel-large-base| 0.479 |0.471|0.050|\n|9.  deberta-large| 0.481 |0.460|0.230|\n|10.  deberta-v2-xlarge | 0.486 |0.466|0.050|\n|11.  mpnet-base | 0.482 |0.470|0.130|\n|12.  deberta-v2-xxlarge | 0.482 |0.465|0.140|\n|13.  funnel-large | 0.475 |0.464|0.110|\n|14.  gpt2-medium | 0.498 |0.478|0.170|\n|15.  albert-v2-xxlarge | 0.486 |0.467|0.120|\n|16.  electra-base | 0.493 ||-0.170|\n|17.  bert-base-uncased |0.507  ||-0.140|\n|18.  t5-large |0.504  ||-0.110|\n|19.  distilbart-cnn-12-6 |0.489  |0.479 |0.090|\n\n## Post process\nPost process improved the score by about 0.001 ~ 0.002. The coefficients were calculated by nelder-mead and then tuned by looking at Public. The threshold was determined by looking at CV and Public\n- pred >= 0.3  ->  pred * 1.07\n- 0.3 > pred >= 0 ->  pred * 1.2\n- 0 > pred >= -0.7 ->  pred * 0.974\n- -0.7 > pred >= -0.9 ->  pred * 1.01\n- -0.9 > pred > =-2 ->  pred * 1.021\n- -2  > pred ->  pred * 1.027\n\n## GPU\n- A100(GCP)\n- V100(GCP)\n- P100(kaggle)\n\n## Edit\nInference notebook(final sub)\nhttps://www.kaggle.com/takoihiraokazu/lb-ensemble-add-electra-base-bert-base2-t5-diba2?scriptVersionId=69659799\nInference notebook(Partially modified version)\nhttps://www.kaggle.com/takoihiraokazu/final-sub1\ncode\nhttps://github.com/TakoiHirokazu/kaggle_commonLit_readability_prize\n\n\n\n\n\n\n",
      "votes": 163
    },
    {
      "id": 1436096,
      "postDate": "2021-08-03T14:57:50.857Z",
      "content": "<p>Congratulations! 🎉<br>\nHad tried the mentioned post-processing approach but it didn't seem to work for me. I wanted to know how did you find those postprocessing weights.</p>",
      "rawMarkdown": "Congratulations! 🎉\nHad tried the mentioned post-processing approach but it didn't seem to work for me. I wanted to know how did you find those postprocessing weights.",
      "votes": 5,
      "replies": [
        {
          "id": 1443807,
          "postDate": "2021-08-04T04:13:03.320Z",
          "content": "<p>Thanks!<br>\nFirst, I  calculated the coefficients with the following. The threshold value was determined by looking at the distribution of the target of the train and LB while changing the threshold value.<br>\nBecause of the slight difference in the distribution of the target between train and test, it was better to increase or decrease the coefficients to get a better LB than to use the results as they are, so I adjusted them while watching the LB.</p>\n<pre><code>from scipy.optimize import minimize\ndef f(x):\n    pred1 = y_test.copy()\n    pred1[y_test &gt;= 0.3] = y_test[y_test&gt;= 0.3] * x[0]\n    pred1[(y_test &lt; 0.3) &amp; (y_test &gt;= 0)] = y_test[(y_test&lt; 0.3) &amp; (y_test &gt;= 0)] * x[1]\n    pred1[(y_test &lt; 0) &amp; (y_test &gt;= -0.7)] = y_test[(y_test &lt; 0) &amp; (y_test &gt;= -0.7)] * x[2]\n    pred1[(y_test &lt; -0.7) &amp; (y_test &gt;= -0.9)] = y_test[(y_test&lt; -0.7) &amp; (y_test&gt;= -0.9)] * x[3]\n    pred1[(y_test &lt; -0.9) &amp; (y_test &gt;= -2.0)] = y_test[(y_test&lt; -0.9) &amp; (y_test&gt;= -2.0)] * x[4]\n    pred1[(y_test &lt; -2.0)] = y_test[(y_test &lt; -2.0)] * x[5]\n    score = np.sqrt(mean_squared_error(y, pred1))\n    return score\nresult = minimize(f, [1,1,1,1,1,1], method=\"Nelder-Mead\")\nresult\n</code></pre>",
          "rawMarkdown": "Thanks!\nFirst, I  calculated the coefficients with the following. The threshold value was determined by looking at the distribution of the target of the train and LB while changing the threshold value.\nBecause of the slight difference in the distribution of the target between train and test, it was better to increase or decrease the coefficients to get a better LB than to use the results as they are, so I adjusted them while watching the LB.\n```\nfrom scipy.optimize import minimize\ndef f(x):\n    pred1 = y_test.copy()\n    pred1[y_test >= 0.3] = y_test[y_test>= 0.3] * x[0]\n    pred1[(y_test < 0.3) & (y_test >= 0)] = y_test[(y_test< 0.3) & (y_test >= 0)] * x[1]\n    pred1[(y_test < 0) & (y_test >= -0.7)] = y_test[(y_test < 0) & (y_test >= -0.7)] * x[2]\n    pred1[(y_test < -0.7) & (y_test >= -0.9)] = y_test[(y_test< -0.7) & (y_test>= -0.9)] * x[3]\n    pred1[(y_test < -0.9) & (y_test >= -2.0)] = y_test[(y_test< -0.9) & (y_test>= -2.0)] * x[4]\n    pred1[(y_test < -2.0)] = y_test[(y_test < -2.0)] * x[5]\n    score = np.sqrt(mean_squared_error(y, pred1))\n    return score\nresult = minimize(f, [1,1,1,1,1,1], method=\"Nelder-Mead\")\nresult\n```",
          "votes": 7,
          "replies": [
            {
              "id": 2459897,
              "postDate": "2023-09-28T12:57:39.363Z",
              "content": "<p><a href=\"https://www.kaggle.com/takoihiraokazu\" target=\"_blank\">@takoihiraokazu</a> , hi :)</p>\n<p>why you use this number for name files:)?</p>\n<p>ex194.py</p>\n<p>ex216.py</p>\n<p>ex237.py</p>\n<p>ex272.py</p>\n<p>ex292.py</p>\n<p>ex384.py</p>\n<p>ex407.py</p>\n<p>ex429.py</p>\n<p>ex434.py</p>\n<p>ex4..</p>",
              "rawMarkdown": "@takoihiraokazu , hi :)\n\nwhy you use this number for name files:)?\n\n\n\nex194.py\n\nex216.py\n\nex237.py\n\nex272.py\n\nex292.py\n\nex384.py\n\nex407.py\n\nex429.py\n\nex434.py\n\nex4.."
            }
          ]
        },
        {
          "id": 1444902,
          "postDate": "2021-08-04T07:17:58.173Z",
          "content": "<p>Awesome, Did it took lots of submissions to figure out that?</p>",
          "rawMarkdown": "Awesome, Did it took lots of submissions to figure out that?",
          "votes": 1
        },
        {
          "id": 1444935,
          "postDate": "2021-08-04T07:23:00.310Z",
          "content": "<p>Yes, roughly 15 subs were used to tune the post process.</p>",
          "rawMarkdown": "Yes, roughly 15 subs were used to tune the post process.",
          "votes": 2
        },
        {
          "id": 1448929,
          "postDate": "2021-08-04T19:14:50.153Z",
          "content": "<p>Congratulations! and I am just wowed with the post processing. I am relatively new to post processing, do you have any good material where I can learn about the process you have described above? </p>\n<p>Thank you for sharing!</p>",
          "rawMarkdown": "Congratulations! and I am just wowed with the post processing. I am relatively new to post processing, do you have any good material where I can learn about the process you have described above? \n\nThank you for sharing!",
          "votes": 1
        },
        {
          "id": 1449798,
          "postDate": "2021-08-05T00:50:59.160Z",
          "content": "<p>I'm sorry, but I don't know of any material to learn about post process. In this case, I decided to post process by comparing the target and predicted values, or by looking at how many weights sum up when optimizing the ensemble weights. The total optimized weight was about 1.02.</p>",
          "rawMarkdown": "I'm sorry, but I don't know of any material to learn about post process. In this case, I decided to post process by comparing the target and predicted values, or by looking at how many weights sum up when optimizing the ensemble weights. The total optimized weight was about 1.02."
        },
        {
          "id": 1455725,
          "postDate": "2021-08-06T17:00:03.550Z",
          "content": "<p>No worries, thank you for the reply!</p>",
          "rawMarkdown": "No worries, thank you for the reply!"
        }
      ]
    },
    {
      "id": 1433509,
      "postDate": "2021-08-03T14:03:55.497Z",
      "content": "<p>おめでとう takoi-san on the strong finish<br>\nwould it be possible to publicize the kernel you used to do the ensemble optimization?<br>\nthank you!</p>",
      "rawMarkdown": "おめでとう takoi-san on the strong finish\nwould it be possible to publicize the kernel you used to do the ensemble optimization?\nthank you!",
      "votes": 3,
      "replies": [
        {
          "id": 1443414,
          "postDate": "2021-08-04T02:55:49.803Z",
          "content": "<p>ありがとうございます！<br>\nFirst, I set the initial value of weight as shown below, and then I adjusted the weight so that the LB would be high by increasing the weight of models with high LB.</p>\n<pre><code>from scipy.optimize import minimize \ndef f(x):\n    pred1 = (ex15 + ex237)/2 * x[0] + ex64 * x[1] + \\\n            (ex72 * 0.8 + ex384 * 0.2)* x[2] +  ex107 * x[3] + \\\n            (ex190 + ex272) / 2 * x[4] + ex182 * x[5] + ex194 * x[6] +\\\n             ex292 * x[7] + ex216 * x[8] + ex407 * x[9] +  ex429 * x[10] +\\\n             ex450 * x[11] + ex465 * x[12] + ex434 * x[13] + \\\n             ex497 * x[14] + ex507 * x[15]\n    score = np.sqrt(mean_squared_error(y, pred1))\n    return score\nweight_init = [ 1 / 16 for _ in range(16)]\nresult = minimize(f, weight_init , method=\"Nelder-Mead\")\nresult\n</code></pre>",
          "rawMarkdown": "ありがとうございます！\nFirst, I set the initial value of weight as shown below, and then I adjusted the weight so that the LB would be high by increasing the weight of models with high LB.\n\n```\nfrom scipy.optimize import minimize \ndef f(x):\n    pred1 = (ex15 + ex237)/2 * x[0] + ex64 * x[1] + \\\n            (ex72 * 0.8 + ex384 * 0.2)* x[2] +  ex107 * x[3] + \\\n            (ex190 + ex272) / 2 * x[4] + ex182 * x[5] + ex194 * x[6] +\\\n             ex292 * x[7] + ex216 * x[8] + ex407 * x[9] +  ex429 * x[10] +\\\n             ex450 * x[11] + ex465 * x[12] + ex434 * x[13] + \\\n             ex497 * x[14] + ex507 * x[15]\n    score = np.sqrt(mean_squared_error(y, pred1))\n    return score\nweight_init = [ 1 / 16 for _ in range(16)]\nresult = minimize(f, weight_init , method=\"Nelder-Mead\")\nresult\n```",
          "votes": 7
        }
      ]
    },
    {
      "id": 1427938,
      "postDate": "2021-08-03T08:57:29.973Z",
      "content": "<p>Thanks, wow you got an A100 !</p>",
      "rawMarkdown": "Thanks, wow you got an A100 !",
      "votes": 4,
      "replies": [
        {
          "id": 1451197,
          "postDate": "2021-08-05T09:27:38.793Z",
          "content": "<p>It was through GCP so anyone can get it though it is expensive. 😄</p>",
          "rawMarkdown": "It was through GCP so anyone can get it though it is expensive. 😄",
          "votes": 1
        }
      ]
    },
    {
      "id": 1460917,
      "postDate": "2021-08-09T06:11:14.883Z",
      "content": "<p>Congrats on your 2nd place!</p>\n<p>You said you tuned model weights for higher LB, however I thought the public data size was too small to<br>\ndo that. Why do you think this approach (tuning to LB) worked in this comp, even though public data size is small?</p>",
      "rawMarkdown": "Congrats on your 2nd place!\n\nYou said you tuned model weights for higher LB, however I thought the public data size was too small to\ndo that. Why do you think this approach (tuning to LB) worked in this comp, even though public data size is small?",
      "votes": 1,
      "replies": [
        {
          "id": 1460943,
          "postDate": "2021-08-09T06:36:43.140Z",
          "content": "<p>Thanks!<br>\nI did not know during the competition if LB tuning would work in this competition. Since the socre I could refer to was LB and CV, I tried to make one sub higher LB, although the data was small.</p>",
          "rawMarkdown": "Thanks!\nI did not know during the competition if LB tuning would work in this competition. Since the socre I could refer to was LB and CV, I tried to make one sub higher LB, although the data was small.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1453131,
      "postDate": "2021-08-05T19:13:08.083Z",
      "content": "<p>Congratulations, thanks for sharing your approach! <br>\nVery interesting idea with negative weights!</p>",
      "rawMarkdown": "Congratulations, thanks for sharing your approach! \nVery interesting idea with negative weights!",
      "votes": 1,
      "replies": [
        {
          "id": 1587560,
          "postDate": "2021-11-18T19:38:15.003Z",
          "content": "<p>How could negative weights possibly work?</p>",
          "rawMarkdown": "How could negative weights possibly work?\n",
          "votes": -1
        }
      ]
    },
    {
      "id": 1452096,
      "postDate": "2021-08-05T14:16:30.643Z",
      "content": "<p>Congratulations and very interesting solution! I have questions concerning the nelder-mead method. </p>\n<p>Do you have any material to understand the principle of this optimization? Which python library are you used to apply it ?<br>\nAnd another one, which head are you used for your models? </p>",
      "rawMarkdown": "Congratulations and very interesting solution! I have questions concerning the nelder-mead method. \n\nDo you have any material to understand the principle of this optimization? Which python library are you used to apply it ?\nAnd another one, which head are you used for your models? ",
      "votes": 1,
      "replies": [
        {
          "id": 1452520,
          "postDate": "2021-08-05T16:17:03.053Z",
          "content": "<p>The numpy documentation is a good starting point: <a href=\"https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html\" target=\"_blank\">https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html</a></p>\n<p>In the reference, there is a link to a paper but it is behind a paywall: <a href=\"https://link.springer.com/article/10.1007/s10589-010-9329-3\" target=\"_blank\">https://link.springer.com/article/10.1007/s10589-010-9329-3</a></p>\n<p>There is another one here accessible: <a href=\"https://www.webpages.uidaho.edu/~fuchang/res/ANMS.pdf\" target=\"_blank\">https://www.webpages.uidaho.edu/~fuchang/res/ANMS.pdf</a> </p>",
          "rawMarkdown": "The numpy documentation is a good starting point: https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html\n\nIn the reference, there is a link to a paper but it is behind a paywall: https://link.springer.com/article/10.1007/s10589-010-9329-3\n\nThere is another one here accessible: https://www.webpages.uidaho.edu/~fuchang/res/ANMS.pdf ",
          "votes": 2
        },
        {
          "id": 1460714,
          "postDate": "2021-08-09T03:36:58.837Z",
          "content": "<p><a href=\"https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method\" target=\"_blank\">https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method</a></p>",
          "rawMarkdown": "https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method",
          "votes": 1
        }
      ]
    },
    {
      "id": 1447467,
      "postDate": "2021-08-04T15:00:00.923Z",
      "content": "<p>Congratulations on the prize!</p>",
      "rawMarkdown": "Congratulations on the prize!",
      "votes": 1
    },
    {
      "id": 1445621,
      "postDate": "2021-08-04T09:32:40.323Z",
      "content": "<p>That's a lot of models. 😄<br>\nCongratulations on the second place and sharing your insights!<br>\nHow much did it cost you to have these GPU instances? 👀</p>",
      "rawMarkdown": "That's a lot of models. 😄\nCongratulations on the second place and sharing your insights!\nHow much did it cost you to have these GPU instances? 👀",
      "votes": 1,
      "replies": [
        {
          "id": 1445903,
          "postDate": "2021-08-04T10:25:13.643Z",
          "content": "<p>Thank you very much. The cost of GPU instances was about $1500.</p>",
          "rawMarkdown": "Thank you very much. The cost of GPU instances was about $1500.",
          "votes": 5
        },
        {
          "id": 1446557,
          "postDate": "2021-08-04T12:33:53.380Z",
          "content": "<p>You are welcome. That's relatively a lot of money but worth it considering you got second place. Again, well done!</p>",
          "rawMarkdown": "You are welcome. That's relatively a lot of money but worth it considering you got second place. Again, well done!",
          "votes": 2
        }
      ]
    },
    {
      "id": 1445219,
      "postDate": "2021-08-04T08:09:52.893Z",
      "content": "<p>Awesome work, thanks for sharing! Congratulations for having A100 as well !</p>",
      "rawMarkdown": "Awesome work, thanks for sharing! Congratulations for having A100 as well !",
      "votes": 1
    },
    {
      "id": 1442413,
      "postDate": "2021-08-03T22:41:55.037Z",
      "content": "<p>Congrats for the great performance, and thanks for sharing your approach </p>",
      "rawMarkdown": "Congrats for the great performance, and thanks for sharing your approach \n",
      "votes": 1
    },
    {
      "id": 1439123,
      "postDate": "2021-08-03T16:45:54.777Z",
      "content": "<p>Did you use any external dateset? </p>",
      "rawMarkdown": "Did you use any external dateset? ",
      "votes": 1,
      "replies": [
        {
          "id": 1443664,
          "postDate": "2021-08-04T03:47:50.850Z",
          "content": "<p>I did not use any external dataset.</p>",
          "rawMarkdown": "I did not use any external dataset.",
          "votes": 1
        }
      ]
    },
    {
      "id": 1434960,
      "postDate": "2021-08-03T14:37:39.127Z",
      "content": "<p>Congrats! Thanks for the sharing, you mentioned that your used dropout in the 1st and 2 end models, did you use dropout in the Bert part or regression part?</p>",
      "rawMarkdown": "Congrats! Thanks for the sharing, you mentioned that your used dropout in the 1st and 2 end models, did you use dropout in the Bert part or regression part?",
      "votes": 1,
      "replies": [
        {
          "id": 1443428,
          "postDate": "2021-08-04T02:58:51.767Z",
          "content": "<p>Thanks! I used drop out in bert part.</p>",
          "rawMarkdown": "Thanks! I used drop out in bert part."
        },
        {
          "id": 1443500,
          "postDate": "2021-08-04T03:11:13.163Z",
          "content": "<p>Hi, Takoi, I also tried to use dropout in the bert part, but I got really bad result after adding dropout, this confused me for a long time to figure out why dropout hurt the performance of bert based models, do you have some idea about that? Or do I need some other additional processing after adding the dropout in the bert part? Thank you!</p>",
          "rawMarkdown": "Hi, Takoi, I also tried to use dropout in the bert part, but I got really bad result after adding dropout, this confused me for a long time to figure out why dropout hurt the performance of bert based models, do you have some idea about that? Or do I need some other additional processing after adding the dropout in the bert part? Thank you!",
          "votes": 1
        },
        {
          "id": 1449801,
          "postDate": "2021-08-05T00:52:39.677Z",
          "content": "<p>I'm sorry for the late reply, but the score of the one with drop out was low as it is, so I output embedding and used it as a feature to train with svr and ridge.</p>",
          "rawMarkdown": "I'm sorry for the late reply, but the score of the one with drop out was low as it is, so I output embedding and used it as a feature to train with svr and ridge."
        },
        {
          "id": 1449869,
          "postDate": "2021-08-05T01:41:00.743Z",
          "content": "<p>Thanks a lot for your reply!</p>",
          "rawMarkdown": "Thanks a lot for your reply!"
        }
      ]
    },
    {
      "id": 1429230,
      "postDate": "2021-08-03T10:26:50.503Z",
      "content": "<p>Congratulations on becoming a Grandmaster! What head did you use for your models? Did you get a boost in private LB from deberta large to xxlarge?</p>",
      "rawMarkdown": "Congratulations on becoming a Grandmaster! What head did you use for your models? Did you get a boost in private LB from deberta large to xxlarge?",
      "votes": 1,
      "replies": [
        {
          "id": 1443289,
          "postDate": "2021-08-04T02:29:00.413Z",
          "content": "<p>Thanks! I mainly used cls embedding and mean pooling. The privete score of deberta large is 0.463 and the private score of deberta xxlarge is 0.466. deberta large was better.</p>",
          "rawMarkdown": "Thanks! I mainly used cls embedding and mean pooling. The privete score of deberta large is 0.463 and the private score of deberta xxlarge is 0.466. deberta large was better."
        }
      ]
    },
    {
      "id": 1427829,
      "postDate": "2021-08-03T08:50:40.423Z",
      "content": "<p>Congratulations!, Thank you for sharing your experience</p>",
      "rawMarkdown": "Congratulations!, Thank you for sharing your experience",
      "votes": 1
    },
    {
      "id": 1450521,
      "postDate": "2021-08-05T05:57:34.487Z",
      "content": "<p>One more question. Why did you use models with negative weights? Maybe it's better to remove them completely.</p>",
      "rawMarkdown": "One more question. Why did you use models with negative weights? Maybe it's better to remove them completely.",
      "votes": 2,
      "replies": [
        {
          "id": 1451018,
          "postDate": "2021-08-05T08:37:07.577Z",
          "content": "<p>Thanks for the question. I used the model with negative weights because it improved my score. In fact, it improved by 0.003 for public and 0.004 for private. </p>",
          "rawMarkdown": "Thanks for the question. I used the model with negative weights because it improved my score. In fact, it improved by 0.003 for public and 0.004 for private. ",
          "votes": 3
        },
        {
          "id": 1451062,
          "postDate": "2021-08-05T08:50:05.337Z",
          "content": "<p>😂I find it really funny when a model is so bad taking a negative of its prediction improves score.</p>",
          "rawMarkdown": "😂I find it really funny when a model is so bad taking a negative of its prediction improves score.",
          "votes": 6
        },
        {
          "id": 1451071,
          "postDate": "2021-08-05T08:53:58.907Z",
          "content": "<p><a href=\"https://www.kaggle.com/potatoc5\" target=\"_blank\">@potatoc5</a> I think so too. This is the first time I've used negative weights in an ensemble.</p>",
          "rawMarkdown": "@potatoc5 I think so too. This is the first time I've used negative weights in an ensemble.",
          "votes": 3
        },
        {
          "id": 1451173,
          "postDate": "2021-08-05T09:22:43.427Z",
          "content": "<p>That's interesting indeed. It seems weights aren't correlated much to your CV score (i.e. some models with better CV have lower weights and even negative as noted above). Do you know what could explain this discrepancy? </p>",
          "rawMarkdown": "That's interesting indeed. It seems weights aren't correlated much to your CV score (i.e. some models with better CV have lower weights and even negative as noted above). Do you know what could explain this discrepancy? ",
          "votes": 1
        },
        {
          "id": 1451258,
          "postDate": "2021-08-05T09:50:58.230Z",
          "content": "<p>In Best public, weight is manually adjusted by looking at LB, so there is not much correlation between CV and weight. As for the negative weight, I was referring to the value when I did the weight optimization.</p>",
          "rawMarkdown": "In Best public, weight is manually adjusted by looking at LB, so there is not much correlation between CV and weight. As for the negative weight, I was referring to the value when I did the weight optimization.",
          "votes": 1
        },
        {
          "id": 1453355,
          "postDate": "2021-08-05T20:39:52.703Z",
          "content": "<blockquote>\n  <p>In fact, it improved by 0.003 for public and 0.004 for private. </p>\n</blockquote>\n<p>Wow, great! Thanks for the answer!</p>",
          "rawMarkdown": ">  In fact, it improved by 0.003 for public and 0.004 for private. \n\nWow, great! Thanks for the answer!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1428453,
      "postDate": "2021-08-03T09:29:12.567Z",
      "content": "<p>Congratulations on your promotion to Grand Master!<br>\n I'll try to catch up with you!!</p>",
      "rawMarkdown": "Congratulations on your promotion to Grand Master!\n I'll try to catch up with you!!",
      "votes": 1,
      "replies": [
        {
          "id": 1428910,
          "postDate": "2021-08-03T10:12:48.037Z",
          "content": "<p>Thanks! I expect you to be GM in next competition!</p>",
          "rawMarkdown": "Thanks! I expect you to be GM in next competition!",
          "votes": 2
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T23:46:59Z",
  "competition_id": 25914,
  "competition_slug": "commonlitreadabilityprize",
  "team_id": 6763060,
  "solution_url": "/competitions/commonlitreadabilityprize/writeups/rist-takoi-2nd-place-solution"
}