{
  "id": 129927,
  "title": "3rd place solution",
  "content": "<p>First of all, I'd like to say thank you to the organizers and all teams, and congrats to all the winners.\nHere is a brief summary of my solution.\n- Solution overview\n- Model architectures\n- Training strategy\n- Post-processing\n- Blending strategy\n- What didn't work for me</p>\n\n<h2>Solution overview</h2>\n\n<ul>\n<li>text columns + category column</li>\n<li>various text truncation\n<ul><li>pre-truncate</li>\n<li>post-truncate</li>\n<li>head + tail tokens (<a href=\"https://arxiv.org/abs/1905.05583\"></a><a href=\"https://arxiv.org/abs/1905.05583\">https://arxiv.org/abs/1905.05583</a>)</li>\n<li>assign longer max_len to answer</li></ul></li>\n<li>ensemble of following models:\n<ul><li>LSTM + Universal Sentence Encoder</li>\n<li>BERT base uncased * 2</li>\n<li>BERT base cased</li>\n<li>BERT large uncased * 2</li>\n<li>BERT large cased * 2</li>\n<li>ALBERT base</li>\n<li>RoBERTa base</li>\n<li>GPT2 base</li>\n<li>XLNet base</li></ul></li>\n</ul>\n\n<h2>Model architectures</h2>\n\n<p>Surprisingly, stacking 2 Linear layers <strong>without</strong> activation performed better than single Linear layer in some cases. (I found it by a mistake. :P)\nSome of the following models have category embedding structure additionally.</p>\n\n<h3>LSTM model</h3>\n\n<p>You can check my LSTM code <a href=\"https://www.kaggle.com/sakami/google-quest-single-lstm?scriptVersionId=28487242\">here</a>.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F5ba7064f215c31d377bb991275d5fa5b%2F2020-02-11%2017.29.36.png?generation=1581409799179315&amp;alt=media\" alt=\"\"></p>\n\n<h3>BERT models</h3>\n\n<p>I used 2 types of model architectures.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Ff5da89f1cff533e174ae24d1cc459da2%2F2020-02-11%2017.14.05.png?generation=1581409945426377&amp;alt=media\" alt=\"\"></p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F80a15ecbc3f970aa7fad8308d01f2af1%2F2020-02-11%2017.14.17.png?generation=1581409964611182&amp;alt=media\" alt=\"\"></p>\n\n<h3>ALBERT model</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F506270dad509f130a53c50d177bd4861%2F2020-02-11%2020.12.38.png?generation=1581419591299259&amp;alt=media\" alt=\"\"></p>\n\n<h3>RoBERTa model</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Fd877c80ce341c7be39715eb3534627d9%2F2020-02-11%2017.14.26.png?generation=1581410441581392&amp;alt=media\" alt=\"\"></p>\n\n<h3>GPT2 model</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Ff123dc23ada1495c68b5be43ccf0241a%2F2020-02-11%2017.14.44.png?generation=1581410465561097&amp;alt=media\" alt=\"\"></p>\n\n<h3>XLNet model</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F483691fcbd3e72cb5a6053e5c8d4873a%2F2020-02-11%2017.14.55.png?generation=1581410490556290&amp;alt=media\" alt=\"\"></p>\n\n<h2>Training strategy</h2>\n\n<ul>\n<li>Min-Max target scaling\n<code>\nt_max = train_y.max(axis=0)[np.newaxis, :]\nt_min = train_y.min(axis=0)[np.newaxis, :]\ntrain_y = (train_y - t_min) / (t_max - t_min)\n</code></li>\n<li>Large weights for minor positive or negative samples\n<code>\nzero_inflated = (train_y &amp;gt; 0).mean(axis=0) &amp;lt; 0.1\npositive_weighted = np.tile(zero_inflated, (len(train_y), 1))\npositive_weighted *= (train_y &amp;gt; 0)\none_inflated = (train_y &amp;lt; 1).mean(axis=0) &amp;lt; 0.1\nnegative_weighted = np.tile(one_inflated, (len(train_y), 1))\nnegative_weighted *= (train_y &amp;lt; 1)\ntrain_weights = np.where(positive_weighted + negative_weighted, 2., 1.)\n</code></li>\n<li>gelu_new activation for BERT-based models</li>\n<li>cosine warmup scheduler</li>\n<li>EMA</li>\n</ul>\n\n<h2>Post-processing</h2>\n\n<p>I just clipped predictions. The thresholds are decided by golden section search.\nThis improve my score from 0.4434 to 0.4852. (CV score)</p>\n\n<p>before post-processing scores:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Ff8ebe510f4885db659da10499a79daea%2Fnewplot.png?generation=1581416684489031&amp;alt=media\" alt=\"\"></p>\n\n<p>after post-processing scores:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F04f51a001c9f734fed93003cfe8e935b%2Fnewplot%20(1\" alt=\"\">.png?generation=1581416716829926&amp;alt=media)</p>\n\n<p>```\nclass OptimizedRounder(BaseEstimator, TransformerMixin):</p>\n\n<pre><code>def __init__(self):\n    self.threshold = [0., 1.]\n    self.ab_start = [(0., 0.2), (0.8, 1.)]\n\ndef fit(self, train_labels, train_preds):\n    assert train_labels.shape == train_preds.shape\n    assert train_labels.ndim == 1\n\n    self.best_score = self.score(train_labels, train_preds)\n    self._golden_section_search(train_labels, train_preds, 0)  # lower threshold\n    score = self.score(train_labels, train_preds)\n    if score &amp;gt; self.best_score + 1e-3:\n        self.best_score = score\n    else:\n        self.threshold[0] = 0.\n\n    self._golden_section_search(train_labels, train_preds, 1)  # higher threshold\n    score = self.score(train_labels, train_preds)\n    if score &amp;gt; self.best_score + 1e-3:\n        self.best_score = score\n    else:\n        self.threshold[1] = 1.\n\ndef _golden_section_search(self, train_labels, train_preds, idx):\n    # idx == 0 -&amp;gt; lower threshold search\n    # idx == 1 -&amp;gt; higher threshold search\n    golden1 = 0.618\n    golden2 = 1 - golden1\n    for _ in range(10):\n        a, b = self.ab_start[idx]\n        # calc losses\n        self.threshold[idx] = a\n        la = -self.score(train_labels, train_preds)\n        self.threshold[idx] = b\n        lb = -self.score(train_labels, train_preds)\n        for _ in range(20):\n            # choose value\n            if la &amp;gt; lb:\n                a = b - (b - a) * golden1\n                self.threshold[idx] = a\n                la = -self.score(train_labels, train_preds)\n            else:\n                b = b - (b - a) * golden2\n                self.threshold[idx] = b\n                lb = -self.score(train_labels, train_preds)\n\ndef transform(self, preds):\n    transformed = np.clip(preds, *self.threshold)\n    if np.unique(transformed).size == 1:\n        return preds\n    return transformed\n\ndef score(self, labels, preds):\n    p = self.transform(preds)\n    score = scipy.stats.spearmanr(labels, p)[0]\n    return score\n</code></pre>\n\n<p>```</p>\n\n<h2>Blending strategy</h2>\n\n<p>I used TPE optimization, same method as <a href=\"https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/97471#562472\">previous solution</a>.</p>\n\n<p>Blending weights:\n<code>\ntest_preds = lstm_test_preds * 0.1145 + \\\n             bert_uncased_test_preds * 0.0112 + \\\n             bert_uncased_2_test_preds * 0.0911 + \\\n             bert_cased_test_preds * 0.0446 + \\\n             bert_large_uncased_test_preds * 0.1670 + \\\n             bert_large_uncased_2_test_preds * 0.0487 + \\\n             bert_large_cased_test_preds * 0.1094 + \\\n             bert_large_cased_2_test_preds * 0.1369 + \\\n             gpt2_test_preds * 0.0478 + \\\n             albert_test_preds * 0.0120 + \\\n             xlnet_test_preds * 0.1607 + \\\n             roberta_test_preds * 0.0560\n</code></p>\n\n<h2>What didn't work for me</h2>\n\n<ul>\n<li>Pre-training\n<ul><li>masked LM</li>\n<li>input-response prediction (<a href=\"https://arxiv.org/abs/1705.00652\"></a><a href=\"https://arxiv.org/abs/1705.00652\">https://arxiv.org/abs/1705.00652</a>)</li></ul></li>\n<li>Data augmentation\n<ul><li>inverse translation (<a href=\"https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038\"></a><a href=\"https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038\">https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038</a>)</li>\n<li>manifold mixup</li>\n<li>up-sample minor labels</li></ul></li>\n</ul>",
  "comments": [
    {
      "id": 751996,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-20T17:41:53.350000",
      "content": "<p>Nice Kernel Great approach you deserve no 1 spot</p>",
      "votes": 15,
      "replies": []
    },
    {
      "id": 742949,
      "author_name": "CPMP",
      "author_url": "",
      "post_date": "2020-02-11T16:56:08.803000",
      "content": "<p>Congrats for amazing solo gold. Your solution seems quite similar to ours (no surprise). You use a lot of models, did you have to do anything special to make sure your inference kernel run fast enough?</p>",
      "votes": 5,
      "replies": [
        {
          "id": 743405,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2020-02-12T03:24:37.913000",
          "content": "<p>Thank you! You too, congrats CPMP.\nI didn't do anything special, I just used sequence bucketing and larger batch size than training.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 742512,
      "author_name": "bubusu",
      "author_url": "",
      "post_date": "2020-02-11T10:34:18.063000",
      "content": "<p>Sakami is God.</p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 742498,
      "author_name": "Khoi Nguyen",
      "author_url": "",
      "post_date": "2020-02-11T10:25:19.110000",
      "content": "<p>Congrats, I'm glad that we made the right decision to not team up earlier :D</p>",
      "votes": 4,
      "replies": [
        {
          "id": 742634,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2020-02-11T12:37:46.357000",
          "content": "<p>Yes, we achieved the best case: both of us got a solo gold medal. :)\nCongrats, Khoi!!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 743411,
          "author_name": "Khoi Nguyen",
          "author_url": "",
          "post_date": "2020-02-12T03:32:59.743000",
          "content": "<p>And some extra money to pay for the electricity bill too, even better!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 749587,
      "author_name": "Imesh Ekanayake",
      "author_url": "",
      "post_date": "2020-02-18T19:46:15.540000",
      "content": "<p>Nice Work</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 749116,
      "author_name": "Ericmututu",
      "author_url": "",
      "post_date": "2020-02-18T10:12:41.783000",
      "content": "<p>Wonderful job ! 👍   It's very useful for me, may I get your complete codes? If wish, thanks so much !!!🙏 </p>",
      "votes": 1,
      "replies": [
        {
          "id": 750499,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2020-02-19T12:52:54.947000",
          "content": "<p>I'll share my code later. ;)</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 750508,
          "author_name": "Abhishek Thakur",
          "author_url": "",
          "post_date": "2020-02-19T13:01:27.470000",
          "content": "<p>Thank you <a href=\"/sakami\">@sakami</a> ! There is always so much to learn from the code you share! :)</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 748976,
      "author_name": "KagroLin",
      "author_url": "",
      "post_date": "2020-02-18T06:02:14.620000",
      "content": "<p>Elegant work！！ Just clip preds !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 746453,
      "author_name": "Raj Saha",
      "author_url": "",
      "post_date": "2020-02-15T02:55:02.583000",
      "content": "<p>Nice</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 745242,
      "author_name": "Muhammet Ikbal Elek",
      "author_url": "",
      "post_date": "2020-02-13T16:38:31.253000",
      "content": "<p>Congrats!!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 745200,
      "author_name": "kenmatsu4",
      "author_url": "",
      "post_date": "2020-02-13T15:55:26.190000",
      "content": "<p>Thank you for sharing your great solution!\nAnd I have a question for first one of BERT model.</p>\n\n<p>question-avg and answer-avg are used from sequence_output, then how to split to question part and answer part? Dynamically changing with each scentence? Or constant number for separation?</p>\n\n<p>I would appreciate it if you could explain to my question! Thanks!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 745609,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2020-02-14T01:45:51.970000",
          "content": "<p>Thank you!\nI dynamically changed with each sentence, using <code>attention_mask</code> and <code>token_type_ids</code>.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 745828,
          "author_name": "kenmatsu4",
          "author_url": "",
          "post_date": "2020-02-14T08:53:05.030000",
          "content": "<p>That's nice! Thank you for your answer!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 744737,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2020-02-13T04:55:11.147000",
      "content": "<p>Congrats and thanks for sharing amazing solution! <a href=\"/sakami\">@sakami</a> ^ ^</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 744550,
      "author_name": "DimitreOliveira",
      "author_url": "",
      "post_date": "2020-02-12T23:30:25.377000",
      "content": "<p>Congratulation, clearly you made a lot of experiments and the result is very well deserved, I'm also surprised by how good your LSTM performed, really awesome.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 744530,
      "author_name": "corochann",
      "author_url": "",
      "post_date": "2020-02-12T23:01:39.547000",
      "content": "<p>Congrats! and thank you for sharing detailed write up.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 744449,
      "author_name": "James Ingram",
      "author_url": "",
      "post_date": "2020-02-12T20:52:16.127000",
      "content": "<p>おめでとうございます <a href=\"/sakami\">@sakami</a> \nNLPの専門知識とエレガントなソリューションを共有していただきありがとうございます。</p>\n\n<p>Congrats!\nThank you for sharing your NLP expertise and elegant solutions.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 744648,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2020-02-13T02:43:24.707000",
          "content": "<p>ありがとうございます！ </p>\n\n<p>Thank you! :)</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 744156,
      "author_name": "YaGana Sheriff-Hussaini",
      "author_url": "",
      "post_date": "2020-02-12T16:08:26.163000",
      "content": "<p>Congrats <a href=\"/sakami\">@sakami</a> and thanks so much for sharing your solution. This solution is definately one for the books. Well done.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 743803,
      "author_name": "Ichimaru Gin",
      "author_url": "",
      "post_date": "2020-02-12T10:19:09.230000",
      "content": "<p>Thank you for the wonderful illustrations. They really help a lot to understand the idea. What tool did you use for making the diagrams?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 743908,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2020-02-12T11:36:00.463000",
          "content": "<p>I used google slides. :)</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 743759,
      "author_name": "stout",
      "author_url": "",
      "post_date": "2020-02-12T09:20:31.113000",
      "content": "<p>Just a piece of art! Thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 743629,
      "author_name": "DtneSEffct",
      "author_url": "",
      "post_date": "2020-02-12T06:36:41.450000",
      "content": "<p>Congratulations <a href=\"/sakami\">@sakami</a> !!!!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742933,
      "author_name": "Miyabon",
      "author_url": "",
      "post_date": "2020-02-11T16:42:20.630000",
      "content": "<p>Congrats &amp; Thanks for sharing <a href=\"/sakami\">@sakami</a> !!🎉 😄 👍 </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742862,
      "author_name": "atfujita",
      "author_url": "",
      "post_date": "2020-02-11T15:50:29.550000",
      "content": "<p>Congrats!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742686,
      "author_name": "Mark Peng",
      "author_url": "",
      "post_date": "2020-02-11T13:40:12.147000",
      "content": "<p><a href=\"/sakami\">@sakami</a> Amazing work!\nOne question: how did you make it to run 12 models in the GPU time limit of 2 hours? Did you use less than 5 K-folds for each model? I tried to use larger batch size for inference, but for models like XLNet, it took ~2.3 minutes per fold. The max. number of models I could run in a kernel is 5.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 743424,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2020-02-12T03:41:24.063000",
          "content": "<p>I used 5-folds for every model.\nHmm... I didn't do anything special, I just used sequence bucketing.\nMy XLNet-base (max length: 1024, batch size: 8) spends 26.5 seconds per fold.</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 742681,
      "author_name": "Camaro",
      "author_url": "",
      "post_date": "2020-02-11T13:32:49.640000",
      "content": "<p>Congrats! What a beautiful solution!!!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742655,
      "author_name": "RB",
      "author_url": "",
      "post_date": "2020-02-11T12:50:44.177000",
      "content": "<p>Congratulations on solo gold!  </p>\n\n<p>Can you elaborate on how you used optuna for blending ? \nDid you use the optimized rounder for every model in ensemble or just once after blending ? </p>\n\n<p>Thank you for sharing your approach. </p>",
      "votes": 1,
      "replies": [
        {
          "id": 743410,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2020-02-12T03:30:08.700000",
          "content": "<p>Thank you!</p>\n\n<p>My blending method is almost same as <a href=\"https://github.com/sakami0000/kaggle_jigsaw/blob/master/compute_blending_weights.py\">this</a>.\nThe difference is that I didn't split OOF prediction this time.</p>\n\n<p>I used optimized rounder just once after blending.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 742649,
      "author_name": "Ailurophile",
      "author_url": "",
      "post_date": "2020-02-11T12:47:39.550000",
      "content": "<p>Congratulations!\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/sakami\">@sakami</a> </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742603,
      "author_name": "sandy1112",
      "author_url": "",
      "post_date": "2020-02-11T12:06:27.483000",
      "content": "<p>thanks a ton for sharing the winning solution in such great details. I learnt a lot reading through the solution and this will remain one of the biggest takeaways for me in this competition.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742562,
      "author_name": "Nirjhar Roy",
      "author_url": "",
      "post_date": "2020-02-11T11:30:32.770000",
      "content": "<p>such elegant solution . I have benefited a lot from your previous codes (as much as i understood ) . I remember putting a tribute to your name as well in my kernel somewhere . Thank you .</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742550,
      "author_name": "chizhu",
      "author_url": "",
      "post_date": "2020-02-11T11:19:59.333000",
      "content": "<p>wow! Congrats!  what a clear and detailed solution! Thank you for sharing</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742474,
      "author_name": "ask9",
      "author_url": "",
      "post_date": "2020-02-11T10:08:34.840000",
      "content": "<p>well done guys ,Thanks for the solution explainations!!!!!!!!!!!!!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742470,
      "author_name": "ilovescience",
      "author_url": "",
      "post_date": "2020-02-11T10:06:20.867000",
      "content": "<p>Congrats on winning the prize and thanks for sharing your solution! </p>\n\n<p>Did you use PyTorch or TF? Any plans to open-source some, if not all the code?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 742635,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2020-02-11T12:38:25.507000",
          "content": "<p>I used PyTorch. I'll share my code after cleaning it. :)</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 752624,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-21T08:21:16.223000",
      "content": "",
      "votes": 2,
      "replies": []
    },
    {
      "id": 743452,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-12T04:25:25.557000",
      "content": "",
      "votes": 2,
      "replies": []
    },
    {
      "id": 743406,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-12T03:26:17.170000",
      "content": "",
      "votes": 2,
      "replies": [
        {
          "id": 743733,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-12T08:42:45.370000",
          "content": "",
          "votes": 7,
          "replies": []
        },
        {
          "id": 744188,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-12T16:39:28.757000",
          "content": "",
          "votes": 1,
          "replies": []
        },
        {
          "id": 744511,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-12T22:32:00.667000",
          "content": "",
          "votes": 1,
          "replies": []
        },
        {
          "id": 744646,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-13T02:41:08.430000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 742552,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-11T11:20:48.533000",
      "content": "",
      "votes": 2,
      "replies": []
    },
    {
      "id": 742546,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-11T11:18:04.630000",
      "content": "",
      "votes": 2,
      "replies": [
        {
          "id": 742639,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-11T12:39:56.720000",
          "content": "",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 742482,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-11T10:11:31.743000",
      "content": "",
      "votes": 2,
      "replies": []
    },
    {
      "id": 743001,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-11T17:40:28.163000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 743323,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-12T01:42:41",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 780091,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-03-20T01:03:34.427000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 752985,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-21T15:41:15.023000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 752090,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-20T18:36:52.253000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 751133,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-20T02:38:33.500000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 742832,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-11T15:11:04.983000",
      "content": "",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742558,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-11T11:27:56.853000",
      "content": "",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 60,
  "messages": [
    {
      "id": 742467,
      "postDate": "2020-02-11T10:03:27.563Z",
      "content": "<p>First of all, I'd like to say thank you to the organizers and all teams, and congrats to all the winners.\nHere is a brief summary of my solution.\n- Solution overview\n- Model architectures\n- Training strategy\n- Post-processing\n- Blending strategy\n- What didn't work for me</p>\n\n<h2>Solution overview</h2>\n\n<ul>\n<li>text columns + category column</li>\n<li>various text truncation\n<ul><li>pre-truncate</li>\n<li>post-truncate</li>\n<li>head + tail tokens (<a href=\"https://arxiv.org/abs/1905.05583\"></a><a href=\"https://arxiv.org/abs/1905.05583\">https://arxiv.org/abs/1905.05583</a>)</li>\n<li>assign longer max_len to answer</li></ul></li>\n<li>ensemble of following models:\n<ul><li>LSTM + Universal Sentence Encoder</li>\n<li>BERT base uncased * 2</li>\n<li>BERT base cased</li>\n<li>BERT large uncased * 2</li>\n<li>BERT large cased * 2</li>\n<li>ALBERT base</li>\n<li>RoBERTa base</li>\n<li>GPT2 base</li>\n<li>XLNet base</li></ul></li>\n</ul>\n\n<h2>Model architectures</h2>\n\n<p>Surprisingly, stacking 2 Linear layers <strong>without</strong> activation performed better than single Linear layer in some cases. (I found it by a mistake. :P)\nSome of the following models have category embedding structure additionally.</p>\n\n<h3>LSTM model</h3>\n\n<p>You can check my LSTM code <a href=\"https://www.kaggle.com/sakami/google-quest-single-lstm?scriptVersionId=28487242\">here</a>.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F5ba7064f215c31d377bb991275d5fa5b%2F2020-02-11%2017.29.36.png?generation=1581409799179315&amp;alt=media\" alt=\"\"></p>\n\n<h3>BERT models</h3>\n\n<p>I used 2 types of model architectures.</p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Ff5da89f1cff533e174ae24d1cc459da2%2F2020-02-11%2017.14.05.png?generation=1581409945426377&amp;alt=media\" alt=\"\"></p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F80a15ecbc3f970aa7fad8308d01f2af1%2F2020-02-11%2017.14.17.png?generation=1581409964611182&amp;alt=media\" alt=\"\"></p>\n\n<h3>ALBERT model</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F506270dad509f130a53c50d177bd4861%2F2020-02-11%2020.12.38.png?generation=1581419591299259&amp;alt=media\" alt=\"\"></p>\n\n<h3>RoBERTa model</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Fd877c80ce341c7be39715eb3534627d9%2F2020-02-11%2017.14.26.png?generation=1581410441581392&amp;alt=media\" alt=\"\"></p>\n\n<h3>GPT2 model</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Ff123dc23ada1495c68b5be43ccf0241a%2F2020-02-11%2017.14.44.png?generation=1581410465561097&amp;alt=media\" alt=\"\"></p>\n\n<h3>XLNet model</h3>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F483691fcbd3e72cb5a6053e5c8d4873a%2F2020-02-11%2017.14.55.png?generation=1581410490556290&amp;alt=media\" alt=\"\"></p>\n\n<h2>Training strategy</h2>\n\n<ul>\n<li>Min-Max target scaling\n<code>\nt_max = train_y.max(axis=0)[np.newaxis, :]\nt_min = train_y.min(axis=0)[np.newaxis, :]\ntrain_y = (train_y - t_min) / (t_max - t_min)\n</code></li>\n<li>Large weights for minor positive or negative samples\n<code>\nzero_inflated = (train_y &amp;gt; 0).mean(axis=0) &amp;lt; 0.1\npositive_weighted = np.tile(zero_inflated, (len(train_y), 1))\npositive_weighted *= (train_y &amp;gt; 0)\none_inflated = (train_y &amp;lt; 1).mean(axis=0) &amp;lt; 0.1\nnegative_weighted = np.tile(one_inflated, (len(train_y), 1))\nnegative_weighted *= (train_y &amp;lt; 1)\ntrain_weights = np.where(positive_weighted + negative_weighted, 2., 1.)\n</code></li>\n<li>gelu_new activation for BERT-based models</li>\n<li>cosine warmup scheduler</li>\n<li>EMA</li>\n</ul>\n\n<h2>Post-processing</h2>\n\n<p>I just clipped predictions. The thresholds are decided by golden section search.\nThis improve my score from 0.4434 to 0.4852. (CV score)</p>\n\n<p>before post-processing scores:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Ff8ebe510f4885db659da10499a79daea%2Fnewplot.png?generation=1581416684489031&amp;alt=media\" alt=\"\"></p>\n\n<p>after post-processing scores:\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F04f51a001c9f734fed93003cfe8e935b%2Fnewplot%20(1\" alt=\"\">.png?generation=1581416716829926&amp;alt=media)</p>\n\n<p>```\nclass OptimizedRounder(BaseEstimator, TransformerMixin):</p>\n\n<pre><code>def __init__(self):\n    self.threshold = [0., 1.]\n    self.ab_start = [(0., 0.2), (0.8, 1.)]\n\ndef fit(self, train_labels, train_preds):\n    assert train_labels.shape == train_preds.shape\n    assert train_labels.ndim == 1\n\n    self.best_score = self.score(train_labels, train_preds)\n    self._golden_section_search(train_labels, train_preds, 0)  # lower threshold\n    score = self.score(train_labels, train_preds)\n    if score &amp;gt; self.best_score + 1e-3:\n        self.best_score = score\n    else:\n        self.threshold[0] = 0.\n\n    self._golden_section_search(train_labels, train_preds, 1)  # higher threshold\n    score = self.score(train_labels, train_preds)\n    if score &amp;gt; self.best_score + 1e-3:\n        self.best_score = score\n    else:\n        self.threshold[1] = 1.\n\ndef _golden_section_search(self, train_labels, train_preds, idx):\n    # idx == 0 -&amp;gt; lower threshold search\n    # idx == 1 -&amp;gt; higher threshold search\n    golden1 = 0.618\n    golden2 = 1 - golden1\n    for _ in range(10):\n        a, b = self.ab_start[idx]\n        # calc losses\n        self.threshold[idx] = a\n        la = -self.score(train_labels, train_preds)\n        self.threshold[idx] = b\n        lb = -self.score(train_labels, train_preds)\n        for _ in range(20):\n            # choose value\n            if la &amp;gt; lb:\n                a = b - (b - a) * golden1\n                self.threshold[idx] = a\n                la = -self.score(train_labels, train_preds)\n            else:\n                b = b - (b - a) * golden2\n                self.threshold[idx] = b\n                lb = -self.score(train_labels, train_preds)\n\ndef transform(self, preds):\n    transformed = np.clip(preds, *self.threshold)\n    if np.unique(transformed).size == 1:\n        return preds\n    return transformed\n\ndef score(self, labels, preds):\n    p = self.transform(preds)\n    score = scipy.stats.spearmanr(labels, p)[0]\n    return score\n</code></pre>\n\n<p>```</p>\n\n<h2>Blending strategy</h2>\n\n<p>I used TPE optimization, same method as <a href=\"https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/97471#562472\">previous solution</a>.</p>\n\n<p>Blending weights:\n<code>\ntest_preds = lstm_test_preds * 0.1145 + \\\n             bert_uncased_test_preds * 0.0112 + \\\n             bert_uncased_2_test_preds * 0.0911 + \\\n             bert_cased_test_preds * 0.0446 + \\\n             bert_large_uncased_test_preds * 0.1670 + \\\n             bert_large_uncased_2_test_preds * 0.0487 + \\\n             bert_large_cased_test_preds * 0.1094 + \\\n             bert_large_cased_2_test_preds * 0.1369 + \\\n             gpt2_test_preds * 0.0478 + \\\n             albert_test_preds * 0.0120 + \\\n             xlnet_test_preds * 0.1607 + \\\n             roberta_test_preds * 0.0560\n</code></p>\n\n<h2>What didn't work for me</h2>\n\n<ul>\n<li>Pre-training\n<ul><li>masked LM</li>\n<li>input-response prediction (<a href=\"https://arxiv.org/abs/1705.00652\"></a><a href=\"https://arxiv.org/abs/1705.00652\">https://arxiv.org/abs/1705.00652</a>)</li></ul></li>\n<li>Data augmentation\n<ul><li>inverse translation (<a href=\"https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038\"></a><a href=\"https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038\">https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038</a>)</li>\n<li>manifold mixup</li>\n<li>up-sample minor labels</li></ul></li>\n</ul>",
      "rawMarkdown": "First of all, I'd like to say thank you to the organizers and all teams, and congrats to all the winners.\nHere is a brief summary of my solution.\n- Solution overview\n- Model architectures\n- Training strategy\n- Post-processing\n- Blending strategy\n- What didn't work for me\n\n## Solution overview\n- text columns + category column\n- various text truncation\n  - pre-truncate\n  - post-truncate\n  - head + tail tokens (https://arxiv.org/abs/1905.05583)\n  - assign longer max_len to answer\n- ensemble of following models:\n  - LSTM + Universal Sentence Encoder\n  - BERT base uncased * 2\n  - BERT base cased\n  - BERT large uncased * 2\n  - BERT large cased * 2\n  - ALBERT base\n  - RoBERTa base\n  - GPT2 base\n  - XLNet base\n\n## Model architectures\nSurprisingly, stacking 2 Linear layers **without** activation performed better than single Linear layer in some cases. (I found it by a mistake. :P)\nSome of the following models have category embedding structure additionally.\n\n### LSTM model\nYou can check my LSTM code [here](https://www.kaggle.com/sakami/google-quest-single-lstm?scriptVersionId=28487242).\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F5ba7064f215c31d377bb991275d5fa5b%2F2020-02-11%2017.29.36.png?generation=1581409799179315&amp;alt=media)\n\n### BERT models\nI used 2 types of model architectures.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Ff5da89f1cff533e174ae24d1cc459da2%2F2020-02-11%2017.14.05.png?generation=1581409945426377&amp;alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F80a15ecbc3f970aa7fad8308d01f2af1%2F2020-02-11%2017.14.17.png?generation=1581409964611182&amp;alt=media)\n\n### ALBERT model\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F506270dad509f130a53c50d177bd4861%2F2020-02-11%2020.12.38.png?generation=1581419591299259&amp;alt=media)\n\n### RoBERTa model\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Fd877c80ce341c7be39715eb3534627d9%2F2020-02-11%2017.14.26.png?generation=1581410441581392&amp;alt=media)\n\n### GPT2 model\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Ff123dc23ada1495c68b5be43ccf0241a%2F2020-02-11%2017.14.44.png?generation=1581410465561097&amp;alt=media)\n\n### XLNet model\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F483691fcbd3e72cb5a6053e5c8d4873a%2F2020-02-11%2017.14.55.png?generation=1581410490556290&amp;alt=media)\n\n## Training strategy\n- Min-Max target scaling\n```\n    t_max = train_y.max(axis=0)[np.newaxis, :]\n    t_min = train_y.min(axis=0)[np.newaxis, :]\n    train_y = (train_y - t_min) / (t_max - t_min)\n```\n- Large weights for minor positive or negative samples\n```\n    zero_inflated = (train_y &gt; 0).mean(axis=0) &lt; 0.1\n    positive_weighted = np.tile(zero_inflated, (len(train_y), 1))\n    positive_weighted *= (train_y &gt; 0)\n    one_inflated = (train_y &lt; 1).mean(axis=0) &lt; 0.1\n    negative_weighted = np.tile(one_inflated, (len(train_y), 1))\n    negative_weighted *= (train_y &lt; 1)\n    train_weights = np.where(positive_weighted + negative_weighted, 2., 1.)\n```\n- gelu_new activation for BERT-based models\n- cosine warmup scheduler\n- EMA\n\n## Post-processing\nI just clipped predictions. The thresholds are decided by golden section search.\nThis improve my score from 0.4434 to 0.4852. (CV score)\n\nbefore post-processing scores:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2Ff8ebe510f4885db659da10499a79daea%2Fnewplot.png?generation=1581416684489031&amp;alt=media)\n\nafter post-processing scores:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2133818%2F04f51a001c9f734fed93003cfe8e935b%2Fnewplot%20(1).png?generation=1581416716829926&amp;alt=media)\n\n\n```\nclass OptimizedRounder(BaseEstimator, TransformerMixin):\n    \n    def __init__(self):\n        self.threshold = [0., 1.]\n        self.ab_start = [(0., 0.2), (0.8, 1.)]\n    \n    def fit(self, train_labels, train_preds):\n        assert train_labels.shape == train_preds.shape\n        assert train_labels.ndim == 1\n        \n        self.best_score = self.score(train_labels, train_preds)\n        self._golden_section_search(train_labels, train_preds, 0)  # lower threshold\n        score = self.score(train_labels, train_preds)\n        if score &gt; self.best_score + 1e-3:\n            self.best_score = score\n        else:\n            self.threshold[0] = 0.\n        \n        self._golden_section_search(train_labels, train_preds, 1)  # higher threshold\n        score = self.score(train_labels, train_preds)\n        if score &gt; self.best_score + 1e-3:\n            self.best_score = score\n        else:\n            self.threshold[1] = 1.\n\n    def _golden_section_search(self, train_labels, train_preds, idx):\n        # idx == 0 -&gt; lower threshold search\n        # idx == 1 -&gt; higher threshold search\n        golden1 = 0.618\n        golden2 = 1 - golden1\n        for _ in range(10):\n            a, b = self.ab_start[idx]\n            # calc losses\n            self.threshold[idx] = a\n            la = -self.score(train_labels, train_preds)\n            self.threshold[idx] = b\n            lb = -self.score(train_labels, train_preds)\n            for _ in range(20):\n                # choose value\n                if la &gt; lb:\n                    a = b - (b - a) * golden1\n                    self.threshold[idx] = a\n                    la = -self.score(train_labels, train_preds)\n                else:\n                    b = b - (b - a) * golden2\n                    self.threshold[idx] = b\n                    lb = -self.score(train_labels, train_preds)\n\n    def transform(self, preds):\n        transformed = np.clip(preds, *self.threshold)\n        if np.unique(transformed).size == 1:\n            return preds\n        return transformed\n        \n    def score(self, labels, preds):\n        p = self.transform(preds)\n        score = scipy.stats.spearmanr(labels, p)[0]\n        return score\n```\n\n## Blending strategy\nI used TPE optimization, same method as [previous solution](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/97471#562472).\n\nBlending weights:\n```\ntest_preds = lstm_test_preds * 0.1145 + \\\n             bert_uncased_test_preds * 0.0112 + \\\n             bert_uncased_2_test_preds * 0.0911 + \\\n             bert_cased_test_preds * 0.0446 + \\\n             bert_large_uncased_test_preds * 0.1670 + \\\n             bert_large_uncased_2_test_preds * 0.0487 + \\\n             bert_large_cased_test_preds * 0.1094 + \\\n             bert_large_cased_2_test_preds * 0.1369 + \\\n             gpt2_test_preds * 0.0478 + \\\n             albert_test_preds * 0.0120 + \\\n             xlnet_test_preds * 0.1607 + \\\n             roberta_test_preds * 0.0560\n```\n\n## What didn't work for me\n- Pre-training\n  - masked LM\n  - input-response prediction (https://arxiv.org/abs/1705.00652)\n- Data augmentation\n  - inverse translation (https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038)\n  - manifold mixup\n  - up-sample minor labels",
      "votes": 143
    },
    {
      "id": 751996,
      "postDate": "2020-02-20T17:41:53.350Z",
      "content": "<p>Nice Kernel Great approach you deserve no 1 spot</p>",
      "rawMarkdown": "Nice Kernel Great approach you deserve no 1 spot",
      "votes": 15
    },
    {
      "id": 742949,
      "postDate": "2020-02-11T16:56:08.803Z",
      "content": "<p>Congrats for amazing solo gold. Your solution seems quite similar to ours (no surprise). You use a lot of models, did you have to do anything special to make sure your inference kernel run fast enough?</p>",
      "rawMarkdown": "Congrats for amazing solo gold. Your solution seems quite similar to ours (no surprise). You use a lot of models, did you have to do anything special to make sure your inference kernel run fast enough?",
      "votes": 5,
      "replies": [
        {
          "id": 743405,
          "postDate": "2020-02-12T03:24:37.913Z",
          "content": "<p>Thank you! You too, congrats CPMP.\nI didn't do anything special, I just used sequence bucketing and larger batch size than training.</p>",
          "rawMarkdown": "Thank you! You too, congrats CPMP.\nI didn't do anything special, I just used sequence bucketing and larger batch size than training.",
          "votes": 2
        }
      ]
    },
    {
      "id": 742512,
      "postDate": "2020-02-11T10:34:18.063Z",
      "content": "<p>Sakami is God.</p>",
      "rawMarkdown": "Sakami is God.",
      "votes": 4
    },
    {
      "id": 742498,
      "postDate": "2020-02-11T10:25:19.110Z",
      "content": "<p>Congrats, I'm glad that we made the right decision to not team up earlier :D</p>",
      "rawMarkdown": "Congrats, I'm glad that we made the right decision to not team up earlier :D",
      "votes": 4,
      "replies": [
        {
          "id": 742634,
          "postDate": "2020-02-11T12:37:46.357Z",
          "content": "<p>Yes, we achieved the best case: both of us got a solo gold medal. :)\nCongrats, Khoi!!</p>",
          "rawMarkdown": "Yes, we achieved the best case: both of us got a solo gold medal. :)\nCongrats, Khoi!!"
        },
        {
          "id": 743411,
          "postDate": "2020-02-12T03:32:59.743Z",
          "content": "<p>And some extra money to pay for the electricity bill too, even better!</p>",
          "rawMarkdown": "And some extra money to pay for the electricity bill too, even better!",
          "votes": 1
        }
      ]
    },
    {
      "id": 749587,
      "postDate": "2020-02-18T19:46:15.540Z",
      "content": "<p>Nice Work</p>",
      "rawMarkdown": "Nice Work",
      "votes": 1
    },
    {
      "id": 749116,
      "postDate": "2020-02-18T10:12:41.783Z",
      "content": "<p>Wonderful job ! 👍   It's very useful for me, may I get your complete codes? If wish, thanks so much !!!🙏 </p>",
      "rawMarkdown": "Wonderful job ! 👍   It's very useful for me, may I get your complete codes? If wish, thanks so much !!!🙏 ",
      "votes": 1,
      "replies": [
        {
          "id": 750499,
          "postDate": "2020-02-19T12:52:54.947Z",
          "content": "<p>I'll share my code later. ;)</p>",
          "rawMarkdown": "I'll share my code later. ;)",
          "votes": 1
        },
        {
          "id": 750508,
          "postDate": "2020-02-19T13:01:27.470Z",
          "content": "<p>Thank you <a href=\"/sakami\">@sakami</a> ! There is always so much to learn from the code you share! :)</p>",
          "rawMarkdown": "Thank you @sakami ! There is always so much to learn from the code you share! :)",
          "votes": 1
        }
      ]
    },
    {
      "id": 748976,
      "postDate": "2020-02-18T06:02:14.620Z",
      "content": "<p>Elegant work！！ Just clip preds !</p>",
      "rawMarkdown": "Elegant work！！ Just clip preds !",
      "votes": 1
    },
    {
      "id": 746453,
      "postDate": "2020-02-15T02:55:02.583Z",
      "content": "<p>Nice</p>",
      "rawMarkdown": "Nice",
      "votes": 1
    },
    {
      "id": 745242,
      "postDate": "2020-02-13T16:38:31.253Z",
      "content": "<p>Congrats!!</p>",
      "rawMarkdown": "Congrats!!",
      "votes": 1
    },
    {
      "id": 745200,
      "postDate": "2020-02-13T15:55:26.190Z",
      "content": "<p>Thank you for sharing your great solution!\nAnd I have a question for first one of BERT model.</p>\n\n<p>question-avg and answer-avg are used from sequence_output, then how to split to question part and answer part? Dynamically changing with each scentence? Or constant number for separation?</p>\n\n<p>I would appreciate it if you could explain to my question! Thanks!</p>",
      "rawMarkdown": "Thank you for sharing your great solution!\nAnd I have a question for first one of BERT model.\n\nquestion-avg and answer-avg are used from sequence\\_output, then how to split to question part and answer part? Dynamically changing with each scentence? Or constant number for separation?\n\nI would appreciate it if you could explain to my question! Thanks!",
      "votes": 1,
      "replies": [
        {
          "id": 745609,
          "postDate": "2020-02-14T01:45:51.970Z",
          "content": "<p>Thank you!\nI dynamically changed with each sentence, using <code>attention_mask</code> and <code>token_type_ids</code>.</p>",
          "rawMarkdown": "Thank you!\nI dynamically changed with each sentence, using `attention_mask` and `token_type_ids`.",
          "votes": 1
        },
        {
          "id": 745828,
          "postDate": "2020-02-14T08:53:05.030Z",
          "content": "<p>That's nice! Thank you for your answer!</p>",
          "rawMarkdown": "That's nice! Thank you for your answer!",
          "votes": 1
        }
      ]
    },
    {
      "id": 744737,
      "postDate": "2020-02-13T04:55:11.147Z",
      "content": "<p>Congrats and thanks for sharing amazing solution! <a href=\"/sakami\">@sakami</a> ^ ^</p>",
      "rawMarkdown": "Congrats and thanks for sharing amazing solution! @sakami ^ ^",
      "votes": 1
    },
    {
      "id": 744550,
      "postDate": "2020-02-12T23:30:25.377Z",
      "content": "<p>Congratulation, clearly you made a lot of experiments and the result is very well deserved, I'm also surprised by how good your LSTM performed, really awesome.</p>",
      "rawMarkdown": "Congratulation, clearly you made a lot of experiments and the result is very well deserved, I'm also surprised by how good your LSTM performed, really awesome.",
      "votes": 1
    },
    {
      "id": 744530,
      "postDate": "2020-02-12T23:01:39.547Z",
      "content": "<p>Congrats! and thank you for sharing detailed write up.</p>",
      "rawMarkdown": "Congrats! and thank you for sharing detailed write up.",
      "votes": 1
    },
    {
      "id": 744449,
      "postDate": "2020-02-12T20:52:16.127Z",
      "content": "<p>おめでとうございます <a href=\"/sakami\">@sakami</a> \nNLPの専門知識とエレガントなソリューションを共有していただきありがとうございます。</p>\n\n<p>Congrats!\nThank you for sharing your NLP expertise and elegant solutions.</p>",
      "rawMarkdown": "おめでとうございます @sakami \nNLPの専門知識とエレガントなソリューションを共有していただきありがとうございます。\n\nCongrats!\nThank you for sharing your NLP expertise and elegant solutions.",
      "votes": 1,
      "replies": [
        {
          "id": 744648,
          "postDate": "2020-02-13T02:43:24.707Z",
          "content": "<p>ありがとうございます！ </p>\n\n<p>Thank you! :)</p>",
          "rawMarkdown": "ありがとうございます！ \n\nThank you! :)",
          "votes": 1
        }
      ]
    },
    {
      "id": 744156,
      "postDate": "2020-02-12T16:08:26.163Z",
      "content": "<p>Congrats <a href=\"/sakami\">@sakami</a> and thanks so much for sharing your solution. This solution is definately one for the books. Well done.</p>",
      "rawMarkdown": "Congrats @sakami and thanks so much for sharing your solution. This solution is definately one for the books. Well done.",
      "votes": 1
    },
    {
      "id": 743803,
      "postDate": "2020-02-12T10:19:09.230Z",
      "content": "<p>Thank you for the wonderful illustrations. They really help a lot to understand the idea. What tool did you use for making the diagrams?</p>",
      "rawMarkdown": "Thank you for the wonderful illustrations. They really help a lot to understand the idea. What tool did you use for making the diagrams?",
      "votes": 1,
      "replies": [
        {
          "id": 743908,
          "postDate": "2020-02-12T11:36:00.463Z",
          "content": "<p>I used google slides. :)</p>",
          "rawMarkdown": "I used google slides. :)",
          "votes": 1
        }
      ]
    },
    {
      "id": 743759,
      "postDate": "2020-02-12T09:20:31.113Z",
      "content": "<p>Just a piece of art! Thanks for sharing!</p>",
      "rawMarkdown": "Just a piece of art! Thanks for sharing!",
      "votes": 1
    },
    {
      "id": 743629,
      "postDate": "2020-02-12T06:36:41.450Z",
      "content": "<p>Congratulations <a href=\"/sakami\">@sakami</a> !!!!</p>",
      "rawMarkdown": "Congratulations @sakami !!!!",
      "votes": 1
    },
    {
      "id": 742933,
      "postDate": "2020-02-11T16:42:20.630Z",
      "content": "<p>Congrats &amp; Thanks for sharing <a href=\"/sakami\">@sakami</a> !!🎉 😄 👍 </p>",
      "rawMarkdown": "Congrats &amp; Thanks for sharing @sakami !!🎉 😄 👍 ",
      "votes": 1
    },
    {
      "id": 742862,
      "postDate": "2020-02-11T15:50:29.550Z",
      "content": "<p>Congrats!</p>",
      "rawMarkdown": "Congrats!",
      "votes": 1
    },
    {
      "id": 742686,
      "postDate": "2020-02-11T13:40:12.147Z",
      "content": "<p><a href=\"/sakami\">@sakami</a> Amazing work!\nOne question: how did you make it to run 12 models in the GPU time limit of 2 hours? Did you use less than 5 K-folds for each model? I tried to use larger batch size for inference, but for models like XLNet, it took ~2.3 minutes per fold. The max. number of models I could run in a kernel is 5.</p>",
      "rawMarkdown": "@sakami Amazing work!\nOne question: how did you make it to run 12 models in the GPU time limit of 2 hours? Did you use less than 5 K-folds for each model? I tried to use larger batch size for inference, but for models like XLNet, it took ~2.3 minutes per fold. The max. number of models I could run in a kernel is 5.",
      "votes": 1,
      "replies": [
        {
          "id": 743424,
          "postDate": "2020-02-12T03:41:24.063Z",
          "content": "<p>I used 5-folds for every model.\nHmm... I didn't do anything special, I just used sequence bucketing.\nMy XLNet-base (max length: 1024, batch size: 8) spends 26.5 seconds per fold.</p>",
          "rawMarkdown": "I used 5-folds for every model.\nHmm... I didn't do anything special, I just used sequence bucketing.\nMy XLNet-base (max length: 1024, batch size: 8) spends 26.5 seconds per fold.",
          "votes": 3
        }
      ]
    },
    {
      "id": 742681,
      "postDate": "2020-02-11T13:32:49.640Z",
      "content": "<p>Congrats! What a beautiful solution!!!</p>",
      "rawMarkdown": "Congrats! What a beautiful solution!!!",
      "votes": 1
    },
    {
      "id": 742655,
      "postDate": "2020-02-11T12:50:44.177Z",
      "content": "<p>Congratulations on solo gold!  </p>\n\n<p>Can you elaborate on how you used optuna for blending ? \nDid you use the optimized rounder for every model in ensemble or just once after blending ? </p>\n\n<p>Thank you for sharing your approach. </p>",
      "rawMarkdown": "Congratulations on solo gold!  \n\nCan you elaborate on how you used optuna for blending ? \nDid you use the optimized rounder for every model in ensemble or just once after blending ? \n\nThank you for sharing your approach. ",
      "votes": 1,
      "replies": [
        {
          "id": 743410,
          "postDate": "2020-02-12T03:30:08.700Z",
          "content": "<p>Thank you!</p>\n\n<p>My blending method is almost same as <a href=\"https://github.com/sakami0000/kaggle_jigsaw/blob/master/compute_blending_weights.py\">this</a>.\nThe difference is that I didn't split OOF prediction this time.</p>\n\n<p>I used optimized rounder just once after blending.</p>",
          "rawMarkdown": "Thank you!\n\nMy blending method is almost same as [this](https://github.com/sakami0000/kaggle_jigsaw/blob/master/compute_blending_weights.py).\nThe difference is that I didn't split OOF prediction this time.\n\nI used optimized rounder just once after blending.",
          "votes": 1
        }
      ]
    },
    {
      "id": 742649,
      "postDate": "2020-02-11T12:47:39.550Z",
      "content": "<p>Congratulations!\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/sakami\">@sakami</a> </p>",
      "rawMarkdown": "Congratulations!\nGreat Write-Up\nThanks for Sharing your Approach &amp; Insights!! @sakami ",
      "votes": 1
    },
    {
      "id": 742603,
      "postDate": "2020-02-11T12:06:27.483Z",
      "content": "<p>thanks a ton for sharing the winning solution in such great details. I learnt a lot reading through the solution and this will remain one of the biggest takeaways for me in this competition.</p>",
      "rawMarkdown": "thanks a ton for sharing the winning solution in such great details. I learnt a lot reading through the solution and this will remain one of the biggest takeaways for me in this competition.",
      "votes": 1
    },
    {
      "id": 742562,
      "postDate": "2020-02-11T11:30:32.770Z",
      "content": "<p>such elegant solution . I have benefited a lot from your previous codes (as much as i understood ) . I remember putting a tribute to your name as well in my kernel somewhere . Thank you .</p>",
      "rawMarkdown": "such elegant solution . I have benefited a lot from your previous codes (as much as i understood ) . I remember putting a tribute to your name as well in my kernel somewhere . Thank you .",
      "votes": 1
    },
    {
      "id": 742550,
      "postDate": "2020-02-11T11:19:59.333Z",
      "content": "<p>wow! Congrats!  what a clear and detailed solution! Thank you for sharing</p>",
      "rawMarkdown": "wow! Congrats!  what a clear and detailed solution! Thank you for sharing",
      "votes": 1
    },
    {
      "id": 742474,
      "postDate": "2020-02-11T10:08:34.840Z",
      "content": "<p>well done guys ,Thanks for the solution explainations!!!!!!!!!!!!!</p>",
      "rawMarkdown": "well done guys ,Thanks for the solution explainations!!!!!!!!!!!!!",
      "votes": 1
    },
    {
      "id": 742470,
      "postDate": "2020-02-11T10:06:20.867Z",
      "content": "<p>Congrats on winning the prize and thanks for sharing your solution! </p>\n\n<p>Did you use PyTorch or TF? Any plans to open-source some, if not all the code?</p>",
      "rawMarkdown": "Congrats on winning the prize and thanks for sharing your solution! \n\nDid you use PyTorch or TF? Any plans to open-source some, if not all the code?",
      "votes": 1,
      "replies": [
        {
          "id": 742635,
          "postDate": "2020-02-11T12:38:25.507Z",
          "content": "<p>I used PyTorch. I'll share my code after cleaning it. :)</p>",
          "rawMarkdown": "I used PyTorch. I'll share my code after cleaning it. :)",
          "votes": 2
        }
      ]
    },
    {
      "id": 752624,
      "postDate": "2020-02-21T08:21:16.223Z",
      "content": "<p>Nice</p>",
      "rawMarkdown": "Nice",
      "votes": 2
    },
    {
      "id": 743452,
      "postDate": "2020-02-12T04:25:25.557Z",
      "content": "<p>This is absolutely amazing. Congratulations. Thanks for sharing. I'm gonna use all these techniques in the next NLP comp.</p>",
      "rawMarkdown": "This is absolutely amazing. Congratulations. Thanks for sharing. I'm gonna use all these techniques in the next NLP comp.",
      "votes": 2
    },
    {
      "id": 743406,
      "postDate": "2020-02-12T03:26:17.170Z",
      "content": "<p>Big congrats on your amazing solo gold!  </p>\n\n<p><code>The thresholds are decided by golden section search</code> Could you explain a bit how golden section search is done and why it is helpful in this particular problem?</p>",
      "rawMarkdown": "Big congrats on your amazing solo gold!  \n\n`The thresholds are decided by golden section search` Could you explain a bit how golden section search is done and why it is helpful in this particular problem?",
      "votes": 2,
      "replies": [
        {
          "id": 743733,
          "postDate": "2020-02-12T08:42:45.370Z",
          "content": "<p>Thank you!</p>\n\n<p>golden section search:\n1. take two points and evaluate scores of each points.\n2. take the inner dividing point at the golden ratio closer to the worse point.\n3. take the better point and the new point, and evaluate scores.\n4. repeat 2 and 3.</p>\n\n<p>It is better to see <a href=\"http://fourier.eng.hmc.edu/e176/lectures/ch3/node3.html\">here</a> for more explanation.</p>\n\n<p>Golden section search assumes convexity of search space, so it is hard to fall to local optima.\nI didn't want to overfit CV nor public LB, so I used weak optimization method.</p>",
          "rawMarkdown": "Thank you!\n\ngolden section search:\n1. take two points and evaluate scores of each points.\n2. take the inner dividing point at the golden ratio closer to the worse point.\n3. take the better point and the new point, and evaluate scores.\n4. repeat 2 and 3.\n\nIt is better to see [here](http://fourier.eng.hmc.edu/e176/lectures/ch3/node3.html) for more explanation.\n\nGolden section search assumes convexity of search space, so it is hard to fall to local optima.\nI didn't want to overfit CV nor public LB, so I used weak optimization method.",
          "votes": 7
        },
        {
          "id": 744188,
          "postDate": "2020-02-12T16:39:28.757Z",
          "content": "<p>Great explanation! Thanks!</p>",
          "rawMarkdown": "Great explanation! Thanks!",
          "votes": 1
        },
        {
          "id": 744511,
          "postDate": "2020-02-12T22:32:00.667Z",
          "content": "<p><a href=\"/sakami\">@sakami</a> Why did you choose to clip predictions?  Did you end up with narrow intervals for predictions?  In our case, discretization works pretty well.</p>",
          "rawMarkdown": "@sakami Why did you choose to clip predictions?  Did you end up with narrow intervals for predictions?  In our case, discretization works pretty well.",
          "votes": 1
        },
        {
          "id": 744646,
          "postDate": "2020-02-13T02:41:08.430Z",
          "content": "<p>&gt; Why did you choose to clip predictions?</p>\n\n<p>I chose clipping because I want to avoid over-post-processing, which leads to unstable scores of CV and LB.</p>\n\n<p>&gt; Did you end up with narrow intervals for predictions?</p>\n\n<p>No, I restricted possible range of thresholds in advance.\n<code>\nself.ab_start = [(0., 0.2), (0.8, 1.)]\n</code></p>",
          "rawMarkdown": "&gt; Why did you choose to clip predictions?\n\nI chose clipping because I want to avoid over-post-processing, which leads to unstable scores of CV and LB.\n\n&gt; Did you end up with narrow intervals for predictions?\n\nNo, I restricted possible range of thresholds in advance.\n```\nself.ab_start = [(0., 0.2), (0.8, 1.)]\n```"
        }
      ]
    },
    {
      "id": 742552,
      "postDate": "2020-02-11T11:20:48.533Z",
      "content": "<p>Congratulations! Excellent solution.</p>",
      "rawMarkdown": "Congratulations! Excellent solution.",
      "votes": 2
    },
    {
      "id": 742546,
      "postDate": "2020-02-11T11:18:04.630Z",
      "content": "<p>From jigsaw, tf2qa all the way here, I learned a lot from sakami sensei !!!👍 </p>",
      "rawMarkdown": "From jigsaw, tf2qa all the way here, I learned a lot from sakami sensei !!!👍 ",
      "votes": 2,
      "replies": [
        {
          "id": 742639,
          "postDate": "2020-02-11T12:39:56.720Z",
          "content": "<p>My pleasure. ;)</p>",
          "rawMarkdown": "My pleasure. ;)",
          "votes": 1
        }
      ]
    },
    {
      "id": 742482,
      "postDate": "2020-02-11T10:11:31.743Z",
      "content": "<p>Beautiful post and diagrams!!! Congratulations!</p>",
      "rawMarkdown": "Beautiful post and diagrams!!! Congratulations!",
      "votes": 2
    },
    {
      "id": 743001,
      "postDate": "2020-02-11T17:40:28.163Z",
      "content": "<p>Congrats, How did you decide on those model weights in blending? Any CV strategies?</p>",
      "rawMarkdown": "Congrats, How did you decide on those model weights in blending? Any CV strategies?",
      "replies": [
        {
          "id": 743323,
          "postDate": "2020-02-12T01:42:41Z",
          "content": "<p>Thank you!\nI decided weights to maximize OOF score.</p>",
          "rawMarkdown": "Thank you!\nI decided weights to maximize OOF score."
        }
      ]
    },
    {
      "id": 780091,
      "postDate": "2020-03-20T01:03:34.427Z",
      "content": "<p>where is your code?</p>",
      "rawMarkdown": "where is your code?"
    },
    {
      "id": 752985,
      "postDate": "2020-02-21T15:41:15.023Z",
      "content": "<p>Congrats! And thank you for sharing detailed write up.</p>",
      "rawMarkdown": "Congrats! And thank you for sharing detailed write up."
    },
    {
      "id": 752090,
      "postDate": "2020-02-20T18:36:52.253Z",
      "content": "<p>Nice kernel good </p>",
      "rawMarkdown": "Nice kernel good "
    },
    {
      "id": 751133,
      "postDate": "2020-02-20T02:38:33.500Z",
      "content": "<p>Hi, I'm curious about how you could fit the title+body+answer at the same time into the model? For me, I trained title+question and title+answer separately because the input is so long otherwise. Could you help me briefly explain how you truncate your input? Thanks! </p>",
      "rawMarkdown": "Hi, I'm curious about how you could fit the title+body+answer at the same time into the model? For me, I trained title+question and title+answer separately because the input is so long otherwise. Could you help me briefly explain how you truncate your input? Thanks! "
    },
    {
      "id": 742832,
      "postDate": "2020-02-11T15:11:04.983Z",
      "content": "<p>Thanks for your detailed write-up!</p>",
      "rawMarkdown": "Thanks for your detailed write-up!",
      "votes": 1
    },
    {
      "id": 742558,
      "postDate": "2020-02-11T11:27:56.853Z",
      "content": "<p>Congrats!! Thanks for the writeup :)</p>",
      "rawMarkdown": "Congrats!! Thanks for the writeup :)",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:08:05Z",
  "competition_id": 7968,
  "competition_slug": "google-quest-challenge",
  "team_id": 4015049,
  "solution_url": "/competitions/google-quest-challenge/writeups/sakami-3rd-place-solution"
}