{
  "id": 129923,
  "title": "My First Medal (Bronze) Solution",
  "content": "<p>Thank you Google and Kaggle for the challenging competition (a good mixture of science and art imo)! </p>\n\n<p>Also, congrats to all teams! :)</p>\n\n<h2><strong>Pre-Processing:</strong></h2>\n\n<ul>\n<li>unidecode and html unescape</li>\n</ul>\n\n<h2><strong>Modelling:</strong></h2>\n\n<ol>\n<li>Fine-tune Bert base uncased on raw data + statistical features (LB: 0.384)</li>\n<li>Fine-tune Bert base uncased on pre-processed data + statistical features (LB: 0.384)</li>\n<li><p>Feature extraction model using GloVe, FastText, USE 5, Paragram and distilBert (LB: 0.371)</p>\n\n<ul><li>I used TF + hugging face (on hindsight, reproducibility was really problematic)</li>\n<li>All had similar architecture using Adam + BCE loss with sigmoid output on all 30 labels</li>\n<li>I used spatial dropout to reduce overfitting when fine-tuning Bert</li>\n<li>Plain vanilla 5 KFolds (lucky seed 42) combined with 2 epochs after early stopping</li>\n<li>TTA using pre-processed text and raw text</li></ul></li>\n</ol>\n\n<p>Ensembling them using simple mean (across folds, epochs, and models) got me LB: 0.394</p>\n\n<h2>Post-Processing:</h2>\n\n<ul>\n<li>Simple quantile threshold on \"easy targets\" mimicing train set labels (with a margin)\ne.g. \n<code>threshold = submission[c].quantile(0.75)</code>\n<code>submission[c].apply(lambda x: 1 if x &gt; threshold else x)</code></li>\n</ul>\n\n<p>\"easy targets\" selected based on std and nunique of train set labels\n- Rounding prediction to 4 significant figures to account for \"tied ranks\"</p>\n\n<p>Post-processing outputs gave me a boost to LB: 0.410</p>\n\n<h2>What I tried that didn't work:</h2>\n\n<ul>\n<li>GRUs/LSTMs </li>\n<li>Regression NN with target transformation (similar CV but doesn't generalise well to public LB)</li>\n<li>Data augmentation (random deletion, random insertion, switching question/answers sequences, etc)</li>\n<li>XLNet base, Roberta base, Albert base, Bert cased could not beat my Bert base uncased baseline</li>\n<li>Rank averaging only helped if no post-processing was done</li>\n<li>Rounding predictions to nearest 1/90 </li>\n</ul>\n\n<h2>Biggest take-away:</h2>\n\n<p>Don't focus <em>too much</em> on public kernels and discussion. Just have a look, get some ideas, look at the data again and try your own ideas. (p.s. I think someone mentioned this on the discussion thread but I forgot which and who)</p>",
  "comments": [
    {
      "id": 742463,
      "author_name": "Aditya Soni",
      "author_url": "",
      "post_date": "2020-02-11T10:02:56.790000",
      "content": "<p><a href=\"/yuval6967\">@yuval6967</a> mentioned this and many others!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 742471,
          "author_name": "Gerwyn",
          "author_url": "",
          "post_date": "2020-02-11T10:07:15.210000",
          "content": "<p>words of wisdom :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 742455,
      "postDate": "2020-02-11T09:56:16.153Z",
      "content": "<p>Thank you Google and Kaggle for the challenging competition (a good mixture of science and art imo)! </p>\n\n<p>Also, congrats to all teams! :)</p>\n\n<h2><strong>Pre-Processing:</strong></h2>\n\n<ul>\n<li>unidecode and html unescape</li>\n</ul>\n\n<h2><strong>Modelling:</strong></h2>\n\n<ol>\n<li>Fine-tune Bert base uncased on raw data + statistical features (LB: 0.384)</li>\n<li>Fine-tune Bert base uncased on pre-processed data + statistical features (LB: 0.384)</li>\n<li><p>Feature extraction model using GloVe, FastText, USE 5, Paragram and distilBert (LB: 0.371)</p>\n\n<ul><li>I used TF + hugging face (on hindsight, reproducibility was really problematic)</li>\n<li>All had similar architecture using Adam + BCE loss with sigmoid output on all 30 labels</li>\n<li>I used spatial dropout to reduce overfitting when fine-tuning Bert</li>\n<li>Plain vanilla 5 KFolds (lucky seed 42) combined with 2 epochs after early stopping</li>\n<li>TTA using pre-processed text and raw text</li></ul></li>\n</ol>\n\n<p>Ensembling them using simple mean (across folds, epochs, and models) got me LB: 0.394</p>\n\n<h2>Post-Processing:</h2>\n\n<ul>\n<li>Simple quantile threshold on \"easy targets\" mimicing train set labels (with a margin)\ne.g. \n<code>threshold = submission[c].quantile(0.75)</code>\n<code>submission[c].apply(lambda x: 1 if x &gt; threshold else x)</code></li>\n</ul>\n\n<p>\"easy targets\" selected based on std and nunique of train set labels\n- Rounding prediction to 4 significant figures to account for \"tied ranks\"</p>\n\n<p>Post-processing outputs gave me a boost to LB: 0.410</p>\n\n<h2>What I tried that didn't work:</h2>\n\n<ul>\n<li>GRUs/LSTMs </li>\n<li>Regression NN with target transformation (similar CV but doesn't generalise well to public LB)</li>\n<li>Data augmentation (random deletion, random insertion, switching question/answers sequences, etc)</li>\n<li>XLNet base, Roberta base, Albert base, Bert cased could not beat my Bert base uncased baseline</li>\n<li>Rank averaging only helped if no post-processing was done</li>\n<li>Rounding predictions to nearest 1/90 </li>\n</ul>\n\n<h2>Biggest take-away:</h2>\n\n<p>Don't focus <em>too much</em> on public kernels and discussion. Just have a look, get some ideas, look at the data again and try your own ideas. (p.s. I think someone mentioned this on the discussion thread but I forgot which and who)</p>",
      "rawMarkdown": "Thank you Google and Kaggle for the challenging competition (a good mixture of science and art imo)! \n\nAlso, congrats to all teams! :)\n\n## **Pre-Processing:** ###\n- unidecode and html unescape\n\n## **Modelling:** ###\n1. Fine-tune Bert base uncased on raw data + statistical features (LB: 0.384)\n2. Fine-tune Bert base uncased on pre-processed data + statistical features (LB: 0.384)\n3. Feature extraction model using GloVe, FastText, USE 5, Paragram and distilBert (LB: 0.371)\n\n- I used TF + hugging face (on hindsight, reproducibility was really problematic)\n- All had similar architecture using Adam + BCE loss with sigmoid output on all 30 labels\n- I used spatial dropout to reduce overfitting when fine-tuning Bert\n- Plain vanilla 5 KFolds (lucky seed 42) combined with 2 epochs after early stopping\n- TTA using pre-processed text and raw text\n\nEnsembling them using simple mean (across folds, epochs, and models) got me LB: 0.394\n\n## Post-Processing:\n- Simple quantile threshold on \"easy targets\" mimicing train set labels (with a margin)\ne.g. \n`threshold = submission[c].quantile(0.75) `\n`submission[c].apply(lambda x: 1 if x &gt; threshold else x)`\n\n\"easy targets\" selected based on std and nunique of train set labels\n- Rounding prediction to 4 significant figures to account for \"tied ranks\"\n\nPost-processing outputs gave me a boost to LB: 0.410\n\n\n## What I tried that didn't work:\n- GRUs/LSTMs \n- Regression NN with target transformation (similar CV but doesn't generalise well to public LB)\n- Data augmentation (random deletion, random insertion, switching question/answers sequences, etc)\n- XLNet base, Roberta base, Albert base, Bert cased could not beat my Bert base uncased baseline\n- Rank averaging only helped if no post-processing was done\n- Rounding predictions to nearest 1/90 \n \n## Biggest take-away:\nDon't focus *too much* on public kernels and discussion. Just have a look, get some ideas, look at the data again and try your own ideas. (p.s. I think someone mentioned this on the discussion thread but I forgot which and who)",
      "votes": 10
    },
    {
      "id": 742463,
      "postDate": "2020-02-11T10:02:56.790Z",
      "content": "<p><a href=\"/yuval6967\">@yuval6967</a> mentioned this and many others!</p>",
      "rawMarkdown": "@yuval6967 mentioned this and many others!",
      "votes": 1,
      "replies": [
        {
          "id": 742471,
          "postDate": "2020-02-11T10:07:15.210Z",
          "content": "<p>words of wisdom :)</p>",
          "rawMarkdown": "words of wisdom :)"
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:08:03Z",
  "competition_id": 7968,
  "competition_slug": "google-quest-challenge",
  "team_id": 4207170,
  "solution_url": "/competitions/google-quest-challenge/writeups/gerwyn-ng-my-first-medal-bronze-solution"
}