{
  "id": 129895,
  "title": "19th solution",
  "content": "<p>First of all, thank you for holding this great competition google and kaggle team. \nI think the balance of need for latest NLP knowledge and ideas applied to this specific task is the exciting point of this competition! I could not get the gold medal, but I’ve enjoyed and learned a lot.</p>\n\n<p>I want to share my insights which I’ve achieved from more than 150 experiments. My <a href=\"https://github.com/guchio3/kaggle-google-quest\">codes</a> and <a href=\"https://www.kaggle.com/guchio3/refactored-bert2-robert1-xlnet1-blend-pseudo1\">kernels</a> are publicly available now.</p>\n\n<h1>preprocess</h1>\n\n<ul>\n<li>add special token to separate title and question.\n<ul><li>ex. [CLS] title [NEW_SEP] question [SEP] answer [SEP]</li></ul></li>\n<li>add category info to the head of the title.\n<ul><li>ex. [CLS] CAT-TECHNOLOGY title [NEW_SEP] question [SEP] answer [SEP]</li>\n<li>This method should be better than the case just concatenating the category to the last classification layer because the hugging-face model can use the category info while it retrieves the texts.</li></ul></li>\n<li>trim head and tail part of the texts <a href=\"https://github.com/guchio3/kaggle-google-quest/blob/decffc69d5657f5114970eb2ea226df8ec8cfaf6/scripts/refactor/datasets.py#L137-L157\">like this</a>.\n<ul><li>ex. I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners will be able to sit down together at the table of brotherhood.
\n-&gt; I have a dream that + at the table of brotherhood.</li></ul></li>\n</ul>\n\n<h1>modeling (used pytorch hugging-face library)</h1>\n\n<ul>\n<li>GKF using question body</li>\n<li>AVG pool final layer seq output and use it for the input 1 linear layer</li>\n<li>use 2 separate models for question type label and answer type label\n<ul><li>(title-max-len, question-max-len, answer-max-len) : (30, 239*2, 0) and (30, 0, 239*2) for each (if each part is less than the max-len, the other parts use the rest len).</li></ul></li>\n<li>Ensemble of four models (1 : 0.5 : 0.5 : 1)\n<ul><li>bert-base-uncased (best snapshot)</li>\n<li>robert-base (best snapshot, second-best snapshot)</li>\n<li>xlnet-base-cased (best smapshot)</li></ul></li>\n<li>batch size 8, 6 epochs (freezing is crucial)\n<ul><li>freeze only hugging-face model for the first 1 epoch, and unfreeze it the rest 5 epochs</li></ul></li>\n<li>BCEWithLogitsLoss</li>\n<li>Adam w/ cosine scheduler (3e-5 -&gt; 1e-5)</li>\n<li>1 cycle pseudo labeling\n<ul><li>concat soft target, hard target, and half hard target. Half hard target is the target where the threshold is applied to the col only if the application improve the oof score</li></ul></li>\n</ul>\n\n<h1>postprocess</h1>\n\n<ul>\n<li>Search and apply thresholds\n<ul><li>Get optimized thresholds using scipy.optimize.minimize for oofs. The initial value is calculated from the percentile point of the oofs. (<a href=\"https://github.com/guchio3/kaggle-google-quest/blob/master/scripts/get_optR3.py\">ref</a>)</li>\n<li>Each thresholds is applied to a column only if the oof score improves by using it.</li>\n<li>Each thresholds is applied to a column only if the cardinality of a column is more than one for public/private part of the test set.</li></ul></li>\n<li>Set 0 or 0.5 to question-type-spelling based on the host (kernel based)</li>\n</ul>\n\n<h1>What did not work for me</h1>\n\n<ul>\n<li>Larger batch size</li>\n<li>Longer sequence size\n<ul><li>Longer position embedding matrix than 512</li>\n<li>Cyclic position ids (0 - 29 for title, 0 - 511 for quesiton, 0 - 511 for answer)</li></ul></li>\n<li>Shorter sequence size</li>\n<li>Using meta features except for category\n<ul><li>host, word counts, tokenized word counts, etc …</li></ul></li>\n<li>Using last N hidden outputs of the hugging-face models</li>\n<li>Augmentations</li>\n<li>Manual text preprocessings</li>\n<li>Changing loss functions\n<ul><li>MSE, Pair type loss, Focal loss</li></ul></li>\n<li>Changing optimizers and schedulers\n<ul><li>AdamW, RMSProp, SGD, Step type schedulers</li></ul></li>\n<li>Using SWA</li>\n<li>Expert models for difficult or imbalanced classes</li>\n<li>Oversampling for the minor values of imbalanced classes</li>\n<li>Using the other type of models for ensemble\n<ul><li>RNN based models, GPT2, XLM</li></ul></li>\n</ul>\n\n<p>P.S.\nLike some people say, I used so long time to reproduce the result of <a href=\"https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer\">great TF kernel</a> by pytorch. At first, I concluded that the situation is caused by the pytorch’s nature <a href=\"https://discuss.pytorch.org/t/suboptimal-convergence-when-compared-with-tensorflow-model/5099/18\">like this</a>. However, I noticed that the kernel use snapshot ensemble in the callback function, and this should cause the difference of the performance. I think pytorch users are not familiar with callback functions, so we should overlook it.</p>",
  "comments": [
    {
      "id": 752021,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-20T17:50:14.737000",
      "content": "<p>Great work</p>",
      "votes": 14,
      "replies": []
    },
    {
      "id": 752340,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-20T22:28:55.917000",
      "content": "<p>Great</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742334,
      "author_name": "ilovescience",
      "author_url": "",
      "post_date": "2020-02-11T08:38:01.403000",
      "content": "<p>Thanks for sharing your great approach and congrats!</p>\n\n<p>Also, thanks for pointing out the situation about snapshot ensembling in <a href=\"/akensert\">@akensert</a>'s kernel.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742301,
      "author_name": "Ailurophile",
      "author_url": "",
      "post_date": "2020-02-11T08:16:23.263000",
      "content": "<p>Congratulations!\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/guchio3\">@guchio3</a> </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742300,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-11T08:15:24.213000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 742287,
      "author_name": "Marek Nurzynski",
      "author_url": "",
      "post_date": "2020-02-11T08:02:31.810000",
      "content": "<p>Very usefull! Thanks for sharing :)</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 6,
  "messages": [
    {
      "id": 742243,
      "postDate": "2020-02-11T07:19:40.743Z",
      "content": "<p>First of all, thank you for holding this great competition google and kaggle team. \nI think the balance of need for latest NLP knowledge and ideas applied to this specific task is the exciting point of this competition! I could not get the gold medal, but I’ve enjoyed and learned a lot.</p>\n\n<p>I want to share my insights which I’ve achieved from more than 150 experiments. My <a href=\"https://github.com/guchio3/kaggle-google-quest\">codes</a> and <a href=\"https://www.kaggle.com/guchio3/refactored-bert2-robert1-xlnet1-blend-pseudo1\">kernels</a> are publicly available now.</p>\n\n<h1>preprocess</h1>\n\n<ul>\n<li>add special token to separate title and question.\n<ul><li>ex. [CLS] title [NEW_SEP] question [SEP] answer [SEP]</li></ul></li>\n<li>add category info to the head of the title.\n<ul><li>ex. [CLS] CAT-TECHNOLOGY title [NEW_SEP] question [SEP] answer [SEP]</li>\n<li>This method should be better than the case just concatenating the category to the last classification layer because the hugging-face model can use the category info while it retrieves the texts.</li></ul></li>\n<li>trim head and tail part of the texts <a href=\"https://github.com/guchio3/kaggle-google-quest/blob/decffc69d5657f5114970eb2ea226df8ec8cfaf6/scripts/refactor/datasets.py#L137-L157\">like this</a>.\n<ul><li>ex. I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners will be able to sit down together at the table of brotherhood.
\n-&gt; I have a dream that + at the table of brotherhood.</li></ul></li>\n</ul>\n\n<h1>modeling (used pytorch hugging-face library)</h1>\n\n<ul>\n<li>GKF using question body</li>\n<li>AVG pool final layer seq output and use it for the input 1 linear layer</li>\n<li>use 2 separate models for question type label and answer type label\n<ul><li>(title-max-len, question-max-len, answer-max-len) : (30, 239*2, 0) and (30, 0, 239*2) for each (if each part is less than the max-len, the other parts use the rest len).</li></ul></li>\n<li>Ensemble of four models (1 : 0.5 : 0.5 : 1)\n<ul><li>bert-base-uncased (best snapshot)</li>\n<li>robert-base (best snapshot, second-best snapshot)</li>\n<li>xlnet-base-cased (best smapshot)</li></ul></li>\n<li>batch size 8, 6 epochs (freezing is crucial)\n<ul><li>freeze only hugging-face model for the first 1 epoch, and unfreeze it the rest 5 epochs</li></ul></li>\n<li>BCEWithLogitsLoss</li>\n<li>Adam w/ cosine scheduler (3e-5 -&gt; 1e-5)</li>\n<li>1 cycle pseudo labeling\n<ul><li>concat soft target, hard target, and half hard target. Half hard target is the target where the threshold is applied to the col only if the application improve the oof score</li></ul></li>\n</ul>\n\n<h1>postprocess</h1>\n\n<ul>\n<li>Search and apply thresholds\n<ul><li>Get optimized thresholds using scipy.optimize.minimize for oofs. The initial value is calculated from the percentile point of the oofs. (<a href=\"https://github.com/guchio3/kaggle-google-quest/blob/master/scripts/get_optR3.py\">ref</a>)</li>\n<li>Each thresholds is applied to a column only if the oof score improves by using it.</li>\n<li>Each thresholds is applied to a column only if the cardinality of a column is more than one for public/private part of the test set.</li></ul></li>\n<li>Set 0 or 0.5 to question-type-spelling based on the host (kernel based)</li>\n</ul>\n\n<h1>What did not work for me</h1>\n\n<ul>\n<li>Larger batch size</li>\n<li>Longer sequence size\n<ul><li>Longer position embedding matrix than 512</li>\n<li>Cyclic position ids (0 - 29 for title, 0 - 511 for quesiton, 0 - 511 for answer)</li></ul></li>\n<li>Shorter sequence size</li>\n<li>Using meta features except for category\n<ul><li>host, word counts, tokenized word counts, etc …</li></ul></li>\n<li>Using last N hidden outputs of the hugging-face models</li>\n<li>Augmentations</li>\n<li>Manual text preprocessings</li>\n<li>Changing loss functions\n<ul><li>MSE, Pair type loss, Focal loss</li></ul></li>\n<li>Changing optimizers and schedulers\n<ul><li>AdamW, RMSProp, SGD, Step type schedulers</li></ul></li>\n<li>Using SWA</li>\n<li>Expert models for difficult or imbalanced classes</li>\n<li>Oversampling for the minor values of imbalanced classes</li>\n<li>Using the other type of models for ensemble\n<ul><li>RNN based models, GPT2, XLM</li></ul></li>\n</ul>\n\n<p>P.S.\nLike some people say, I used so long time to reproduce the result of <a href=\"https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer\">great TF kernel</a> by pytorch. At first, I concluded that the situation is caused by the pytorch’s nature <a href=\"https://discuss.pytorch.org/t/suboptimal-convergence-when-compared-with-tensorflow-model/5099/18\">like this</a>. However, I noticed that the kernel use snapshot ensemble in the callback function, and this should cause the difference of the performance. I think pytorch users are not familiar with callback functions, so we should overlook it.</p>",
      "rawMarkdown": "First of all, thank you for holding this great competition google and kaggle team. \nI think the balance of need for latest NLP knowledge and ideas applied to this specific task is the exciting point of this competition! I could not get the gold medal, but I’ve enjoyed and learned a lot.\n\n\nI want to share my insights which I’ve achieved from more than 150 experiments. My [codes](https://github.com/guchio3/kaggle-google-quest) and [kernels](https://www.kaggle.com/guchio3/refactored-bert2-robert1-xlnet1-blend-pseudo1) are publicly available now.\n\n\n\n# preprocess \n- add special token to separate title and question.\n    - ex. [CLS] title [NEW_SEP] question [SEP] answer [SEP]\n- add category info to the head of the title.\n    - ex. [CLS] CAT-TECHNOLOGY title [NEW_SEP] question [SEP] answer [SEP]\n    - This method should be better than the case just concatenating the category to the last classification layer because the hugging-face model can use the category info while it retrieves the texts.\n- trim head and tail part of the texts [like this](https://github.com/guchio3/kaggle-google-quest/blob/decffc69d5657f5114970eb2ea226df8ec8cfaf6/scripts/refactor/datasets.py#L137-L157).\n    - ex. I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners will be able to sit down together at the table of brotherhood.
  \n-&gt; I have a dream that + at the table of brotherhood.\n\n\n# modeling (used pytorch hugging-face library)\n- GKF using question body\n- AVG pool final layer seq output and use it for the input 1 linear layer\n- use 2 separate models for question type label and answer type label\n    - (title-max-len, question-max-len, answer-max-len) : (30, 239*2, 0) and (30, 0, 239*2) for each (if each part is less than the max-len, the other parts use the rest len).\n- Ensemble of four models (1 : 0.5 : 0.5 : 1)\n    - bert-base-uncased (best snapshot)\n    - robert-base (best snapshot, second-best snapshot)\n    - xlnet-base-cased (best smapshot)\n- batch size 8, 6 epochs (freezing is crucial)\n    - freeze only hugging-face model for the first 1 epoch, and unfreeze it the rest 5 epochs\n- BCEWithLogitsLoss\n- Adam w/ cosine scheduler (3e-5 -&gt; 1e-5)\n- 1 cycle pseudo labeling\n    - concat soft target, hard target, and half hard target. Half hard target is the target where the threshold is applied to the col only if the application improve the oof score\n\n\n# postprocess\n- Search and apply thresholds\n    - Get optimized thresholds using scipy.optimize.minimize for oofs. The initial value is calculated from the percentile point of the oofs. ([ref](https://github.com/guchio3/kaggle-google-quest/blob/master/scripts/get_optR3.py))\n    - Each thresholds is applied to a column only if the oof score improves by using it.\n    - Each thresholds is applied to a column only if the cardinality of a column is more than one for public/private part of the test set.\n- Set 0 or 0.5 to question-type-spelling based on the host (kernel based)\n\n  \n# What did not work for me\n- Larger batch size\n- Longer sequence size\n    - Longer position embedding matrix than 512\n    - Cyclic position ids (0 - 29 for title, 0 - 511 for quesiton, 0 - 511 for answer)\n- Shorter sequence size\n- Using meta features except for category\n    - host, word counts, tokenized word counts, etc …\n- Using last N hidden outputs of the hugging-face models\n- Augmentations\n- Manual text preprocessings\n- Changing loss functions\n    - MSE, Pair type loss, Focal loss\n- Changing optimizers and schedulers\n    - AdamW, RMSProp, SGD, Step type schedulers\n- Using SWA\n- Expert models for difficult or imbalanced classes\n- Oversampling for the minor values of imbalanced classes\n- Using the other type of models for ensemble\n    - RNN based models, GPT2, XLM\n\n\nP.S.\nLike some people say, I used so long time to reproduce the result of [great TF kernel](https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer) by pytorch. At first, I concluded that the situation is caused by the pytorch’s nature [like this](https://discuss.pytorch.org/t/suboptimal-convergence-when-compared-with-tensorflow-model/5099/18). However, I noticed that the kernel use snapshot ensemble in the callback function, and this should cause the difference of the performance. I think pytorch users are not familiar with callback functions, so we should overlook it.",
      "votes": 26
    },
    {
      "id": 752021,
      "postDate": "2020-02-20T17:50:14.737Z",
      "content": "<p>Great work</p>",
      "rawMarkdown": "Great work",
      "votes": 14
    },
    {
      "id": 752340,
      "postDate": "2020-02-20T22:28:55.917Z",
      "content": "<p>Great</p>",
      "rawMarkdown": "Great",
      "votes": 1
    },
    {
      "id": 742334,
      "postDate": "2020-02-11T08:38:01.403Z",
      "content": "<p>Thanks for sharing your great approach and congrats!</p>\n\n<p>Also, thanks for pointing out the situation about snapshot ensembling in <a href=\"/akensert\">@akensert</a>'s kernel.</p>",
      "rawMarkdown": "Thanks for sharing your great approach and congrats!\n\nAlso, thanks for pointing out the situation about snapshot ensembling in @akensert's kernel.",
      "votes": 1
    },
    {
      "id": 742301,
      "postDate": "2020-02-11T08:16:23.263Z",
      "content": "<p>Congratulations!\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/guchio3\">@guchio3</a> </p>",
      "rawMarkdown": "Congratulations!\nThanks for Sharing your Approach &amp; Insights!! @guchio3 ",
      "votes": 1
    },
    {
      "id": 742300,
      "postDate": "2020-02-11T08:15:24.213Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 742287,
      "postDate": "2020-02-11T08:02:31.810Z",
      "content": "<p>Very usefull! Thanks for sharing :)</p>",
      "rawMarkdown": "Very usefull! Thanks for sharing :)",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:07:50Z",
  "competition_id": 7968,
  "competition_slug": "google-quest-challenge",
  "team_id": 4096085,
  "solution_url": "/competitions/google-quest-challenge/writeups/guchio3-19th-solution"
}