{
  "id": 130158,
  "title": "22nd place solution(NLP beginner)",
  "content": "<p>Thanks kaggle for holding a great competition!</p>\n\n<p>I'm novice at NLP, so I learned a lot from great kernels about preprocessing, how to build Bert model, etc...\nMy code is based on <a href=\"https://www.kaggle.com/idv2005/pytorch-bert-baseline\">this kernel</a> by @idv2005  and <a href=\"https://www.kaggle.com/adityaecdrid/pytorch-bert-end-to-end-with-cv\">this kernel</a> by @adityaecdrid ! Thank you very much!</p>\n\n<p>Here is the summary of my work.(Sorry for poor English.)</p>\n\n<h1>Solution</h1>\n\n<ul>\n<li>bert-base-cased, bert-base-uncased, xlnet-base-uncased and Twin models(XLNet, Bert).</li>\n<li>Twin model idea is from <a href=\"https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer\">https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer</a>. thanks a lot!. Twin models have Bert for Question and Bert for Answer and treat sequence output with 3 custom layers.\n<ol><li>Average pooling for QBert/ABert =&gt; concatenate(shape=768*2) =&gt; dense(30)</li>\n<li>Average pooling for QBert =&gt; dense(22), Average pooling for ABert =&gt; dense(8), concatenate QBert dense output and ABert dense output.</li>\n<li>concatenate QBert/ABert sequence output(shape=512, 768*2) =&gt; Transformers =&gt; average pooling =&gt; dense(30)</li></ol></li>\n<li>Final output: a * 0.25 + b * 0.25 + c * 0.5</li>\n<li>postprocess (thank you for <a href=\"https://www.kaggle.com/c/google-quest-challenge/discussion/118724\">this discussion</a>)\n```\nbest_bin_dict = {'question_type_spelling': 4, 'question_type_instructions': 8, 'question_type_entity': 9, 'question_type_definition': 8, 'question_type_consequence': 4, 'question_type_compare': 8, 'question_type_choice': 8, 'question_opinion_seeking': 50, 'question_not_really_a_question': 4, 'question_multi_intent': 9, 'question_interestingness_self': 9, 'question_interestingness_others': 100, 'question_has_commonly_accepted_answer': 5, 'question_fact_seeking': 10, 'question_expect_short_answer': 50, 'question_conversational': 8, 'question_body_critical': 100, 'question_asker_intent_understanding': 100, 'answer_well_written': 9, 'answer_type_procedure': 50, 'answer_type_instructions': 9, 'answer_relevance': 100, 'answer_level_of_information': 50}</li>\n</ul>\n\n<p>df_sub = pd.read_csv(\"../input/google-quest-challenge/sample_submission.csv\")\npred_final = np.array(preds).sum(axis=0)\nfor i, col in enumerate(df_sub.columns[1:]):\n    df_sub[col] = pred_final[:, i]\nnot_type_spelling_idx = df_test.query(\"host not in ['ell.stackexchange.com', 'english.stackexchange.com']\").index</p>\n\n<p>for col in df_sub.columns[1:]:\n    if col in best_bin_dict:\n        n_bins = best_bin_dict[col]\n        binned = pd.cut(df_sub[col].values, n_bins, retbins=True, labels=np.arange(n_bins)/(n_bins-1))[0]\n        if col == \"question_type_spelling\":\n           binned[not_type_spelling_idx] = 0\n        df_sub[col] = binned\ndf_sub.to_csv(\"submission.csv\", index=False)\n```</p>\n\n<h1>Not worked</h1>\n\n<ul>\n<li>DistilBert</li>\n<li>bert-large-cased, bert-large-uncased, bert-large-uncased-whole-masking</li>\n<li>Categories/hosts Embedding</li>\n<li>L1loss, L2Loss, Focal Loss (Finally I use BCELoss... but I can't understand why BCELoss is best unless it is not symmetric in this case... refer=&gt; <a href=\"https://www.kaggle.com/mavillan/understanding-the-binary-cross-entropy\">https://www.kaggle.com/mavillan/understanding-the-binary-cross-entropy</a>)</li>\n<li>Subtask: predict categories.</li>\n<li>Concat CLS token sequence and average pooling.</li>\n</ul>\n\n<h1>Others</h1>\n\n<ul>\n<li>I consumed about $200 for this competition.</li>\n</ul>",
  "comments": [
    {
      "id": 744091,
      "author_name": "Victor Menuzzo",
      "author_url": "",
      "post_date": "2020-02-12T14:50:23.557000",
      "content": "<p>congrats bro!!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 744068,
      "author_name": "Aditya Soni",
      "author_url": "",
      "post_date": "2020-02-12T14:35:24.300000",
      "content": "<p>No ways now you are a Novice :)\nCongratulations!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 746504,
      "author_name": "jijeng",
      "author_url": "",
      "post_date": "2020-02-15T05:44:20.890000",
      "content": "<p><code>I consumed about $200 for this competition.</code> \nInteresting ...</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 745338,
      "author_name": "arjun",
      "author_url": "",
      "post_date": "2020-02-13T18:14:49.093000",
      "content": "<p>Thanks for sharing your insights, and congratulations!! Pretty much the winning scores are in favour of transformer based models. I presume you had used GPU beyond what kaggle had to offer (when you mention $200 consumption), is that true? I embraced bert (albert), only after many solutions starting from word2vec (bag of words and fastai based models -sg), bpe encoding, plain old word embeddings with lstm (attention based), before finally settling on albert only a week or two before the final submission. By that time, it was too late to even try a few changes (didn't plan to spend more than what kaggle had to offer).</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 745013,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-13T11:46:21.547000",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 5,
  "messages": [
    {
      "id": 744004,
      "postDate": "2020-02-12T13:19:54.660Z",
      "content": "<p>Thanks kaggle for holding a great competition!</p>\n\n<p>I'm novice at NLP, so I learned a lot from great kernels about preprocessing, how to build Bert model, etc...\nMy code is based on <a href=\"https://www.kaggle.com/idv2005/pytorch-bert-baseline\">this kernel</a> by @idv2005  and <a href=\"https://www.kaggle.com/adityaecdrid/pytorch-bert-end-to-end-with-cv\">this kernel</a> by @adityaecdrid ! Thank you very much!</p>\n\n<p>Here is the summary of my work.(Sorry for poor English.)</p>\n\n<h1>Solution</h1>\n\n<ul>\n<li>bert-base-cased, bert-base-uncased, xlnet-base-uncased and Twin models(XLNet, Bert).</li>\n<li>Twin model idea is from <a href=\"https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer\">https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer</a>. thanks a lot!. Twin models have Bert for Question and Bert for Answer and treat sequence output with 3 custom layers.\n<ol><li>Average pooling for QBert/ABert =&gt; concatenate(shape=768*2) =&gt; dense(30)</li>\n<li>Average pooling for QBert =&gt; dense(22), Average pooling for ABert =&gt; dense(8), concatenate QBert dense output and ABert dense output.</li>\n<li>concatenate QBert/ABert sequence output(shape=512, 768*2) =&gt; Transformers =&gt; average pooling =&gt; dense(30)</li></ol></li>\n<li>Final output: a * 0.25 + b * 0.25 + c * 0.5</li>\n<li>postprocess (thank you for <a href=\"https://www.kaggle.com/c/google-quest-challenge/discussion/118724\">this discussion</a>)\n```\nbest_bin_dict = {'question_type_spelling': 4, 'question_type_instructions': 8, 'question_type_entity': 9, 'question_type_definition': 8, 'question_type_consequence': 4, 'question_type_compare': 8, 'question_type_choice': 8, 'question_opinion_seeking': 50, 'question_not_really_a_question': 4, 'question_multi_intent': 9, 'question_interestingness_self': 9, 'question_interestingness_others': 100, 'question_has_commonly_accepted_answer': 5, 'question_fact_seeking': 10, 'question_expect_short_answer': 50, 'question_conversational': 8, 'question_body_critical': 100, 'question_asker_intent_understanding': 100, 'answer_well_written': 9, 'answer_type_procedure': 50, 'answer_type_instructions': 9, 'answer_relevance': 100, 'answer_level_of_information': 50}</li>\n</ul>\n\n<p>df_sub = pd.read_csv(\"../input/google-quest-challenge/sample_submission.csv\")\npred_final = np.array(preds).sum(axis=0)\nfor i, col in enumerate(df_sub.columns[1:]):\n    df_sub[col] = pred_final[:, i]\nnot_type_spelling_idx = df_test.query(\"host not in ['ell.stackexchange.com', 'english.stackexchange.com']\").index</p>\n\n<p>for col in df_sub.columns[1:]:\n    if col in best_bin_dict:\n        n_bins = best_bin_dict[col]\n        binned = pd.cut(df_sub[col].values, n_bins, retbins=True, labels=np.arange(n_bins)/(n_bins-1))[0]\n        if col == \"question_type_spelling\":\n           binned[not_type_spelling_idx] = 0\n        df_sub[col] = binned\ndf_sub.to_csv(\"submission.csv\", index=False)\n```</p>\n\n<h1>Not worked</h1>\n\n<ul>\n<li>DistilBert</li>\n<li>bert-large-cased, bert-large-uncased, bert-large-uncased-whole-masking</li>\n<li>Categories/hosts Embedding</li>\n<li>L1loss, L2Loss, Focal Loss (Finally I use BCELoss... but I can't understand why BCELoss is best unless it is not symmetric in this case... refer=&gt; <a href=\"https://www.kaggle.com/mavillan/understanding-the-binary-cross-entropy\">https://www.kaggle.com/mavillan/understanding-the-binary-cross-entropy</a>)</li>\n<li>Subtask: predict categories.</li>\n<li>Concat CLS token sequence and average pooling.</li>\n</ul>\n\n<h1>Others</h1>\n\n<ul>\n<li>I consumed about $200 for this competition.</li>\n</ul>",
      "rawMarkdown": "Thanks kaggle for holding a great competition!\n\nI'm novice at NLP, so I learned a lot from great kernels about preprocessing, how to build Bert model, etc...\nMy code is based on <a href=\"https://www.kaggle.com/idv2005/pytorch-bert-baseline\">this kernel</a> by @idv2005  and <a href=\"https://www.kaggle.com/adityaecdrid/pytorch-bert-end-to-end-with-cv\">this kernel</a> by @adityaecdrid ! Thank you very much!\n\nHere is the summary of my work.(Sorry for poor English.)\n\n# Solution\n* bert-base-cased, bert-base-uncased, xlnet-base-uncased and Twin models(XLNet, Bert).\n* Twin model idea is from https://www.kaggle.com/akensert/bert-base-tf2-0-now-huggingface-transformer. thanks a lot!. Twin models have Bert for Question and Bert for Answer and treat sequence output with 3 custom layers.\n  1. Average pooling for QBert/ABert =&gt; concatenate(shape=768*2) =&gt; dense(30)\n  2. Average pooling for QBert =&gt; dense(22), Average pooling for ABert =&gt; dense(8), concatenate QBert dense output and ABert dense output.\n  3. concatenate QBert/ABert sequence output(shape=512, 768*2) =&gt; Transformers =&gt; average pooling =&gt; dense(30)\n* Final output: a * 0.25 + b * 0.25 + c * 0.5\n* postprocess (thank you for <a href=\"https://www.kaggle.com/c/google-quest-challenge/discussion/118724\">this discussion</a>)\n```\nbest_bin_dict = {'question_type_spelling': 4, 'question_type_instructions': 8, 'question_type_entity': 9, 'question_type_definition': 8, 'question_type_consequence': 4, 'question_type_compare': 8, 'question_type_choice': 8, 'question_opinion_seeking': 50, 'question_not_really_a_question': 4, 'question_multi_intent': 9, 'question_interestingness_self': 9, 'question_interestingness_others': 100, 'question_has_commonly_accepted_answer': 5, 'question_fact_seeking': 10, 'question_expect_short_answer': 50, 'question_conversational': 8, 'question_body_critical': 100, 'question_asker_intent_understanding': 100, 'answer_well_written': 9, 'answer_type_procedure': 50, 'answer_type_instructions': 9, 'answer_relevance': 100, 'answer_level_of_information': 50}\n\ndf_sub = pd.read_csv(\"../input/google-quest-challenge/sample_submission.csv\")\npred_final = np.array(preds).sum(axis=0)\nfor i, col in enumerate(df_sub.columns[1:]):\n    df_sub[col] = pred_final[:, i]\nnot_type_spelling_idx = df_test.query(\"host not in ['ell.stackexchange.com', 'english.stackexchange.com']\").index\n\nfor col in df_sub.columns[1:]:\n    if col in best_bin_dict:\n        n_bins = best_bin_dict[col]\n        binned = pd.cut(df_sub[col].values, n_bins, retbins=True, labels=np.arange(n_bins)/(n_bins-1))[0]\n        if col == \"question_type_spelling\":\n           binned[not_type_spelling_idx] = 0\n        df_sub[col] = binned\ndf_sub.to_csv(\"submission.csv\", index=False)\n```\n\n# Not worked\n* DistilBert\n* bert-large-cased, bert-large-uncased, bert-large-uncased-whole-masking\n* Categories/hosts Embedding\n* L1loss, L2Loss, Focal Loss (Finally I use BCELoss... but I can't understand why BCELoss is best unless it is not symmetric in this case... refer=&gt; https://www.kaggle.com/mavillan/understanding-the-binary-cross-entropy)\n* Subtask: predict categories.\n* Concat CLS token sequence and average pooling.\n\n# Others\n* I consumed about $200 for this competition.",
      "votes": 11
    },
    {
      "id": 744091,
      "postDate": "2020-02-12T14:50:23.557Z",
      "content": "<p>congrats bro!!</p>",
      "rawMarkdown": "congrats bro!!",
      "votes": 1
    },
    {
      "id": 744068,
      "postDate": "2020-02-12T14:35:24.300Z",
      "content": "<p>No ways now you are a Novice :)\nCongratulations!</p>",
      "rawMarkdown": "No ways now you are a Novice :)\nCongratulations!",
      "votes": 1
    },
    {
      "id": 746504,
      "postDate": "2020-02-15T05:44:20.890Z",
      "content": "<p><code>I consumed about $200 for this competition.</code> \nInteresting ...</p>",
      "rawMarkdown": "`I consumed about $200 for this competition.` \nInteresting ..."
    },
    {
      "id": 745338,
      "postDate": "2020-02-13T18:14:49.093Z",
      "content": "<p>Thanks for sharing your insights, and congratulations!! Pretty much the winning scores are in favour of transformer based models. I presume you had used GPU beyond what kaggle had to offer (when you mention $200 consumption), is that true? I embraced bert (albert), only after many solutions starting from word2vec (bag of words and fastai based models -sg), bpe encoding, plain old word embeddings with lstm (attention based), before finally settling on albert only a week or two before the final submission. By that time, it was too late to even try a few changes (didn't plan to spend more than what kaggle had to offer).</p>",
      "rawMarkdown": "Thanks for sharing your insights, and congratulations!! Pretty much the winning scores are in favour of transformer based models. I presume you had used GPU beyond what kaggle had to offer (when you mention $200 consumption), is that true? I embraced bert (albert), only after many solutions starting from word2vec (bag of words and fastai based models -sg), bpe encoding, plain old word embeddings with lstm (attention based), before finally settling on albert only a week or two before the final submission. By that time, it was too late to even try a few changes (didn't plan to spend more than what kaggle had to offer)."
    },
    {
      "id": 745013,
      "postDate": "2020-02-13T11:46:21.547Z",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "rawMarkdown": "Congratulations and thanks for sharing!",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:09:09Z",
  "competition_id": 7968,
  "competition_slug": "google-quest-challenge",
  "team_id": 4101058,
  "solution_url": "/competitions/google-quest-challenge/writeups/kaggler-ja-kurupical-22nd-place-solution-nlp-begin"
}