{
  "id": 129859,
  "title": "15th solution with codes",
  "content": "<p><strong>Congrats to all winners, and thanks to my teammates</strong>! It's a long journey to finish this competition and get a silver medal, it's the first time for me to work with nlp projects.</p>\n\n<h2>Our solution could be summarized as:</h2>\n\n<ol>\n<li>Post processing, our idea is to group values to discrete values existing in training set. We chose columns to apply post processing by comparing oof.  Post processing could be found in <a href=\"https://www.kaggle.com/jionie/models-with-optimization-v5/\">my inference kernel</a></li>\n<li>Ensemble, our final submission consists of 8 base models as you can see in the inference kernel, the best model is two part roberta base model which got 0.412 oof without post processing and 0.439 lb with post processing.</li>\n<li>Modified pooler layer and multiple hidden_states , we used multiple hidden_states for the input of pooler layer inspired from <a href=\"https://www.kaggle.com/c/google-quest-challenge/discussion/123770\">this great post</a> we followed source codes in transformers for pooler class. This could also be found in my inference kernel. We chose hidden_state_layers based on validation results.</li>\n<li>Differential learning rate, we used differential learning rate inspired by <a href=\"https://www.kaggle.com/melissarajaram/roberta-fastai-huggingface-transformers\">this great kernel</a>. With differential learning rate, our models got higher oof and converged faster.</li>\n<li>Two part model,  inspired from discussions, we trained models in two style. One is using title + question_body + answer as input for question labels and answer labels; one is use title + question_body for question labels and using title + answer for answer labels. The second style gives higher oof and lb, I think it's because more information for one problem. (Also using max_len 768 for xlnet gets higher oof is the same idea.</li>\n<li><p>Data augmentation, we used nlpaug for some basic data augmentation, also we used google translate to back translate some content from other languages. I decided to do this at very early stage, so I'm sorry I can't tell how much it could boost. It won't be much I think.</p>\n\n<h2>Things don't work for us:</h2></li>\n<li><p>T5</p></li>\n<li>Extra tokens</li>\n<li>Large models, large models gave similar oof comparing with base models and it's hard to inference, always slow and run out of resources. <br>\n<h2>Our best single model:</h2></li>\n</ol>\n\n<p><a href=\"https://www.kaggle.com/leonshangguan/roberta-single-models-with-optimization\">https://www.kaggle.com/leonshangguan/roberta-single-models-with-optimization</a></p>\n\n<h2>Our codes:</h2>\n\n<p><a href=\"https://github.com/jionie/Google-Quest-Answer\">https://github.com/jionie/Google-Quest-Answer</a></p>",
  "comments": [
    {
      "id": 744719,
      "author_name": "Ivan Panshin",
      "author_url": "",
      "post_date": "2020-02-13T04:20:54.313000",
      "content": "<p>In fact, we also tried to make stacking to work for many of our models, but to no avail. </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 742201,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2020-02-11T06:33:04.083000",
      "content": "<p>congrats and thanks for sharing your solution! <a href=\"/jionie\">@jionie</a> </p>",
      "votes": 1,
      "replies": [
        {
          "id": 742704,
          "author_name": "jionie",
          "author_url": "",
          "post_date": "2020-02-11T13:54:14.570000",
          "content": "<p>Thanks!!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 742093,
      "author_name": "chizhu",
      "author_url": "",
      "post_date": "2020-02-11T04:29:17.330000",
      "content": "<p>Congratulations and thank you for detail sharing</p>",
      "votes": 1,
      "replies": [
        {
          "id": 742709,
          "author_name": "jionie",
          "author_url": "",
          "post_date": "2020-02-11T13:56:00.413000",
          "content": "<p>Thanks! Learnt a lot from your team's solution, pesudo labeling is very impressive!!!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 741990,
      "author_name": "Massoud Hosseinali",
      "author_url": "",
      "post_date": "2020-02-11T02:59:49.557000",
      "content": "<p>Congratulations.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 741997,
          "author_name": "jionie",
          "author_url": "",
          "post_date": "2020-02-11T03:05:20.720000",
          "content": "<p>Thanks!!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 741988,
      "author_name": "ilovescience",
      "author_url": "",
      "post_date": "2020-02-11T02:58:10.623000",
      "content": "<p>Your readme indicates usage of SWA?</p>",
      "votes": 2,
      "replies": [
        {
          "id": 741996,
          "author_name": "jionie",
          "author_url": "",
          "post_date": "2020-02-11T03:05:02.303000",
          "content": "<p>I don't know how to use the official codes for swa, so I wrote a simple version for every epoch weights = ((epochs - 1) * (weights in last epoch) + weights in this epoch) / epoch; since I don't know it's correct or not so I didn't write it in solution. I used very small lr for that stage, it could boost a little for oof like 0.0005 to 0.002 in each fold.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 742006,
          "author_name": "ilovescience",
          "author_url": "",
          "post_date": "2020-02-11T03:13:57.997000",
          "content": "<p>SWA is supported in PyTorch <a href=\"https://github.com/pytorch/contrib/blob/master/torchcontrib/optim/swa.py\">over here</a> but that sounds like it would work. So this was used in your final solution?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 742013,
          "author_name": "jionie",
          "author_url": "",
          "post_date": "2020-02-11T03:17:40.580000",
          "content": "<p>I used models after my \"swa\" stage, lol. Thanks for your swa link! It looks promising, I found many people in cv competitions used swa so I tried to apply it here.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 742027,
          "author_name": "ilovescience",
          "author_url": "",
          "post_date": "2020-02-11T03:27:32.170000",
          "content": "<p>I tried SWA on BERT with CyclicLR with noticeable CV gains. I think basically if you have the cycle end at each epoch, and update the weights at the end of the epoch, then that would work nicely. SWA definitely has a lot of promise.</p>",
          "votes": 3,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 12,
  "messages": [
    {
      "id": 741981,
      "postDate": "2020-02-11T02:53:18.407Z",
      "content": "<p><strong>Congrats to all winners, and thanks to my teammates</strong>! It's a long journey to finish this competition and get a silver medal, it's the first time for me to work with nlp projects.</p>\n\n<h2>Our solution could be summarized as:</h2>\n\n<ol>\n<li>Post processing, our idea is to group values to discrete values existing in training set. We chose columns to apply post processing by comparing oof.  Post processing could be found in <a href=\"https://www.kaggle.com/jionie/models-with-optimization-v5/\">my inference kernel</a></li>\n<li>Ensemble, our final submission consists of 8 base models as you can see in the inference kernel, the best model is two part roberta base model which got 0.412 oof without post processing and 0.439 lb with post processing.</li>\n<li>Modified pooler layer and multiple hidden_states , we used multiple hidden_states for the input of pooler layer inspired from <a href=\"https://www.kaggle.com/c/google-quest-challenge/discussion/123770\">this great post</a> we followed source codes in transformers for pooler class. This could also be found in my inference kernel. We chose hidden_state_layers based on validation results.</li>\n<li>Differential learning rate, we used differential learning rate inspired by <a href=\"https://www.kaggle.com/melissarajaram/roberta-fastai-huggingface-transformers\">this great kernel</a>. With differential learning rate, our models got higher oof and converged faster.</li>\n<li>Two part model,  inspired from discussions, we trained models in two style. One is using title + question_body + answer as input for question labels and answer labels; one is use title + question_body for question labels and using title + answer for answer labels. The second style gives higher oof and lb, I think it's because more information for one problem. (Also using max_len 768 for xlnet gets higher oof is the same idea.</li>\n<li><p>Data augmentation, we used nlpaug for some basic data augmentation, also we used google translate to back translate some content from other languages. I decided to do this at very early stage, so I'm sorry I can't tell how much it could boost. It won't be much I think.</p>\n\n<h2>Things don't work for us:</h2></li>\n<li><p>T5</p></li>\n<li>Extra tokens</li>\n<li>Large models, large models gave similar oof comparing with base models and it's hard to inference, always slow and run out of resources. <br>\n<h2>Our best single model:</h2></li>\n</ol>\n\n<p><a href=\"https://www.kaggle.com/leonshangguan/roberta-single-models-with-optimization\">https://www.kaggle.com/leonshangguan/roberta-single-models-with-optimization</a></p>\n\n<h2>Our codes:</h2>\n\n<p><a href=\"https://github.com/jionie/Google-Quest-Answer\">https://github.com/jionie/Google-Quest-Answer</a></p>",
      "rawMarkdown": "**Congrats to all winners, and thanks to my teammates**! It's a long journey to finish this competition and get a silver medal, it's the first time for me to work with nlp projects.\n## Our solution could be summarized as:\n1. Post processing, our idea is to group values to discrete values existing in training set. We chose columns to apply post processing by comparing oof.  Post processing could be found in [my inference kernel](https://www.kaggle.com/jionie/models-with-optimization-v5/)\n1. Ensemble, our final submission consists of 8 base models as you can see in the inference kernel, the best model is two part roberta base model which got 0.412 oof without post processing and 0.439 lb with post processing.\n1. Modified pooler layer and multiple hidden\\_states , we used multiple hidden\\_states for the input of pooler layer inspired from [this great post](https://www.kaggle.com/c/google-quest-challenge/discussion/123770) we followed source codes in transformers for pooler class. This could also be found in my inference kernel. We chose hidden\\_state\\_layers based on validation results.\n1. Differential learning rate, we used differential learning rate inspired by [this great kernel](https://www.kaggle.com/melissarajaram/roberta-fastai-huggingface-transformers). With differential learning rate, our models got higher oof and converged faster.\n1. Two part model,  inspired from discussions, we trained models in two style. One is using title + question\\_body + answer as input for question labels and answer labels; one is use title + question\\_body for question labels and using title + answer for answer labels. The second style gives higher oof and lb, I think it's because more information for one problem. (Also using max\\_len 768 for xlnet gets higher oof is the same idea.\n1. Data augmentation, we used nlpaug for some basic data augmentation, also we used google translate to back translate some content from other languages. I decided to do this at very early stage, so I'm sorry I can't tell how much it could boost. It won't be much I think.\n## Things don't work for us:\n1. T5\n1. Extra tokens\n1. Large models, large models gave similar oof comparing with base models and it's hard to inference, always slow and run out of resources.  \n## Our best single model:\nhttps://www.kaggle.com/leonshangguan/roberta-single-models-with-optimization\n## Our codes:\nhttps://github.com/jionie/Google-Quest-Answer\n",
      "votes": 36
    },
    {
      "id": 744719,
      "postDate": "2020-02-13T04:20:54.313Z",
      "content": "<p>In fact, we also tried to make stacking to work for many of our models, but to no avail. </p>",
      "rawMarkdown": "In fact, we also tried to make stacking to work for many of our models, but to no avail. ",
      "votes": 1
    },
    {
      "id": 742201,
      "postDate": "2020-02-11T06:33:04.083Z",
      "content": "<p>congrats and thanks for sharing your solution! <a href=\"/jionie\">@jionie</a> </p>",
      "rawMarkdown": "congrats and thanks for sharing your solution! @jionie ",
      "votes": 1,
      "replies": [
        {
          "id": 742704,
          "postDate": "2020-02-11T13:54:14.570Z",
          "content": "<p>Thanks!!</p>",
          "rawMarkdown": "Thanks!!",
          "votes": 1
        }
      ]
    },
    {
      "id": 742093,
      "postDate": "2020-02-11T04:29:17.330Z",
      "content": "<p>Congratulations and thank you for detail sharing</p>",
      "rawMarkdown": "Congratulations and thank you for detail sharing",
      "votes": 1,
      "replies": [
        {
          "id": 742709,
          "postDate": "2020-02-11T13:56:00.413Z",
          "content": "<p>Thanks! Learnt a lot from your team's solution, pesudo labeling is very impressive!!!</p>",
          "rawMarkdown": "Thanks! Learnt a lot from your team's solution, pesudo labeling is very impressive!!!",
          "votes": 1
        }
      ]
    },
    {
      "id": 741990,
      "postDate": "2020-02-11T02:59:49.557Z",
      "content": "<p>Congratulations.</p>",
      "rawMarkdown": "Congratulations.",
      "votes": 1,
      "replies": [
        {
          "id": 741997,
          "postDate": "2020-02-11T03:05:20.720Z",
          "content": "<p>Thanks!!</p>",
          "rawMarkdown": "Thanks!!",
          "votes": 1
        }
      ]
    },
    {
      "id": 741988,
      "postDate": "2020-02-11T02:58:10.623Z",
      "content": "<p>Your readme indicates usage of SWA?</p>",
      "rawMarkdown": "Your readme indicates usage of SWA?",
      "votes": 2,
      "replies": [
        {
          "id": 741996,
          "postDate": "2020-02-11T03:05:02.303Z",
          "content": "<p>I don't know how to use the official codes for swa, so I wrote a simple version for every epoch weights = ((epochs - 1) * (weights in last epoch) + weights in this epoch) / epoch; since I don't know it's correct or not so I didn't write it in solution. I used very small lr for that stage, it could boost a little for oof like 0.0005 to 0.002 in each fold.</p>",
          "rawMarkdown": "I don't know how to use the official codes for swa, so I wrote a simple version for every epoch weights = ((epochs - 1) * (weights in last epoch) + weights in this epoch) / epoch; since I don't know it's correct or not so I didn't write it in solution. I used very small lr for that stage, it could boost a little for oof like 0.0005 to 0.002 in each fold.",
          "votes": 1
        },
        {
          "id": 742006,
          "postDate": "2020-02-11T03:13:57.997Z",
          "content": "<p>SWA is supported in PyTorch <a href=\"https://github.com/pytorch/contrib/blob/master/torchcontrib/optim/swa.py\">over here</a> but that sounds like it would work. So this was used in your final solution?</p>",
          "rawMarkdown": "SWA is supported in PyTorch [over here](https://github.com/pytorch/contrib/blob/master/torchcontrib/optim/swa.py) but that sounds like it would work. So this was used in your final solution?",
          "votes": 1
        },
        {
          "id": 742013,
          "postDate": "2020-02-11T03:17:40.580Z",
          "content": "<p>I used models after my \"swa\" stage, lol. Thanks for your swa link! It looks promising, I found many people in cv competitions used swa so I tried to apply it here.</p>",
          "rawMarkdown": "I used models after my \"swa\" stage, lol. Thanks for your swa link! It looks promising, I found many people in cv competitions used swa so I tried to apply it here.",
          "votes": 1
        },
        {
          "id": 742027,
          "postDate": "2020-02-11T03:27:32.170Z",
          "content": "<p>I tried SWA on BERT with CyclicLR with noticeable CV gains. I think basically if you have the cycle end at each epoch, and update the weights at the end of the epoch, then that would work nicely. SWA definitely has a lot of promise.</p>",
          "rawMarkdown": "I tried SWA on BERT with CyclicLR with noticeable CV gains. I think basically if you have the cycle end at each epoch, and update the weights at the end of the epoch, then that would work nicely. SWA definitely has a lot of promise.",
          "votes": 3
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:07:36Z",
  "competition_id": 7968,
  "competition_slug": "google-quest-challenge",
  "team_id": 4074279,
  "solution_url": "/competitions/google-quest-challenge/writeups/jionie-15th-solution-with-codes"
}