{
  "id": 130167,
  "title": "9th public and private solution: 30 models, 1 for each column?",
  "content": "<p>First of all, I'd thank the organizers and my teammate, <a href=\"/murphy89\">@murphy89</a>  for this competition, and congrats to all the teams participating. I know the metric is not ideal, but I guess we all gained something from the competition.\nSecondly, well, that is a click bait title, but we did something sort of the same when coming to blending parts. Anyway here is the overview of our solution:</p>\n\n<h1>Overview</h1>\n\n<ul>\n<li><p>There is no special pre-processing of text, just basic cleaning like replacing &amp; with &amp; and &lt; with &lt; and some more broken signs.</p>\n\n<h2>Models</h2></li>\n<li><p>We all split the models into 2 parts: models for predicting first 21 question columns and for last 9 answer columns.</p>\n\n<h3>My models</h3></li>\n</ul>\n\n<p><strong>Question part</strong></p>\n\n<p>Question models take only text from question title and question body. If the text length is long, use head and tail method.\nUsing special tokens like site pages, categories...\nFreezing last 4 layers, concatenating them and feeding into 1-layer LSTM before taking the average pool and max pool. Then 1-2 more FC layers at the end.\nI used bert base, bert large and xlnet. Then from each model I vary a bit like different layers to freeze or instead of concatenating layers, just take the sum.\nSum of BCELogitsLoss on each column work better for me. I guess it's similar to weighted loss.</p>\n\n<p><strong>Answer part</strong></p>\n\n<p>Answer models take text from all 3 columns and also use head-tail method.\nNot using special tokens, as my CV is worse with special tokens in answer part.\nFreezing last 4 layers, and doing almost the same as question part, except that this time it's most ly 2-layer BiLSTM.\nMostly bert base and bert large of simply varying structures from the main model.\nNormal BCELogitsLoss.  </p>\n\n<h2>Blending</h2>\n\n<p>I believe this is the key part in boosting our score. We discovered the right blending strategy only on the last day, when we saved the oof predictions and cross validated offline. Although our single models have different CV score 0.411-0.426, they seemed better when blending together at certain weights. Later I found out that each model better in predicting some columns, so I use heuristic search to search for best CV score of each column. For those columns that depending heavily on post-processing, the search is on the final values after post-processing. This is why there is such title: 30 models, since I believe tuning some specific models for some columns would be better with post-processing part. In fact, some models with initial lower CV score seem to have higher weightage in this part. The best CV score after blending is 0.435, and after post-processing is 0.478.\nThe only regret is that when we were doing this, it was only 4 hours till the deadline, so some of our columns are not maximized in the final solution. In fact, when I tried again after the deadline, our CV could reach 0.482 with post-processing.</p>\n\n<h2>Post-processing</h2>\n\n<p>Our post-processing is not different from others'; on the contrary, I think it's even less effective since it's the simple rounding of predictions of some columns to the nearest values with certain thresholds. </p>\n\n<h2>What did not work for me</h2>\n\n<ul>\n<li>Translation augmentation. I had translated back and forth in French and Italian, but adding them did not help.</li>\n<li>SWA.</li>\n<li>Multi-dropout for me, but it worked for my teammate's, Morphy, architecture.</li>\n<li>Other models: xlnet-large, albert... The scores are worse in my case.</li>\n</ul>",
  "comments": [
    {
      "id": 744981,
      "author_name": "MaChaogong",
      "author_url": "",
      "post_date": "2020-02-13T10:56:09.900000",
      "content": "<p>Really nice blending! thanks for sharing</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 744753,
      "author_name": "Rehan Guha",
      "author_url": "",
      "post_date": "2020-02-13T05:21:28.863000",
      "content": "<p>Congratulations... </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 744373,
      "author_name": "Melih Kanbay",
      "author_url": "",
      "post_date": "2020-02-12T19:17:51.420000",
      "content": "<p>congratulations</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 744664,
      "author_name": "Ailurophile",
      "author_url": "",
      "post_date": "2020-02-13T02:59:37.270000",
      "content": "<p>Congratulations!\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/ajinomoto132\">@ajinomoto132</a> </p>",
      "votes": 0,
      "replies": [
        {
          "id": 744935,
          "author_name": "Phi",
          "author_url": "",
          "post_date": "2020-02-13T10:03:24.300000",
          "content": "<p>thanks, it's my pleasure 😄 </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 745012,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-13T11:46:02.743000",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 6,
  "messages": [
    {
      "id": 744042,
      "postDate": "2020-02-12T14:04:14.283Z",
      "content": "<p>First of all, I'd thank the organizers and my teammate, <a href=\"/murphy89\">@murphy89</a>  for this competition, and congrats to all the teams participating. I know the metric is not ideal, but I guess we all gained something from the competition.\nSecondly, well, that is a click bait title, but we did something sort of the same when coming to blending parts. Anyway here is the overview of our solution:</p>\n\n<h1>Overview</h1>\n\n<ul>\n<li><p>There is no special pre-processing of text, just basic cleaning like replacing &amp; with &amp; and &lt; with &lt; and some more broken signs.</p>\n\n<h2>Models</h2></li>\n<li><p>We all split the models into 2 parts: models for predicting first 21 question columns and for last 9 answer columns.</p>\n\n<h3>My models</h3></li>\n</ul>\n\n<p><strong>Question part</strong></p>\n\n<p>Question models take only text from question title and question body. If the text length is long, use head and tail method.\nUsing special tokens like site pages, categories...\nFreezing last 4 layers, concatenating them and feeding into 1-layer LSTM before taking the average pool and max pool. Then 1-2 more FC layers at the end.\nI used bert base, bert large and xlnet. Then from each model I vary a bit like different layers to freeze or instead of concatenating layers, just take the sum.\nSum of BCELogitsLoss on each column work better for me. I guess it's similar to weighted loss.</p>\n\n<p><strong>Answer part</strong></p>\n\n<p>Answer models take text from all 3 columns and also use head-tail method.\nNot using special tokens, as my CV is worse with special tokens in answer part.\nFreezing last 4 layers, and doing almost the same as question part, except that this time it's most ly 2-layer BiLSTM.\nMostly bert base and bert large of simply varying structures from the main model.\nNormal BCELogitsLoss.  </p>\n\n<h2>Blending</h2>\n\n<p>I believe this is the key part in boosting our score. We discovered the right blending strategy only on the last day, when we saved the oof predictions and cross validated offline. Although our single models have different CV score 0.411-0.426, they seemed better when blending together at certain weights. Later I found out that each model better in predicting some columns, so I use heuristic search to search for best CV score of each column. For those columns that depending heavily on post-processing, the search is on the final values after post-processing. This is why there is such title: 30 models, since I believe tuning some specific models for some columns would be better with post-processing part. In fact, some models with initial lower CV score seem to have higher weightage in this part. The best CV score after blending is 0.435, and after post-processing is 0.478.\nThe only regret is that when we were doing this, it was only 4 hours till the deadline, so some of our columns are not maximized in the final solution. In fact, when I tried again after the deadline, our CV could reach 0.482 with post-processing.</p>\n\n<h2>Post-processing</h2>\n\n<p>Our post-processing is not different from others'; on the contrary, I think it's even less effective since it's the simple rounding of predictions of some columns to the nearest values with certain thresholds. </p>\n\n<h2>What did not work for me</h2>\n\n<ul>\n<li>Translation augmentation. I had translated back and forth in French and Italian, but adding them did not help.</li>\n<li>SWA.</li>\n<li>Multi-dropout for me, but it worked for my teammate's, Morphy, architecture.</li>\n<li>Other models: xlnet-large, albert... The scores are worse in my case.</li>\n</ul>",
      "rawMarkdown": "First of all, I'd thank the organizers and my teammate, @murphy89  for this competition, and congrats to all the teams participating. I know the metric is not ideal, but I guess we all gained something from the competition.\nSecondly, well, that is a click bait title, but we did something sort of the same when coming to blending parts. Anyway here is the overview of our solution:\n# Overview\n- There is no special pre-processing of text, just basic cleaning like replacing &amp; with &amp; and &lt; with &lt; and some more broken signs.\n## Models\n- We all split the models into 2 parts: models for predicting first 21 question columns and for last 9 answer columns.\n### My models\n**Question part**\n\nQuestion models take only text from question title and question body. If the text length is long, use head and tail method.\nUsing special tokens like site pages, categories...\nFreezing last 4 layers, concatenating them and feeding into 1-layer LSTM before taking the average pool and max pool. Then 1-2 more FC layers at the end.\nI used bert base, bert large and xlnet. Then from each model I vary a bit like different layers to freeze or instead of concatenating layers, just take the sum.\nSum of BCELogitsLoss on each column work better for me. I guess it's similar to weighted loss.\n\n**Answer part**\n\nAnswer models take text from all 3 columns and also use head-tail method.\nNot using special tokens, as my CV is worse with special tokens in answer part.\nFreezing last 4 layers, and doing almost the same as question part, except that this time it's most ly 2-layer BiLSTM.\nMostly bert base and bert large of simply varying structures from the main model.\nNormal BCELogitsLoss.  \n## Blending\n\nI believe this is the key part in boosting our score. We discovered the right blending strategy only on the last day, when we saved the oof predictions and cross validated offline. Although our single models have different CV score 0.411-0.426, they seemed better when blending together at certain weights. Later I found out that each model better in predicting some columns, so I use heuristic search to search for best CV score of each column. For those columns that depending heavily on post-processing, the search is on the final values after post-processing. This is why there is such title: 30 models, since I believe tuning some specific models for some columns would be better with post-processing part. In fact, some models with initial lower CV score seem to have higher weightage in this part. The best CV score after blending is 0.435, and after post-processing is 0.478.\nThe only regret is that when we were doing this, it was only 4 hours till the deadline, so some of our columns are not maximized in the final solution. In fact, when I tried again after the deadline, our CV could reach 0.482 with post-processing.\n## Post-processing\n\nOur post-processing is not different from others'; on the contrary, I think it's even less effective since it's the simple rounding of predictions of some columns to the nearest values with certain thresholds. \n## What did not work for me\n- Translation augmentation. I had translated back and forth in French and Italian, but adding them did not help.\n- SWA.\n- Multi-dropout for me, but it worked for my teammate's, Morphy, architecture.\n- Other models: xlnet-large, albert... The scores are worse in my case.",
      "votes": 15
    },
    {
      "id": 744981,
      "postDate": "2020-02-13T10:56:09.900Z",
      "content": "<p>Really nice blending! thanks for sharing</p>",
      "rawMarkdown": "Really nice blending! thanks for sharing",
      "votes": 1
    },
    {
      "id": 744753,
      "postDate": "2020-02-13T05:21:28.863Z",
      "content": "<p>Congratulations... </p>",
      "rawMarkdown": "Congratulations... ",
      "votes": 1
    },
    {
      "id": 744373,
      "postDate": "2020-02-12T19:17:51.420Z",
      "content": "<p>congratulations</p>",
      "rawMarkdown": "congratulations",
      "votes": 2
    },
    {
      "id": 744664,
      "postDate": "2020-02-13T02:59:37.270Z",
      "content": "<p>Congratulations!\nThanks for Sharing your Approach &amp; Insights!! <a href=\"/ajinomoto132\">@ajinomoto132</a> </p>",
      "rawMarkdown": "Congratulations!\nThanks for Sharing your Approach &amp; Insights!! @ajinomoto132 ",
      "replies": [
        {
          "id": 744935,
          "postDate": "2020-02-13T10:03:24.300Z",
          "content": "<p>thanks, it's my pleasure 😄 </p>",
          "rawMarkdown": "thanks, it's my pleasure 😄 "
        }
      ]
    },
    {
      "id": 745012,
      "postDate": "2020-02-13T11:46:02.743Z",
      "content": "<p>Congratulations and thanks for sharing!</p>",
      "rawMarkdown": "Congratulations and thanks for sharing!",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:09:16Z",
  "competition_id": 7968,
  "competition_slug": "google-quest-challenge",
  "team_id": 4030934,
  "solution_url": "/competitions/google-quest-challenge/writeups/no-shake-for-us-9th-public-and-private-solution-30"
}