{
  "id": 90483,
  "title": "11th place solution",
  "content": "<h2>Base model</h2>\n\n<p>I have 2 base model all coming from public kernel. </p>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline\">Taming the BERT</a> -&gt; <code>Model 1</code></li>\n<li><a href=\"https://www.kaggle.com/chanhu/bert-score-layer-lb-0-475\">Bert + Score Layer </a> -&gt; <code>Model 2</code></li>\n</ul>\n\n<p>Thanks for their selfless sharing.</p>\n\n<h2>Modification</h2>\n\n<ol>\n<li>Using large-bert instead of base-bert(~0.04).</li>\n<li>Concat last 8 layers instead of last layer(~0.1).  Here I made some attempts. If I increase the number of layers after the 8th floor, it will not be much better.</li>\n<li>Concat uncased-bert and cased-bert(~0.04). In fact, they can also be used separately in different models and then merged, but doing so will increase the number of models, I don't have time to adjust them one by one, so I used them directly together. If you want to go deeper on ensemble, use uncased model and cased model separately to provide better diversity. Or instead of using the concat method but using the averaging method. But based on my experience in <code>Quora</code> and <code>Jigsaw</code>, this method usually performs slightly worse, at least on local-cv.</li>\n<li>Using 10fold-cv instead of 5fold-cv(~0.015). The promotion is much bigger than I expected. Maybe it’s because the training data is too small, I guess A-A.</li>\n<li>Adding  <code>A*P, B*P</code>  (~0.05), only for <code>Model 1</code>. I tried some combinations like <code>A + B</code>, <code>A - B</code>, etc. In the end, only this set of features has an effect. This finally stabilized the gap between <code>Model 1</code> and <code>Model 2</code> at 0.02.</li>\n<li>Tuning mlp and ffnn(~0.01). In fact, this step is very important, and it is executed earlier than other steps. It ensures that each fold can converge normally and avoid over-fitting problems caused by excessive use of bert. </li>\n</ol>\n\n<h2>Ensemble</h2>\n\n<p>I chose four models for ensemble: <code>Model 1</code> * 1, <code>Model 2</code> * 3(Architecture is slightly different). These four models are handed over to lgb and lasso for ensemble. The final result is as follows:\n- LGB - CV 0.3059 - LB 0.2047\n- Lasso - CV 0.3074 - LB 0.2336</p>\n\n<h2>What didn't work</h2>\n\n<ul>\n<li>Fine-tuning. Until now, I don’t know the correct posture of fine-tuning, it’s very awkward.</li>\n<li>Correct the wrong label. </li>\n<li>Elmo.</li>\n<li>Siamese network. Later I found out that it is helpful to solve this problem, but it requires some skill when using it.</li>\n</ul>\n\n<h2>Written at the end</h2>\n\n<ul>\n<li>Thanks to Google AI and Kaggle for organizing this research competition giving everyone the opportunity to try the LM. During Quora, I often wondered how good it would be if I could use bert. The only pity is that I had a lot of time at the time but the rules didn't allow it, but this time I didn't have a lot of time to try LM.</li>\n<li>I think augmentation is also important in addition to fine tuning. I am even curious as to whether the promotion brought by fine tuning will actually be larger than some of the structures or features specifically designed for this problem.</li>\n<li>I think the name of my team is really good, isn't it?</li>\n</ul>\n\n<p>See you in Jigsaw. Have a good time.</p>",
  "comments": [
    {
      "id": 522330,
      "author_name": "Yury Kashnitsky",
      "author_url": "",
      "post_date": "2019-04-24T08:47:56.530000",
      "content": "<p>Actually, we had exactly the same problems - with fine-tuning and augmentations. Cool that you coped with stacking. Alone while there were 5 of us :)</p>",
      "votes": 3,
      "replies": [
        {
          "id": 523088,
          "author_name": "Hikkiiiiiiiii",
          "author_url": "",
          "post_date": "2019-04-25T14:59:19.507000",
          "content": "<p>It's really a pity. I think you just lack a little bit of luck. All zero problems sometimes make us look less motivated. Nevertheless, your contribution to the forum is obvious to all. Looking forward to your performance in Jigsaw. I hope we can find the right way of find-tuning there A-A. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 522854,
      "author_name": "Himanshu Soni",
      "author_url": "",
      "post_date": "2019-04-25T06:26:34.857000",
      "content": "<p>How you choose ensembling parameters so that your score is quite good? </p>",
      "votes": 1,
      "replies": [
        {
          "id": 523095,
          "author_name": "Hikkiiiiiiiii",
          "author_url": "",
          "post_date": "2019-04-25T15:10:39.027000",
          "content": "<p>There are two main aspects:\n1. Use BN to avoid some fold non-convergence problems.\n2. Because the over-fitting is very serious, dropout can be added directly after bert-feature. And it can add up a lot like <code>Dropout(0.9)</code></p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 522693,
      "author_name": "Kepler456b",
      "author_url": "",
      "post_date": "2019-04-24T20:38:08.420000",
      "content": "<p>Congrats and thanks!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 523090,
          "author_name": "Hikkiiiiiiiii",
          "author_url": "",
          "post_date": "2019-04-25T15:00:19.607000",
          "content": "<p>Thanks~</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 522514,
      "author_name": "CPMP",
      "author_url": "",
      "post_date": "2019-04-24T14:20:16.563000",
      "content": "<p>Congrats. Glad to see you're back to Kaggle!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 523092,
          "author_name": "Hikkiiiiiiiii",
          "author_url": "",
          "post_date": "2019-04-25T15:07:12.183000",
          "content": "<p>Thank you very much for your congratulations. In fact, up to now I still don't know how to face you and Giba ... So this message really means a lot to me. One of the reasons I chose to stay in Kaggle was to hope that one day I could make up for what I had done before. Now it seems that things are really heading in the right direction. Thanks ; )</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 523212,
          "author_name": "CPMP",
          "author_url": "",
          "post_date": "2019-04-25T18:33:31.853000",
          "content": "<p>Don't worry, it was clear you didn't mean to harm us.</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 523374,
      "author_name": "Prashanth Thangavel",
      "author_url": "",
      "post_date": "2019-04-26T05:50:03.583000",
      "content": "<p>Congratulations! Thanks for sharing the approach. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 523091,
      "author_name": "Hamish",
      "author_url": "",
      "post_date": "2019-04-25T15:04:12.617000",
      "content": "<p>congrats and great work!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 522328,
      "author_name": "Yury Kashnitsky",
      "author_url": "",
      "post_date": "2019-04-24T08:41:49.680000",
      "content": "<p>Congrats with 🏅 and thanks for the summary!\nYour solution is very-very similar to ours :) One more takeaway for me is to build OOFs (for stacking) with our <em>best</em> models. We combined two experiments: stacking and augmentation. So most probably stacking didn't work for us because augmentations didn't work out well. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 523417,
      "author_name": "Chanhu",
      "author_url": "",
      "post_date": "2019-04-26T08:01:50.903000",
      "content": "<p>Great Work! Thanks for sharing.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 13,
  "messages": [
    {
      "id": 522301,
      "postDate": "2019-04-24T07:36:00.073Z",
      "content": "<h2>Base model</h2>\n\n<p>I have 2 base model all coming from public kernel. </p>\n\n<ul>\n<li><a href=\"https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline\">Taming the BERT</a> -&gt; <code>Model 1</code></li>\n<li><a href=\"https://www.kaggle.com/chanhu/bert-score-layer-lb-0-475\">Bert + Score Layer </a> -&gt; <code>Model 2</code></li>\n</ul>\n\n<p>Thanks for their selfless sharing.</p>\n\n<h2>Modification</h2>\n\n<ol>\n<li>Using large-bert instead of base-bert(~0.04).</li>\n<li>Concat last 8 layers instead of last layer(~0.1).  Here I made some attempts. If I increase the number of layers after the 8th floor, it will not be much better.</li>\n<li>Concat uncased-bert and cased-bert(~0.04). In fact, they can also be used separately in different models and then merged, but doing so will increase the number of models, I don't have time to adjust them one by one, so I used them directly together. If you want to go deeper on ensemble, use uncased model and cased model separately to provide better diversity. Or instead of using the concat method but using the averaging method. But based on my experience in <code>Quora</code> and <code>Jigsaw</code>, this method usually performs slightly worse, at least on local-cv.</li>\n<li>Using 10fold-cv instead of 5fold-cv(~0.015). The promotion is much bigger than I expected. Maybe it’s because the training data is too small, I guess A-A.</li>\n<li>Adding  <code>A*P, B*P</code>  (~0.05), only for <code>Model 1</code>. I tried some combinations like <code>A + B</code>, <code>A - B</code>, etc. In the end, only this set of features has an effect. This finally stabilized the gap between <code>Model 1</code> and <code>Model 2</code> at 0.02.</li>\n<li>Tuning mlp and ffnn(~0.01). In fact, this step is very important, and it is executed earlier than other steps. It ensures that each fold can converge normally and avoid over-fitting problems caused by excessive use of bert. </li>\n</ol>\n\n<h2>Ensemble</h2>\n\n<p>I chose four models for ensemble: <code>Model 1</code> * 1, <code>Model 2</code> * 3(Architecture is slightly different). These four models are handed over to lgb and lasso for ensemble. The final result is as follows:\n- LGB - CV 0.3059 - LB 0.2047\n- Lasso - CV 0.3074 - LB 0.2336</p>\n\n<h2>What didn't work</h2>\n\n<ul>\n<li>Fine-tuning. Until now, I don’t know the correct posture of fine-tuning, it’s very awkward.</li>\n<li>Correct the wrong label. </li>\n<li>Elmo.</li>\n<li>Siamese network. Later I found out that it is helpful to solve this problem, but it requires some skill when using it.</li>\n</ul>\n\n<h2>Written at the end</h2>\n\n<ul>\n<li>Thanks to Google AI and Kaggle for organizing this research competition giving everyone the opportunity to try the LM. During Quora, I often wondered how good it would be if I could use bert. The only pity is that I had a lot of time at the time but the rules didn't allow it, but this time I didn't have a lot of time to try LM.</li>\n<li>I think augmentation is also important in addition to fine tuning. I am even curious as to whether the promotion brought by fine tuning will actually be larger than some of the structures or features specifically designed for this problem.</li>\n<li>I think the name of my team is really good, isn't it?</li>\n</ul>\n\n<p>See you in Jigsaw. Have a good time.</p>",
      "rawMarkdown": "## Base model\nI have 2 base model all coming from public kernel. \n\n- [Taming the BERT](https://www.kaggle.com/mateiionita/taming-the-bert-a-baseline) -&gt; `Model 1`\n- [Bert + Score Layer ](https://www.kaggle.com/chanhu/bert-score-layer-lb-0-475) -&gt; `Model 2`\n\nThanks for their selfless sharing.\n\n## Modification\n1. Using large-bert instead of base-bert(~0.04).\n2. Concat last 8 layers instead of last layer(~0.1).  Here I made some attempts. If I increase the number of layers after the 8th floor, it will not be much better.\n3. Concat uncased-bert and cased-bert(~0.04). In fact, they can also be used separately in different models and then merged, but doing so will increase the number of models, I don't have time to adjust them one by one, so I used them directly together. If you want to go deeper on ensemble, use uncased model and cased model separately to provide better diversity. Or instead of using the concat method but using the averaging method. But based on my experience in `Quora` and `Jigsaw`, this method usually performs slightly worse, at least on local-cv.\n4. Using 10fold-cv instead of 5fold-cv(~0.015). The promotion is much bigger than I expected. Maybe it’s because the training data is too small, I guess A-A.\n5. Adding  `A*P, B*P`  (~0.05), only for `Model 1`. I tried some combinations like `A + B`, `A - B`, etc. In the end, only this set of features has an effect. This finally stabilized the gap between `Model 1` and `Model 2` at 0.02.\n6. Tuning mlp and ffnn(~0.01). In fact, this step is very important, and it is executed earlier than other steps. It ensures that each fold can converge normally and avoid over-fitting problems caused by excessive use of bert. \n\n## Ensemble\nI chose four models for ensemble: `Model 1` * 1, `Model 2` * 3(Architecture is slightly different). These four models are handed over to lgb and lasso for ensemble. The final result is as follows:\n- LGB - CV 0.3059 - LB 0.2047\n- Lasso - CV 0.3074 - LB 0.2336\n\n## What didn't work\n- Fine-tuning. Until now, I don’t know the correct posture of fine-tuning, it’s very awkward.\n- Correct the wrong label. \n- Elmo.\n- Siamese network. Later I found out that it is helpful to solve this problem, but it requires some skill when using it.\n\n## Written at the end\n- Thanks to Google AI and Kaggle for organizing this research competition giving everyone the opportunity to try the LM. During Quora, I often wondered how good it would be if I could use bert. The only pity is that I had a lot of time at the time but the rules didn't allow it, but this time I didn't have a lot of time to try LM.\n- I think augmentation is also important in addition to fine tuning. I am even curious as to whether the promotion brought by fine tuning will actually be larger than some of the structures or features specifically designed for this problem.\n- I think the name of my team is really good, isn't it?\n\n\nSee you in Jigsaw. Have a good time.",
      "votes": 21
    },
    {
      "id": 522330,
      "postDate": "2019-04-24T08:47:56.530Z",
      "content": "<p>Actually, we had exactly the same problems - with fine-tuning and augmentations. Cool that you coped with stacking. Alone while there were 5 of us :)</p>",
      "rawMarkdown": "Actually, we had exactly the same problems - with fine-tuning and augmentations. Cool that you coped with stacking. Alone while there were 5 of us :)",
      "votes": 3,
      "replies": [
        {
          "id": 523088,
          "postDate": "2019-04-25T14:59:19.507Z",
          "content": "<p>It's really a pity. I think you just lack a little bit of luck. All zero problems sometimes make us look less motivated. Nevertheless, your contribution to the forum is obvious to all. Looking forward to your performance in Jigsaw. I hope we can find the right way of find-tuning there A-A. </p>",
          "rawMarkdown": "It's really a pity. I think you just lack a little bit of luck. All zero problems sometimes make us look less motivated. Nevertheless, your contribution to the forum is obvious to all. Looking forward to your performance in Jigsaw. I hope we can find the right way of find-tuning there A-A. "
        }
      ]
    },
    {
      "id": 522854,
      "postDate": "2019-04-25T06:26:34.857Z",
      "content": "<p>How you choose ensembling parameters so that your score is quite good? </p>",
      "rawMarkdown": "How you choose ensembling parameters so that your score is quite good? ",
      "votes": 1,
      "replies": [
        {
          "id": 523095,
          "postDate": "2019-04-25T15:10:39.027Z",
          "content": "<p>There are two main aspects:\n1. Use BN to avoid some fold non-convergence problems.\n2. Because the over-fitting is very serious, dropout can be added directly after bert-feature. And it can add up a lot like <code>Dropout(0.9)</code></p>",
          "rawMarkdown": "There are two main aspects:\n1. Use BN to avoid some fold non-convergence problems.\n2. Because the over-fitting is very serious, dropout can be added directly after bert-feature. And it can add up a lot like `Dropout(0.9)`",
          "votes": 1
        }
      ]
    },
    {
      "id": 522693,
      "postDate": "2019-04-24T20:38:08.420Z",
      "content": "<p>Congrats and thanks!</p>",
      "rawMarkdown": "Congrats and thanks!",
      "votes": 1,
      "replies": [
        {
          "id": 523090,
          "postDate": "2019-04-25T15:00:19.607Z",
          "content": "<p>Thanks~</p>",
          "rawMarkdown": "Thanks~"
        }
      ]
    },
    {
      "id": 522514,
      "postDate": "2019-04-24T14:20:16.563Z",
      "content": "<p>Congrats. Glad to see you're back to Kaggle!</p>",
      "rawMarkdown": "Congrats. Glad to see you're back to Kaggle!",
      "votes": 2,
      "replies": [
        {
          "id": 523092,
          "postDate": "2019-04-25T15:07:12.183Z",
          "content": "<p>Thank you very much for your congratulations. In fact, up to now I still don't know how to face you and Giba ... So this message really means a lot to me. One of the reasons I chose to stay in Kaggle was to hope that one day I could make up for what I had done before. Now it seems that things are really heading in the right direction. Thanks ; )</p>",
          "rawMarkdown": "Thank you very much for your congratulations. In fact, up to now I still don't know how to face you and Giba ... So this message really means a lot to me. One of the reasons I chose to stay in Kaggle was to hope that one day I could make up for what I had done before. Now it seems that things are really heading in the right direction. Thanks ; )",
          "votes": 2
        },
        {
          "id": 523212,
          "postDate": "2019-04-25T18:33:31.853Z",
          "content": "<p>Don't worry, it was clear you didn't mean to harm us.</p>",
          "rawMarkdown": "Don't worry, it was clear you didn't mean to harm us.",
          "votes": 3
        }
      ]
    },
    {
      "id": 523374,
      "postDate": "2019-04-26T05:50:03.583Z",
      "content": "<p>Congratulations! Thanks for sharing the approach. </p>",
      "rawMarkdown": "Congratulations! Thanks for sharing the approach. "
    },
    {
      "id": 523091,
      "postDate": "2019-04-25T15:04:12.617Z",
      "content": "<p>congrats and great work!</p>",
      "rawMarkdown": "congrats and great work!"
    },
    {
      "id": 522328,
      "postDate": "2019-04-24T08:41:49.680Z",
      "content": "<p>Congrats with 🏅 and thanks for the summary!\nYour solution is very-very similar to ours :) One more takeaway for me is to build OOFs (for stacking) with our <em>best</em> models. We combined two experiments: stacking and augmentation. So most probably stacking didn't work for us because augmentations didn't work out well. </p>",
      "rawMarkdown": "Congrats with 🏅 and thanks for the summary!\nYour solution is very-very similar to ours :) One more takeaway for me is to build OOFs (for stacking) with our *best* models. We combined two experiments: stacking and augmentation. So most probably stacking didn't work for us because augmentations didn't work out well. "
    },
    {
      "id": 523417,
      "postDate": "2019-04-26T08:01:50.903Z",
      "content": "<p>Great Work! Thanks for sharing.</p>",
      "rawMarkdown": "Great Work! Thanks for sharing."
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T22:43:40Z",
  "competition_id": 12797,
  "competition_slug": "gendered-pronoun-resolution",
  "team_id": 2748264,
  "solution_url": "/competitions/gendered-pronoun-resolution/writeups/kobe-is-better-11th-place-solution"
}