{
  "id": 288110,
  "title": "9th place solution",
  "content": "<p>Hi, somewhat surprisingly we achieved 9th place. Our aim for this competition was to try models trained with WECHSEL (<a href=\"https://openreview.net/forum?id=JcfISE1-u4\" target=\"_blank\">https://openreview.net/forum?id=JcfISE1-u4</a>, you've probably seen the models <a href=\"https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/286603\" target=\"_blank\">here</a>) as kind of a sanity-check of the method when compared to SOTA multilingual models. To make our submission competitive we combined the WECHSEL models with some others:</p>\n<ul>\n<li><code>deepset/xlm-roberta-large-squad2</code> with <br>\nmax_len=384<br>\ndoc_stride=128<br>\nbatch_size=32<br>\nlearning_rate=3e-5<br>\nepochs=2<br>\nmax_grad_norm=1.0<br>\nand appending MLQA and XQUAD data to the train set.</li>\n<li><code>muril-large-cased</code> with<br>\nmax_len=384<br>\ndoc_stride=128<br>\nbatch_size=32<br>\nlearning_rate=3e-5<br>\nepochs=2<br>\nmax_grad_norm=0.5 (this helped stabilize training for Muril)<br>\nand appending MLQA and XQUAD data to the train set.</li>\n<li><code>roberta-large-wechsel-hindi</code> with<br>\nmax_len=384<br>\ndoc_stride=128<br>\nbatch_size=32<br>\nlearning_rate=3e-5<br>\nepochs=2<br>\nand appending MLQA and XQUAD data to the train set.</li>\n</ul>\n<h2>Tamil data</h2>\n<p>We did not get <code>roberta-large-wechsel-tamil</code> to work properly because the Tamil dataset is very small. <a href=\"https://www.kaggle.com/cdminix\" target=\"_blank\">@cdminix</a> investigated adding Tamil data by scraping questions in Tamil from the web and automatically finding Wikipedia passages where the answer occurs. We ended up with ~200 extra Tamil samples this way. This improved the score of our Tamil WECHSEL model but it was still not on par with multilingual models, and we didn't really have time to investigate further.</p>\n<h2>Preprocessing</h2>\n<p>We removed references from the text (all square brackets where only numbers are between the open and closing brackets) since a lot of those occured in the text.</p>\n<h2>Stacking</h2>\n<p>We combined models by converting logits to the character level: For every token, we simply repeated the logit for each character in the token. We subtracted <code>0.01 * character position</code> for the start logits and <code>0.01 * (token length - character position)</code> for the end logits to make sure the longest character span within a token is selected as the answer.</p>\n<p>After getting logits to the character level we simply summed up the predictions of different models, without any weighting.</p>\n<h2>Submission</h2>\n<p>After disclosing our WECHSEL models and making our submission more or less competitive by stacking with other models, we got frustrated with tuning hyperparameters and not being able to improve our public score that much. So thinking that we would need to tune more parameters and that there where probably some tricks necessary to achieve high scores (like fixing answer start, more external data) we decided not to put any more work into the competition five days ago and didn't choose any final submissions.</p>\n<p>So our submission was automatically chosen to be the best submission on the public LB, which was 2x each model trained on the full dataset. Our last submission which we would have expected to be best scored 0.787 on the public LB using 5-fold CV of each model. It scored 0.779 on the private LB but was not automatically selected as one of the final submissions.</p>\n<p><img src=\"https://i.imgur.com/oSryczx.png\" alt=\"Best model score\"></p>\n<p>That would have been 2nd or 3rd place so it's a bit frustrating that we didn't select anything, but overall it's still obviously much better than we expected so I can't complain.</p>\n<p>My main takeaways are:</p>\n<ul>\n<li>It is really, really easy to overfit to the public LB when there is this little data, so it might have been a blessing in disguise that we didn't spend much time on this competition.</li>\n<li>Select a final submission, even if you don't expect to do very well :)</li>\n</ul>",
  "comments": [
    {
      "id": 1584979,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2021-11-17T00:05:53.227000",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/bminixhofer\" target=\"_blank\">@bminixhofer</a> on results and thanks for sharing solution!</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 1586354,
      "author_name": "Ayush Singh",
      "author_url": "",
      "post_date": "2021-11-18T02:26:16.720000",
      "content": "<p>congratulations <a href=\"https://www.kaggle.com/bminixhofer\" target=\"_blank\">@bminixhofer</a> </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1585083,
      "author_name": "Raghawendra Singh",
      "author_url": "",
      "post_date": "2021-11-17T03:52:28.730000",
      "content": "<p>Congrats for Gold and thanks for sharing solution.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1584987,
      "author_name": "Kiran Chaudhary",
      "author_url": "",
      "post_date": "2021-11-17T00:26:19.317000",
      "content": "<p>Congratulations!!!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1584832,
      "author_name": "Darek Kłeczek",
      "author_url": "",
      "post_date": "2021-11-16T19:47:03.143000",
      "content": "<p>Congratulations, very interesting and different approach! I didn't expect the single-language models to perform well here due to limited training data, lesson learned! </p>",
      "votes": 1,
      "replies": [
        {
          "id": 1584835,
          "author_name": "Benjamin Minixhofer",
          "author_url": "",
          "post_date": "2021-11-16T19:50:40.937000",
          "content": "<p>Thanks, and congrats on first place!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1584774,
      "author_name": "Nicholas Broad",
      "author_url": "",
      "post_date": "2021-11-16T18:53:59.150000",
      "content": "<p>I tried the Wechsel Tamil model and it did terribly so I gave up. I should have tried the Hindi one! Nice job implementing Wechsel!</p>\n<h1>👍</h1>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1584751,
      "author_name": "DrHB",
      "author_url": "",
      "post_date": "2021-11-16T18:19:43.183000",
      "content": "<p>Congratulations on gold! </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1584703,
      "author_name": "Christoph Minixhofer",
      "author_url": "",
      "post_date": "2021-11-16T17:39:26.887000",
      "content": "<p>👏      </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1590691,
      "author_name": "MINGOOK KIM",
      "author_url": "",
      "post_date": "2021-11-21T15:06:23.127000",
      "content": "<p>Congratulations on your achievements.</p>\n<p>Congratulations~</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1588974,
      "author_name": "Stephen Broadus",
      "author_url": "",
      "post_date": "2021-11-19T20:12:57.640000",
      "content": "<p>Congratulation! Good work </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 11,
  "messages": [
    {
      "id": 1584702,
      "postDate": "2021-11-16T17:38:18.757Z",
      "content": "<p>Hi, somewhat surprisingly we achieved 9th place. Our aim for this competition was to try models trained with WECHSEL (<a href=\"https://openreview.net/forum?id=JcfISE1-u4\" target=\"_blank\">https://openreview.net/forum?id=JcfISE1-u4</a>, you've probably seen the models <a href=\"https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/286603\" target=\"_blank\">here</a>) as kind of a sanity-check of the method when compared to SOTA multilingual models. To make our submission competitive we combined the WECHSEL models with some others:</p>\n<ul>\n<li><code>deepset/xlm-roberta-large-squad2</code> with <br>\nmax_len=384<br>\ndoc_stride=128<br>\nbatch_size=32<br>\nlearning_rate=3e-5<br>\nepochs=2<br>\nmax_grad_norm=1.0<br>\nand appending MLQA and XQUAD data to the train set.</li>\n<li><code>muril-large-cased</code> with<br>\nmax_len=384<br>\ndoc_stride=128<br>\nbatch_size=32<br>\nlearning_rate=3e-5<br>\nepochs=2<br>\nmax_grad_norm=0.5 (this helped stabilize training for Muril)<br>\nand appending MLQA and XQUAD data to the train set.</li>\n<li><code>roberta-large-wechsel-hindi</code> with<br>\nmax_len=384<br>\ndoc_stride=128<br>\nbatch_size=32<br>\nlearning_rate=3e-5<br>\nepochs=2<br>\nand appending MLQA and XQUAD data to the train set.</li>\n</ul>\n<h2>Tamil data</h2>\n<p>We did not get <code>roberta-large-wechsel-tamil</code> to work properly because the Tamil dataset is very small. <a href=\"https://www.kaggle.com/cdminix\" target=\"_blank\">@cdminix</a> investigated adding Tamil data by scraping questions in Tamil from the web and automatically finding Wikipedia passages where the answer occurs. We ended up with ~200 extra Tamil samples this way. This improved the score of our Tamil WECHSEL model but it was still not on par with multilingual models, and we didn't really have time to investigate further.</p>\n<h2>Preprocessing</h2>\n<p>We removed references from the text (all square brackets where only numbers are between the open and closing brackets) since a lot of those occured in the text.</p>\n<h2>Stacking</h2>\n<p>We combined models by converting logits to the character level: For every token, we simply repeated the logit for each character in the token. We subtracted <code>0.01 * character position</code> for the start logits and <code>0.01 * (token length - character position)</code> for the end logits to make sure the longest character span within a token is selected as the answer.</p>\n<p>After getting logits to the character level we simply summed up the predictions of different models, without any weighting.</p>\n<h2>Submission</h2>\n<p>After disclosing our WECHSEL models and making our submission more or less competitive by stacking with other models, we got frustrated with tuning hyperparameters and not being able to improve our public score that much. So thinking that we would need to tune more parameters and that there where probably some tricks necessary to achieve high scores (like fixing answer start, more external data) we decided not to put any more work into the competition five days ago and didn't choose any final submissions.</p>\n<p>So our submission was automatically chosen to be the best submission on the public LB, which was 2x each model trained on the full dataset. Our last submission which we would have expected to be best scored 0.787 on the public LB using 5-fold CV of each model. It scored 0.779 on the private LB but was not automatically selected as one of the final submissions.</p>\n<p><img src=\"https://i.imgur.com/oSryczx.png\" alt=\"Best model score\"></p>\n<p>That would have been 2nd or 3rd place so it's a bit frustrating that we didn't select anything, but overall it's still obviously much better than we expected so I can't complain.</p>\n<p>My main takeaways are:</p>\n<ul>\n<li>It is really, really easy to overfit to the public LB when there is this little data, so it might have been a blessing in disguise that we didn't spend much time on this competition.</li>\n<li>Select a final submission, even if you don't expect to do very well :)</li>\n</ul>",
      "rawMarkdown": "Hi, somewhat surprisingly we achieved 9th place. Our aim for this competition was to try models trained with WECHSEL (https://openreview.net/forum?id=JcfISE1-u4, you've probably seen the models [here](https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/286603)) as kind of a sanity-check of the method when compared to SOTA multilingual models. To make our submission competitive we combined the WECHSEL models with some others:\n\n- `deepset/xlm-roberta-large-squad2` with \nmax_len=384\ndoc_stride=128\nbatch_size=32\nlearning_rate=3e-5\nepochs=2\nmax_grad_norm=1.0\nand appending MLQA and XQUAD data to the train set.\n- `muril-large-cased` with\nmax_len=384\ndoc_stride=128\nbatch_size=32\nlearning_rate=3e-5\nepochs=2\nmax_grad_norm=0.5 (this helped stabilize training for Muril)\nand appending MLQA and XQUAD data to the train set.\n- `roberta-large-wechsel-hindi` with\nmax_len=384\ndoc_stride=128\nbatch_size=32\nlearning_rate=3e-5\nepochs=2\nand appending MLQA and XQUAD data to the train set.\n\n## Tamil data\n\nWe did not get `roberta-large-wechsel-tamil` to work properly because the Tamil dataset is very small. @cdminix investigated adding Tamil data by scraping questions in Tamil from the web and automatically finding Wikipedia passages where the answer occurs. We ended up with ~200 extra Tamil samples this way. This improved the score of our Tamil WECHSEL model but it was still not on par with multilingual models, and we didn't really have time to investigate further.\n\n## Preprocessing\n\nWe removed references from the text (all square brackets where only numbers are between the open and closing brackets) since a lot of those occured in the text.\n\n## Stacking\n\nWe combined models by converting logits to the character level: For every token, we simply repeated the logit for each character in the token. We subtracted `0.01 * character position` for the start logits and `0.01 * (token length - character position)` for the end logits to make sure the longest character span within a token is selected as the answer.\n\nAfter getting logits to the character level we simply summed up the predictions of different models, without any weighting.\n\n## Submission\n\nAfter disclosing our WECHSEL models and making our submission more or less competitive by stacking with other models, we got frustrated with tuning hyperparameters and not being able to improve our public score that much. So thinking that we would need to tune more parameters and that there where probably some tricks necessary to achieve high scores (like fixing answer start, more external data) we decided not to put any more work into the competition five days ago and didn't choose any final submissions.\n\nSo our submission was automatically chosen to be the best submission on the public LB, which was 2x each model trained on the full dataset. Our last submission which we would have expected to be best scored 0.787 on the public LB using 5-fold CV of each model. It scored 0.779 on the private LB but was not automatically selected as one of the final submissions.\n\n![Best model score](https://i.imgur.com/oSryczx.png)\n\nThat would have been 2nd or 3rd place so it's a bit frustrating that we didn't select anything, but overall it's still obviously much better than we expected so I can't complain.\n\nMy main takeaways are:\n- It is really, really easy to overfit to the public LB when there is this little data, so it might have been a blessing in disguise that we didn't spend much time on this competition.\n- Select a final submission, even if you don't expect to do very well :)",
      "votes": 20
    },
    {
      "id": 1584979,
      "postDate": "2021-11-17T00:05:53.227Z",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/bminixhofer\" target=\"_blank\">@bminixhofer</a> on results and thanks for sharing solution!</p>",
      "rawMarkdown": "Congrats @bminixhofer on results and thanks for sharing solution!",
      "votes": 3
    },
    {
      "id": 1586354,
      "postDate": "2021-11-18T02:26:16.720Z",
      "content": "<p>congratulations <a href=\"https://www.kaggle.com/bminixhofer\" target=\"_blank\">@bminixhofer</a> </p>",
      "rawMarkdown": "congratulations @bminixhofer ",
      "votes": 1
    },
    {
      "id": 1585083,
      "postDate": "2021-11-17T03:52:28.730Z",
      "content": "<p>Congrats for Gold and thanks for sharing solution.</p>",
      "rawMarkdown": "Congrats for Gold and thanks for sharing solution.",
      "votes": 1
    },
    {
      "id": 1584987,
      "postDate": "2021-11-17T00:26:19.317Z",
      "content": "<p>Congratulations!!!</p>",
      "rawMarkdown": "Congratulations!!!",
      "votes": 1
    },
    {
      "id": 1584832,
      "postDate": "2021-11-16T19:47:03.143Z",
      "content": "<p>Congratulations, very interesting and different approach! I didn't expect the single-language models to perform well here due to limited training data, lesson learned! </p>",
      "rawMarkdown": "Congratulations, very interesting and different approach! I didn't expect the single-language models to perform well here due to limited training data, lesson learned! ",
      "votes": 1,
      "replies": [
        {
          "id": 1584835,
          "postDate": "2021-11-16T19:50:40.937Z",
          "content": "<p>Thanks, and congrats on first place!</p>",
          "rawMarkdown": "Thanks, and congrats on first place!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1584774,
      "postDate": "2021-11-16T18:53:59.150Z",
      "content": "<p>I tried the Wechsel Tamil model and it did terribly so I gave up. I should have tried the Hindi one! Nice job implementing Wechsel!</p>\n<h1>👍</h1>",
      "rawMarkdown": "I tried the Wechsel Tamil model and it did terribly so I gave up. I should have tried the Hindi one! Nice job implementing Wechsel!\n\n# 👍",
      "votes": 1
    },
    {
      "id": 1584751,
      "postDate": "2021-11-16T18:19:43.183Z",
      "content": "<p>Congratulations on gold! </p>",
      "rawMarkdown": "Congratulations on gold! ",
      "votes": 1
    },
    {
      "id": 1584703,
      "postDate": "2021-11-16T17:39:26.887Z",
      "content": "<p>👏      </p>",
      "rawMarkdown": "      👏      ",
      "votes": 1
    },
    {
      "id": 1590691,
      "postDate": "2021-11-21T15:06:23.127Z",
      "content": "<p>Congratulations on your achievements.</p>\n<p>Congratulations~</p>",
      "rawMarkdown": "Congratulations on your achievements.\n\nCongratulations~"
    },
    {
      "id": 1588974,
      "postDate": "2021-11-19T20:12:57.640Z",
      "content": "<p>Congratulation! Good work </p>",
      "rawMarkdown": "Congratulation! Good work "
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T16:20:28Z",
  "competition_id": 30060,
  "competition_slug": "chaii-hindi-and-tamil-question-answering",
  "team_id": 7445833,
  "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/minixhofer-9th-place-solution"
}