{
  "id": 288168,
  "title": "14th place submission",
  "content": "<p>I'm a little frustrated as I had a 0.768 solution which I didn't submit, which is the difference between gold and silver. But still I'm happy to get my second competition medal!</p>\n<h2>Fine-tuning on SQuAD</h2>\n<p>I think some of the other solutions have already mentioned this, but MuRIL and RemBERT both significantly benefitted from being fine-tuned on SQuAD before being fine-tuned on the competition data. It seemed like a pretty obvious thing to try since the XLMR checkpoint everyone was basing their models on had also been fine-tuned on the same data. I used this approach for all of my MuRIL and RemBERT models.</p>\n<h2>Torchscript</h2>\n<p>I exported all of my models to torchscript which gave me a slight inference speed up. I was able to ensemble 20 folds, but it wasn't quite enough to fit 25 folds in the time limit. I also tried exporting to ONNX, but this didn't give me any speed up at all. It was my first time using either torchscript or ONNX so I'm sure there are a bunch of additional optimisations which could've been made.</p>\n<h2>0.768 which I didn't submit</h2>\n<p>The one I didn't submit was 10 fold MuRIL-large-cased and 10 fold RemBERT. These models actually had my best CV as well, but the public LB was pretty mediocre so I stupidly ignored them. When I calculated their OOF scores I noticed that although they had very similar CV scores (0.722 for MuRIL and 0.719 for RemBERT) they performed very differently on each language:</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>MuRIL</th>\n<th>RemBERT</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>OOF All</td>\n<td>0.722</td>\n<td>0.719</td>\n</tr>\n<tr>\n<td>OOF Hindi</td>\n<td>0.782</td>\n<td>0.753</td>\n</tr>\n<tr>\n<td>OOF Tamil</td>\n<td>0.601</td>\n<td>0.652</td>\n</tr>\n</tbody>\n</table>\n<p>MuRIL was better at Hindi questions, and RemBERT was much better at Tamil, so for inference I just divided the predictions by language, giving the Hindi questions to MuRIL and the Tamil questions to RemBERT. The final predictions were the mean of 10 folds of each model.</p>\n<p>I also had another submission with these 2 models where I generated the predictions of each fold separately and then had the folds vote on the start and end position of the answer. That one also wasn't submitted, but would've been 0.765.</p>\n<h2>The actual 14th place submission</h2>\n<p>I switched between 5-folds and 10-fold strategies a couple of times during this competition. I ended up using 5-folds as I was concerned that I wouldn't be able to get enough diversity into my ensemble using 10-fold models. 10 folds was also really slowing down the amount of time it took to complete a training run. </p>\n<p>My final 0.758 solution was 5-fold MuRIL x3 and 5-fold XLMR. The different MuRIL checkpoints had all been trained on slightly different data:<br>\nMuRIL 1: fine-tune on SQuAD -&gt; fine-tune on competition data + MLQA Hindi + XQUAD Hindi<br>\nMuRIL 2: fine-tune on SQuAD -&gt; fine-tune on competition data + MLQA (Hindi and English) + XQUAD Hindi<br>\nMuRIL 3: fine-tune on SQuAD -&gt; fine-tune on competition data + Google translated Tamil SQuAD sample + MLQA Hindi + XQUAD Hindi</p>\n<p>The final ensemble contained these models with their folds averaged in a few different ways. The averaged predictions were then combined through voting on the start and end positions.</p>\n<p>Thanks for the enjoyable competition!</p>",
  "comments": [
    {
      "id": 1591249,
      "author_name": "Parv Yadav",
      "author_url": "",
      "post_date": "2021-11-22T05:51:25.193000",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/amontgomerie\" target=\"_blank\">@amontgomerie</a> !😊✨</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1590706,
      "author_name": "MINGOOK KIM",
      "author_url": "",
      "post_date": "2021-11-21T15:13:09.107000",
      "content": "<p>Thanks for the good summary.</p>\n<p>It helped a lot.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1586203,
      "author_name": "Darek Kłeczek",
      "author_url": "",
      "post_date": "2021-11-17T22:19:04.403000",
      "content": "<p>Congrats Adam!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1593628,
      "author_name": "JiyaByju",
      "author_url": "",
      "post_date": "2021-11-24T05:34:14.403000",
      "content": "<p>congrats for your effort..keep going</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 4,
  "messages": [
    {
      "id": 1584999,
      "postDate": "2021-11-17T01:02:00.937Z",
      "content": "<p>I'm a little frustrated as I had a 0.768 solution which I didn't submit, which is the difference between gold and silver. But still I'm happy to get my second competition medal!</p>\n<h2>Fine-tuning on SQuAD</h2>\n<p>I think some of the other solutions have already mentioned this, but MuRIL and RemBERT both significantly benefitted from being fine-tuned on SQuAD before being fine-tuned on the competition data. It seemed like a pretty obvious thing to try since the XLMR checkpoint everyone was basing their models on had also been fine-tuned on the same data. I used this approach for all of my MuRIL and RemBERT models.</p>\n<h2>Torchscript</h2>\n<p>I exported all of my models to torchscript which gave me a slight inference speed up. I was able to ensemble 20 folds, but it wasn't quite enough to fit 25 folds in the time limit. I also tried exporting to ONNX, but this didn't give me any speed up at all. It was my first time using either torchscript or ONNX so I'm sure there are a bunch of additional optimisations which could've been made.</p>\n<h2>0.768 which I didn't submit</h2>\n<p>The one I didn't submit was 10 fold MuRIL-large-cased and 10 fold RemBERT. These models actually had my best CV as well, but the public LB was pretty mediocre so I stupidly ignored them. When I calculated their OOF scores I noticed that although they had very similar CV scores (0.722 for MuRIL and 0.719 for RemBERT) they performed very differently on each language:</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>MuRIL</th>\n<th>RemBERT</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>OOF All</td>\n<td>0.722</td>\n<td>0.719</td>\n</tr>\n<tr>\n<td>OOF Hindi</td>\n<td>0.782</td>\n<td>0.753</td>\n</tr>\n<tr>\n<td>OOF Tamil</td>\n<td>0.601</td>\n<td>0.652</td>\n</tr>\n</tbody>\n</table>\n<p>MuRIL was better at Hindi questions, and RemBERT was much better at Tamil, so for inference I just divided the predictions by language, giving the Hindi questions to MuRIL and the Tamil questions to RemBERT. The final predictions were the mean of 10 folds of each model.</p>\n<p>I also had another submission with these 2 models where I generated the predictions of each fold separately and then had the folds vote on the start and end position of the answer. That one also wasn't submitted, but would've been 0.765.</p>\n<h2>The actual 14th place submission</h2>\n<p>I switched between 5-folds and 10-fold strategies a couple of times during this competition. I ended up using 5-folds as I was concerned that I wouldn't be able to get enough diversity into my ensemble using 10-fold models. 10 folds was also really slowing down the amount of time it took to complete a training run. </p>\n<p>My final 0.758 solution was 5-fold MuRIL x3 and 5-fold XLMR. The different MuRIL checkpoints had all been trained on slightly different data:<br>\nMuRIL 1: fine-tune on SQuAD -&gt; fine-tune on competition data + MLQA Hindi + XQUAD Hindi<br>\nMuRIL 2: fine-tune on SQuAD -&gt; fine-tune on competition data + MLQA (Hindi and English) + XQUAD Hindi<br>\nMuRIL 3: fine-tune on SQuAD -&gt; fine-tune on competition data + Google translated Tamil SQuAD sample + MLQA Hindi + XQUAD Hindi</p>\n<p>The final ensemble contained these models with their folds averaged in a few different ways. The averaged predictions were then combined through voting on the start and end positions.</p>\n<p>Thanks for the enjoyable competition!</p>",
      "rawMarkdown": "I'm a little frustrated as I had a 0.768 solution which I didn't submit, which is the difference between gold and silver. But still I'm happy to get my second competition medal!\n\n## Fine-tuning on SQuAD\n\nI think some of the other solutions have already mentioned this, but MuRIL and RemBERT both significantly benefitted from being fine-tuned on SQuAD before being fine-tuned on the competition data. It seemed like a pretty obvious thing to try since the XLMR checkpoint everyone was basing their models on had also been fine-tuned on the same data. I used this approach for all of my MuRIL and RemBERT models.\n\n## Torchscript\n\nI exported all of my models to torchscript which gave me a slight inference speed up. I was able to ensemble 20 folds, but it wasn't quite enough to fit 25 folds in the time limit. I also tried exporting to ONNX, but this didn't give me any speed up at all. It was my first time using either torchscript or ONNX so I'm sure there are a bunch of additional optimisations which could've been made.\n\n## 0.768 which I didn't submit\n\nThe one I didn't submit was 10 fold MuRIL-large-cased and 10 fold RemBERT. These models actually had my best CV as well, but the public LB was pretty mediocre so I stupidly ignored them. When I calculated their OOF scores I noticed that although they had very similar CV scores (0.722 for MuRIL and 0.719 for RemBERT) they performed very differently on each language:\n\n|  |  MuRIL | RemBERT  |\n| --- | --- |\n| OOF All | 0.722 | 0.719 |\n| OOF Hindi | 0.782 | 0.753 |\n| OOF Tamil | 0.601 | 0.652 |\n\nMuRIL was better at Hindi questions, and RemBERT was much better at Tamil, so for inference I just divided the predictions by language, giving the Hindi questions to MuRIL and the Tamil questions to RemBERT. The final predictions were the mean of 10 folds of each model.\n\nI also had another submission with these 2 models where I generated the predictions of each fold separately and then had the folds vote on the start and end position of the answer. That one also wasn't submitted, but would've been 0.765.\n\n## The actual 14th place submission\n\nI switched between 5-folds and 10-fold strategies a couple of times during this competition. I ended up using 5-folds as I was concerned that I wouldn't be able to get enough diversity into my ensemble using 10-fold models. 10 folds was also really slowing down the amount of time it took to complete a training run. \n\nMy final 0.758 solution was 5-fold MuRIL x3 and 5-fold XLMR. The different MuRIL checkpoints had all been trained on slightly different data:\nMuRIL 1: fine-tune on SQuAD -> fine-tune on competition data + MLQA Hindi + XQUAD Hindi\nMuRIL 2: fine-tune on SQuAD -> fine-tune on competition data + MLQA (Hindi and English) + XQUAD Hindi\nMuRIL 3: fine-tune on SQuAD -> fine-tune on competition data + Google translated Tamil SQuAD sample + MLQA Hindi + XQUAD Hindi\n\nThe final ensemble contained these models with their folds averaged in a few different ways. The averaged predictions were then combined through voting on the start and end positions.\n\n\nThanks for the enjoyable competition!",
      "votes": 19
    },
    {
      "id": 1591249,
      "postDate": "2021-11-22T05:51:25.193Z",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/amontgomerie\" target=\"_blank\">@amontgomerie</a> !😊✨</p>",
      "rawMarkdown": "Congrats @amontgomerie !😊✨",
      "votes": 1
    },
    {
      "id": 1590706,
      "postDate": "2021-11-21T15:13:09.107Z",
      "content": "<p>Thanks for the good summary.</p>\n<p>It helped a lot.</p>",
      "rawMarkdown": "Thanks for the good summary.\n\nIt helped a lot.",
      "votes": 1
    },
    {
      "id": 1586203,
      "postDate": "2021-11-17T22:19:04.403Z",
      "content": "<p>Congrats Adam!</p>",
      "rawMarkdown": "Congrats Adam!",
      "votes": 1
    },
    {
      "id": 1593628,
      "postDate": "2021-11-24T05:34:14.403Z",
      "content": "<p>congrats for your effort..keep going</p>",
      "rawMarkdown": "congrats for your effort..keep going\n"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T16:21:13Z",
  "competition_id": 30060,
  "competition_slug": "chaii-hindi-and-tamil-question-answering",
  "team_id": 7252537,
  "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/adam-montgomerie-14th-place-submission"
}