{
  "id": 289511,
  "title": "8th place solution",
  "content": "<p>Thanks to Kaggle and Google for hosting this exciting competition. It gave me opporutnies for getting more familiar with multilingual NLP and exploring various techniques for accurate reading comprehension. </p>\n<h2>CV strategy</h2>\n<p>I used local 5-fold CV for making my decisions as we usually do in other competitions since I assumed it makes my pipeline simpler. <br>\nActually, I think this choice was not good because the public LB dataset was big enough and \"trust CV\" style didn't allow me to get in the medal-zone during the entire period of this competition (it made me really nervous!!!). </p>\n<h2>Modeling</h2>\n<p>I trained <code>xlm-roberta-large-squad2</code> and <code>muril-large-cased</code> with Adam as most other participants do. The main hyperparameters for my <code>xlm-roberta-large-squad2</code> is the followings: </p>\n<ul>\n<li>dataset<ul>\n<li>max_seq_length: 384</li>\n<li>doc_stride: 128</li></ul></li>\n<li>model <ul>\n<li>dropout: 0.0</li>\n<li>num_layers_to_reinit: 2<ul>\n<li>I became to like weight re-initialization after reading <a href=\"https://arxiv.org/abs/2006.05987\" target=\"_blank\">Revisiting Few-sample BERT Fine-tuning</a> and <a href=\"https://arxiv.org/pdf/2006.04884.pdf\" target=\"_blank\">On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines\n</a>. I couldn't see noticeable score improvement in this competition though :(. </li></ul></li></ul></li>\n<li>training<ul>\n<li>batch_size: 4</li>\n<li>gradient_accumulation_steps: 4</li>\n<li>learning_rate: 0.000015</li>\n<li>weight_decay: 0.01</li>\n<li>warmup_step_ratio: 0.1</li>\n<li>num_epochs: 2</li></ul></li>\n</ul>\n<h3>Minor changes in a training procedure</h3>\n<h4>Preprocessing</h4>\n<p>I often see tokens including both a parenthesis and digits (e.g., \"99)\") after tokenizing with the <code>xlm-roberta-large-squad2</code>'s pretrained tokenizer. I tried to improve model performance by separating a parenthesis (e.g, \"(\") and digits (e.g, \"19\") at the preprocessing stage. In adition, I replaced Hindi and Tamil digits like ०, १, ௦, and ௧ with corresponding ASCII digits to make training task easier. <br>\nScore improvement from these preprocessing was ignorable after post-processing. However, I decided to keep using them to make my solution more different from others' and maximize a chance of huge shake-up. </p>\n<h4>OOF-based relabeling</h4>\n<p>Inspired by <a href=\"https://arxiv.org/pdf/1909.04849.pdf\" target=\"_blank\">Hard EM</a>, I tried to choose better <code>answer_start</code> for <code>answer_text</code> with multiple occurrence. I replaced the original <code>answer_start</code> corresponding to <code>answer_text</code>'s first occurrence with other ones if there exists other spans whose scores are larger than or equal to 0.8 * (the maximum score among all span candidates including texts other than <code>answer_text</code>). I also modified <code>answer_start</code> when <code>answer_text</code> consists of digits and it's surrounded by other digits. This relabeling increased CV and privte LB  by 0.005 if my memory is correct.<br>\nAveraging 5-fold <code>xlm-roberta-large-squad2</code> models trained from the above procedure gave me 0.773 and 0.738 in public and private LB respectively when submitting with <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">nbroad</a>'s post-processing (<a href=\"https://www.kaggle.com/nbroad/chaii-qa-torch-5-fold-with-post-processing-765\" target=\"_blank\">link-to-kernel</a>): </p>\n<h2>Post-processing</h2>\n<p>As other top-teams mentions in their solution, replacing Hindi and Tamil digits with ASCII digits when there are no other types of characters worked pretty well. Combining the original post-processing, this Hindi/Tamil digit handling, and other minor modification increased my LB scores by 0.01 (public LB = 0.779, private LB = 0.750). <br>\nI suspect its successful result came from somewhat broken dataset preparation process because it's weird that all of 133 digit-only answer texts consist of only ASCII digits whereas we often see Hindi and Tamil digits in other answer texts. I guess conversion from Hindi/Tamil digits to ASCII digits happenned when importing the dataset CSV into Google Spreadsheet at some point (surprisingly, Google Spreadsheet automatically does this conversion :o). </p>\n<h2>Model ensembling</h2>\n<p>I aggregated prediction results from base models in a text-wise manner rather than a token-wise or character-wise manner because I didn’t have enough time to explore cool ensemble methods. Implementing text-wise aggregation is easy. </p>\n<ul>\n<li>Collect top-k (k = 10) candidates from each base model. Each candidate should have some score (i.g., start-logit + end-logit). </li>\n<li>Given weights of base models, compute weighted sum of scores for each text in every candidate from base models</li>\n<li>Select one with the largest weighted sum as a final answer</li>\n</ul>\n<h3>Final model</h3>\n<p>I achieved my best private LB score (0.770) with the following weights obtained from hyper-parameter search with Optuna: </p>\n<ul>\n<li>xlm-roberta-large-squad2 (w/ preprocessing):  0.5709</li>\n<li>muril-large-cased (w/ preprocessing): 0.2829</li>\n<li>xlm-roberta-large-squad2 (wo/ preprocessing): 0.1461</li>\n</ul>\n<h2>Other things</h2>\n<h3>Computation resources</h3>\n<p>This is the first competition I participated in with Google Colab Pro+. A100 and V100 from Colab boosted the iteration speed of my experiments a lot :) </p>\n<ul>\n<li>GCP T4 (mainly for debugging)</li>\n<li>Google Colab Pro+</li>\n</ul>\n<h3>Things that didn't work</h3>\n<ul>\n<li>Training with Tamil XQA</li>\n<li>Upsampling of positive examples</li>\n<li>Answer candidate reranking like <a href=\"https://aclanthology.org/P19-1611.pdf\" target=\"_blank\">RankQA</a></li>\n<li>Minor tweaking of architecture and training method<ul>\n<li>Dropout</li>\n<li>Mixout</li>\n<li>Custom head like avg-pooling &amp; max-pooling</li>\n<li>Label smoothing </li>\n<li>Language-wise fine-tuning</li></ul></li>\n</ul>",
  "comments": [
    {
      "id": 1590012,
      "author_name": "Darek Kłeczek",
      "author_url": "",
      "post_date": "2021-11-20T20:04:41.157000",
      "content": "<p>Congrats! Interesting hypothesis on the Hindi/Tamil digits conversion, I missed that opportunity in my postprocessing… </p>",
      "votes": 1,
      "replies": [
        {
          "id": 1590420,
          "author_name": "flowlight",
          "author_url": "",
          "post_date": "2021-11-21T08:51:05.700000",
          "content": "<p>Thanks for your comment! Congratulations on winning this competition:)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1590700,
      "author_name": "MINGOOK KIM",
      "author_url": "",
      "post_date": "2021-11-21T15:09:39.757000",
      "content": "<p>Congratulations on your achievements.</p>\n<p>Congratulations~</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 3,
  "messages": [
    {
      "id": 1589808,
      "postDate": "2021-11-20T15:36:44.337Z",
      "content": "<p>Thanks to Kaggle and Google for hosting this exciting competition. It gave me opporutnies for getting more familiar with multilingual NLP and exploring various techniques for accurate reading comprehension. </p>\n<h2>CV strategy</h2>\n<p>I used local 5-fold CV for making my decisions as we usually do in other competitions since I assumed it makes my pipeline simpler. <br>\nActually, I think this choice was not good because the public LB dataset was big enough and \"trust CV\" style didn't allow me to get in the medal-zone during the entire period of this competition (it made me really nervous!!!). </p>\n<h2>Modeling</h2>\n<p>I trained <code>xlm-roberta-large-squad2</code> and <code>muril-large-cased</code> with Adam as most other participants do. The main hyperparameters for my <code>xlm-roberta-large-squad2</code> is the followings: </p>\n<ul>\n<li>dataset<ul>\n<li>max_seq_length: 384</li>\n<li>doc_stride: 128</li></ul></li>\n<li>model <ul>\n<li>dropout: 0.0</li>\n<li>num_layers_to_reinit: 2<ul>\n<li>I became to like weight re-initialization after reading <a href=\"https://arxiv.org/abs/2006.05987\" target=\"_blank\">Revisiting Few-sample BERT Fine-tuning</a> and <a href=\"https://arxiv.org/pdf/2006.04884.pdf\" target=\"_blank\">On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines\n</a>. I couldn't see noticeable score improvement in this competition though :(. </li></ul></li></ul></li>\n<li>training<ul>\n<li>batch_size: 4</li>\n<li>gradient_accumulation_steps: 4</li>\n<li>learning_rate: 0.000015</li>\n<li>weight_decay: 0.01</li>\n<li>warmup_step_ratio: 0.1</li>\n<li>num_epochs: 2</li></ul></li>\n</ul>\n<h3>Minor changes in a training procedure</h3>\n<h4>Preprocessing</h4>\n<p>I often see tokens including both a parenthesis and digits (e.g., \"99)\") after tokenizing with the <code>xlm-roberta-large-squad2</code>'s pretrained tokenizer. I tried to improve model performance by separating a parenthesis (e.g, \"(\") and digits (e.g, \"19\") at the preprocessing stage. In adition, I replaced Hindi and Tamil digits like ०, १, ௦, and ௧ with corresponding ASCII digits to make training task easier. <br>\nScore improvement from these preprocessing was ignorable after post-processing. However, I decided to keep using them to make my solution more different from others' and maximize a chance of huge shake-up. </p>\n<h4>OOF-based relabeling</h4>\n<p>Inspired by <a href=\"https://arxiv.org/pdf/1909.04849.pdf\" target=\"_blank\">Hard EM</a>, I tried to choose better <code>answer_start</code> for <code>answer_text</code> with multiple occurrence. I replaced the original <code>answer_start</code> corresponding to <code>answer_text</code>'s first occurrence with other ones if there exists other spans whose scores are larger than or equal to 0.8 * (the maximum score among all span candidates including texts other than <code>answer_text</code>). I also modified <code>answer_start</code> when <code>answer_text</code> consists of digits and it's surrounded by other digits. This relabeling increased CV and privte LB  by 0.005 if my memory is correct.<br>\nAveraging 5-fold <code>xlm-roberta-large-squad2</code> models trained from the above procedure gave me 0.773 and 0.738 in public and private LB respectively when submitting with <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">nbroad</a>'s post-processing (<a href=\"https://www.kaggle.com/nbroad/chaii-qa-torch-5-fold-with-post-processing-765\" target=\"_blank\">link-to-kernel</a>): </p>\n<h2>Post-processing</h2>\n<p>As other top-teams mentions in their solution, replacing Hindi and Tamil digits with ASCII digits when there are no other types of characters worked pretty well. Combining the original post-processing, this Hindi/Tamil digit handling, and other minor modification increased my LB scores by 0.01 (public LB = 0.779, private LB = 0.750). <br>\nI suspect its successful result came from somewhat broken dataset preparation process because it's weird that all of 133 digit-only answer texts consist of only ASCII digits whereas we often see Hindi and Tamil digits in other answer texts. I guess conversion from Hindi/Tamil digits to ASCII digits happenned when importing the dataset CSV into Google Spreadsheet at some point (surprisingly, Google Spreadsheet automatically does this conversion :o). </p>\n<h2>Model ensembling</h2>\n<p>I aggregated prediction results from base models in a text-wise manner rather than a token-wise or character-wise manner because I didn’t have enough time to explore cool ensemble methods. Implementing text-wise aggregation is easy. </p>\n<ul>\n<li>Collect top-k (k = 10) candidates from each base model. Each candidate should have some score (i.g., start-logit + end-logit). </li>\n<li>Given weights of base models, compute weighted sum of scores for each text in every candidate from base models</li>\n<li>Select one with the largest weighted sum as a final answer</li>\n</ul>\n<h3>Final model</h3>\n<p>I achieved my best private LB score (0.770) with the following weights obtained from hyper-parameter search with Optuna: </p>\n<ul>\n<li>xlm-roberta-large-squad2 (w/ preprocessing):  0.5709</li>\n<li>muril-large-cased (w/ preprocessing): 0.2829</li>\n<li>xlm-roberta-large-squad2 (wo/ preprocessing): 0.1461</li>\n</ul>\n<h2>Other things</h2>\n<h3>Computation resources</h3>\n<p>This is the first competition I participated in with Google Colab Pro+. A100 and V100 from Colab boosted the iteration speed of my experiments a lot :) </p>\n<ul>\n<li>GCP T4 (mainly for debugging)</li>\n<li>Google Colab Pro+</li>\n</ul>\n<h3>Things that didn't work</h3>\n<ul>\n<li>Training with Tamil XQA</li>\n<li>Upsampling of positive examples</li>\n<li>Answer candidate reranking like <a href=\"https://aclanthology.org/P19-1611.pdf\" target=\"_blank\">RankQA</a></li>\n<li>Minor tweaking of architecture and training method<ul>\n<li>Dropout</li>\n<li>Mixout</li>\n<li>Custom head like avg-pooling &amp; max-pooling</li>\n<li>Label smoothing </li>\n<li>Language-wise fine-tuning</li></ul></li>\n</ul>",
      "rawMarkdown": "\nThanks to Kaggle and Google for hosting this exciting competition. It gave me opporutnies for getting more familiar with multilingual NLP and exploring various techniques for accurate reading comprehension. \n\n## CV strategy\nI used local 5-fold CV for making my decisions as we usually do in other competitions since I assumed it makes my pipeline simpler. \n\n Actually, I think this choice was not good because the public LB dataset was big enough and \"trust CV\" style didn't allow me to get in the medal-zone during the entire period of this competition (it made me really nervous!!!). \n\n## Modeling\nI trained `xlm-roberta-large-squad2` and `muril-large-cased` with Adam as most other participants do. The main hyperparameters for my `xlm-roberta-large-squad2` is the followings: \n* dataset\n  * max_seq_length: 384\n  * doc_stride: 128\n* model \n    * dropout: 0.0\n    * num_layers_to_reinit: 2\n      * I became to like weight re-initialization after reading [Revisiting Few-sample BERT Fine-tuning](https://arxiv.org/abs/2006.05987) and [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines\n](https://arxiv.org/pdf/2006.04884.pdf). I couldn't see noticeable score improvement in this competition though :(. \n* training\n  * batch_size: 4\n  * gradient_accumulation_steps: 4\n  * learning_rate: 0.000015\n  * weight_decay: 0.01\n  * warmup_step_ratio: 0.1\n  * num_epochs: 2\n\n\n### Minor changes in a training procedure\n#### Preprocessing\nI often see tokens including both a parenthesis and digits (e.g., \"99)\") after tokenizing with the `xlm-roberta-large-squad2`'s pretrained tokenizer. I tried to improve model performance by separating a parenthesis (e.g, \"(\") and digits (e.g, \"19\") at the preprocessing stage. In adition, I replaced Hindi and Tamil digits like ०, १, ௦, and ௧ with corresponding ASCII digits to make training task easier. \n\nScore improvement from these preprocessing was ignorable after post-processing. However, I decided to keep using them to make my solution more different from others' and maximize a chance of huge shake-up. \n#### OOF-based relabeling\nInspired by [Hard EM](https://arxiv.org/pdf/1909.04849.pdf), I tried to choose better `answer_start` for `answer_text` with multiple occurrence. I replaced the original `answer_start` corresponding to `answer_text`'s first occurrence with other ones if there exists other spans whose scores are larger than or equal to 0.8 * (the maximum score among all span candidates including texts other than `answer_text`). I also modified `answer_start` when `answer_text` consists of digits and it's surrounded by other digits. This relabeling increased CV and privte LB  by 0.005 if my memory is correct.\n\nAveraging 5-fold `xlm-roberta-large-squad2` models trained from the above procedure gave me 0.773 and 0.738 in public and private LB respectively when submitting with [nbroad](https://www.kaggle.com/nbroad)'s post-processing ([link-to-kernel](https://www.kaggle.com/nbroad/chaii-qa-torch-5-fold-with-post-processing-765)): \n\n\n## Post-processing\nAs other top-teams mentions in their solution, replacing Hindi and Tamil digits with ASCII digits when there are no other types of characters worked pretty well. Combining the original post-processing, this Hindi/Tamil digit handling, and other minor modification increased my LB scores by 0.01 (public LB = 0.779, private LB = 0.750). \n\n\nI suspect its successful result came from somewhat broken dataset preparation process because it's weird that all of 133 digit-only answer texts consist of only ASCII digits whereas we often see Hindi and Tamil digits in other answer texts. I guess conversion from Hindi/Tamil digits to ASCII digits happenned when importing the dataset CSV into Google Spreadsheet at some point (surprisingly, Google Spreadsheet automatically does this conversion :o). \n\n## Model ensembling\nI aggregated prediction results from base models in a text-wise manner rather than a token-wise or character-wise manner because I didn’t have enough time to explore cool ensemble methods. Implementing text-wise aggregation is easy. \n\n- Collect top-k (k = 10) candidates from each base model. Each candidate should have some score (i.g., start-logit + end-logit). \n- Given weights of base models, compute weighted sum of scores for each text in every candidate from base models\n- Select one with the largest weighted sum as a final answer\n\n### Final model\nI achieved my best private LB score (0.770) with the following weights obtained from hyper-parameter search with Optuna: \n-  xlm-roberta-large-squad2 (w/ preprocessing):  0.5709\n- muril-large-cased (w/ preprocessing): 0.2829\n- xlm-roberta-large-squad2 (wo/ preprocessing): 0.1461\n\n## Other things\n### Computation resources\nThis is the first competition I participated in with Google Colab Pro+. A100 and V100 from Colab boosted the iteration speed of my experiments a lot :) \n* GCP T4 (mainly for debugging)\n* Google Colab Pro+\n\n\n### Things that didn't work\n* Training with Tamil XQA\n* Upsampling of positive examples\n* Answer candidate reranking like [RankQA](https://aclanthology.org/P19-1611.pdf)\n* Minor tweaking of architecture and training method\n  * Dropout\n  * Mixout\n  * Custom head like avg-pooling & max-pooling\n  * Label smoothing \n  * Language-wise fine-tuning",
      "votes": 7
    },
    {
      "id": 1590012,
      "postDate": "2021-11-20T20:04:41.157Z",
      "content": "<p>Congrats! Interesting hypothesis on the Hindi/Tamil digits conversion, I missed that opportunity in my postprocessing… </p>",
      "rawMarkdown": "Congrats! Interesting hypothesis on the Hindi/Tamil digits conversion, I missed that opportunity in my postprocessing... ",
      "votes": 1,
      "replies": [
        {
          "id": 1590420,
          "postDate": "2021-11-21T08:51:05.700Z",
          "content": "<p>Thanks for your comment! Congratulations on winning this competition:)</p>",
          "rawMarkdown": "Thanks for your comment! Congratulations on winning this competition:)"
        }
      ]
    },
    {
      "id": 1590700,
      "postDate": "2021-11-21T15:09:39.757Z",
      "content": "<p>Congratulations on your achievements.</p>\n<p>Congratulations~</p>",
      "rawMarkdown": "Congratulations on your achievements.\n\nCongratulations~"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T16:20:14Z",
  "competition_id": 30060,
  "competition_slug": "chaii-hindi-and-tamil-question-answering",
  "team_id": 7251441,
  "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/flowlight-8th-place-solution"
}