Thanks to Kaggle and Google India for hosting this competition and Congratulations to all the winners. That there are many medal winners who don't know either or both languages is very good news for low-resource languages in general.
\nLike many other competitors observed, adding TydiQA Telugu and Bengali datasets to the mix certainly boosted the model performances both in CV and LB. This shouldn't be surprising at all since most of us are using multilingual models and adding any Indic QA dataset will have a lot more real-world context overlap.
\nOne other thing I usually do for all single-answer datasets is to fine-tune on Squad2 or other multiple-answer datasets first, instead of directly fine-tuning on an MLM checkpoint. This gives the models a better intuition on not ignoring missed labels. XLMR already comes with a model we all fine-tuned further on. I did these for Muril and Rembert, both of which gave a consistent 0.8 - 0.81 across the LB. Private LB scores were also in the higher range for Muril and Rembert fine-tuned on top of Squad2. (0.76 - 0.77)
\nChaii dataset is a bit noisy on its own. I didn't fix any incorrect labels. But I fixed the answer-start values by picking the first matching answer candidate from @thedrcat 's fine-tuned version (that he published on the first week of this competition).
\nI trained XLMR, Muril-Large, and Rembert over their Squad2 checkpoints, with Chaii dataset, MLQA Hindi, XQuaD Hindi, TydiQA Telugu and Bengali datasets. Didn't do anything wrt the model perse. Just playing around with hyperparameters, etc. But because this competition provides an entire passage as a context, this is as much as a passage selection task as it is a question-answering task. So I maintained a 1:2 ratio of relevant vs irrelevant contexts, by sampling on Chaii's train dataset and using the remaining external data as-is (they were smaller passages) for all models' training.
\nRembert is a real humdinger of a model. It had the most NLU capabilities among the three model-types (at least the way I trained them). Case in point:
\nThere was a question about the largest mountain in Chaii Dataset and the human annotator marked Mount Everest as the answer. But Rembert (Squad2 checkpoint. Not the Chaii checkpoint) predicted Mauna Loa because the context read The tallest mountain doesn't necessarily mean the largest. For instance, Mauna Loa has a base area of ....
I picked the most interesting example. But it consistently out-performed both Muril-Large and XLMR when it came to picking the right context. It performed well on extracting outputs from Wikipedia's trivia boxes too. It didn't mark the exact word boundaries for predicted answers. But it almost always had the right answer.
\nThis is where I messed up with selecting the submissions. My best internal submission came from a 3-fold Rembert (0.778). But since I assumed that Rembert wasn't right in generating answer boundaries, I fed its passage outputs to Muril-Large to generate word-boundaries. This reduced the scores to 0.757 :P It was a bit of a gamble between relevant context selection vs answer selection, and I picked an approach that weaves both. This didn't work out so well in retrospect :)
\nWill cleanup the notebooks and publish them later today. But for now, here are the Squad2 checkpoints that I fine-tuned on Rembert and Muril-Large.
\n\nThe low-resource nature, combined with the morphological richness of Hindi and Tamil in NLP scene restricted the usage of better pre-training approaches like MPNet and Electra, or even SOTA QA models like LUKE. Hindi is a fusional language. So a word can have multiple morphemes depending on gender, tense, etc (Like go vs went in English). The current fix (that seems to be working well) to compensate for this is to train on a large amount of data.
Agglutinative languages like Tamil are a different kind of reality altogether. One thumb rule I learnt during this competition to evaluate models for agglutinative languages is to look for models' metrics on Turkish datasets (Turkish is an agglutinative beast). Where it isn't available, metrics on Japanese datasets will work just as fine. Character-based transformers like Canine and Charformers sound like promising approaches for this type of language. They aren't mature/stable yet. But this is NLP. They might very well be the norm this time next year.
\nThis competition served as an exponential learning curve for me. So much so that, during the start of the competition, I look for metrics in published NLP papers. By the end of the competition, it was more about ablation studies. It's also the first time I was a bit active in forums, GitHub, etc. So the confidence boost helped a lot too. I might be a bit more active on Kaggle going forward, and invest in a decent GPU source too :)
\nEdit: Final submission inference notebook. Even though this approach didn't work in this particular competition, the usage of a QA model for passage selection might still benefit someone in a better-labelled dataset. All inputs are the three types of the models. Of the two utility scripts, hf-qa-utils is mostly boilerplate code from the HuggingFace example notebooks to tokenize and map back datasets. passage-qa-utils does the sub-context selection from a Rembert QA model.
A second approach of using 3 individual model outputs and passing them through an LGBM Ranker also faired quite well in private LB, that I forgot to mention in the original post.
", "comments": [ { "id": 1584848, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-11-16T20:03:32.810000", "content": "Congrats @jdoesv you did great on this competition, and sorry for the bad luck with the shake down… In the long run, it's the learning and knowledge progress that matters, and that was a definite win! Looking forward to see you in the next Kaggle comp!
", "votes": 1, "replies": [ { "id": 1585155, "author_name": "Sindhu V", "author_url": "", "post_date": "2021-11-17T05:59:51.963000", "content": "Thank you for your kind words @thedrcat !😄
", "votes": 0, "replies": [] } ] }, { "id": 1586742, "author_name": "Vee", "author_url": "", "post_date": "2021-11-18T08:36:49.617000", "content": "Congrats on a great finish! Thanks for introducing Muril on this competition for the first time and nice to hear that Rembert does well too!
", "votes": 0, "replies": [] }, { "id": 1584255, "author_name": "Sindhu V", "author_url": "", "post_date": "2021-11-16T11:46:54.320000", "content": "Also for people who know their Tamil pop culture: https://tinyurl.com/otherbanana Couldn't help, now that the competition is over! :)
", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 1583837, "postDate": "2021-11-16T05:16:59.193Z", "content": "Thanks to Kaggle and Google India for hosting this competition and Congratulations to all the winners. That there are many medal winners who don't know either or both languages is very good news for low-resource languages in general.
\nLike many other competitors observed, adding TydiQA Telugu and Bengali datasets to the mix certainly boosted the model performances both in CV and LB. This shouldn't be surprising at all since most of us are using multilingual models and adding any Indic QA dataset will have a lot more real-world context overlap.
\nOne other thing I usually do for all single-answer datasets is to fine-tune on Squad2 or other multiple-answer datasets first, instead of directly fine-tuning on an MLM checkpoint. This gives the models a better intuition on not ignoring missed labels. XLMR already comes with a model we all fine-tuned further on. I did these for Muril and Rembert, both of which gave a consistent 0.8 - 0.81 across the LB. Private LB scores were also in the higher range for Muril and Rembert fine-tuned on top of Squad2. (0.76 - 0.77)
\nChaii dataset is a bit noisy on its own. I didn't fix any incorrect labels. But I fixed the answer-start values by picking the first matching answer candidate from @thedrcat 's fine-tuned version (that he published on the first week of this competition).
\nI trained XLMR, Muril-Large, and Rembert over their Squad2 checkpoints, with Chaii dataset, MLQA Hindi, XQuaD Hindi, TydiQA Telugu and Bengali datasets. Didn't do anything wrt the model perse. Just playing around with hyperparameters, etc. But because this competition provides an entire passage as a context, this is as much as a passage selection task as it is a question-answering task. So I maintained a 1:2 ratio of relevant vs irrelevant contexts, by sampling on Chaii's train dataset and using the remaining external data as-is (they were smaller passages) for all models' training.
\nRembert is a real humdinger of a model. It had the most NLU capabilities among the three model-types (at least the way I trained them). Case in point:
\nThere was a question about the largest mountain in Chaii Dataset and the human annotator marked Mount Everest as the answer. But Rembert (Squad2 checkpoint. Not the Chaii checkpoint) predicted Mauna Loa because the context read The tallest mountain doesn't necessarily mean the largest. For instance, Mauna Loa has a base area of ....
I picked the most interesting example. But it consistently out-performed both Muril-Large and XLMR when it came to picking the right context. It performed well on extracting outputs from Wikipedia's trivia boxes too. It didn't mark the exact word boundaries for predicted answers. But it almost always had the right answer.
\nThis is where I messed up with selecting the submissions. My best internal submission came from a 3-fold Rembert (0.778). But since I assumed that Rembert wasn't right in generating answer boundaries, I fed its passage outputs to Muril-Large to generate word-boundaries. This reduced the scores to 0.757 :P It was a bit of a gamble between relevant context selection vs answer selection, and I picked an approach that weaves both. This didn't work out so well in retrospect :)
\nWill cleanup the notebooks and publish them later today. But for now, here are the Squad2 checkpoints that I fine-tuned on Rembert and Muril-Large.
\n\nThe low-resource nature, combined with the morphological richness of Hindi and Tamil in NLP scene restricted the usage of better pre-training approaches like MPNet and Electra, or even SOTA QA models like LUKE. Hindi is a fusional language. So a word can have multiple morphemes depending on gender, tense, etc (Like go vs went in English). The current fix (that seems to be working well) to compensate for this is to train on a large amount of data.
Agglutinative languages like Tamil are a different kind of reality altogether. One thumb rule I learnt during this competition to evaluate models for agglutinative languages is to look for models' metrics on Turkish datasets (Turkish is an agglutinative beast). Where it isn't available, metrics on Japanese datasets will work just as fine. Character-based transformers like Canine and Charformers sound like promising approaches for this type of language. They aren't mature/stable yet. But this is NLP. They might very well be the norm this time next year.
\nThis competition served as an exponential learning curve for me. So much so that, during the start of the competition, I look for metrics in published NLP papers. By the end of the competition, it was more about ablation studies. It's also the first time I was a bit active in forums, GitHub, etc. So the confidence boost helped a lot too. I might be a bit more active on Kaggle going forward, and invest in a decent GPU source too :)
\nEdit: Final submission inference notebook. Even though this approach didn't work in this particular competition, the usage of a QA model for passage selection might still benefit someone in a better-labelled dataset. All inputs are the three types of the models. Of the two utility scripts, hf-qa-utils is mostly boilerplate code from the HuggingFace example notebooks to tokenize and map back datasets. passage-qa-utils does the sub-context selection from a Rembert QA model.
A second approach of using 3 individual model outputs and passing them through an LGBM Ranker also faired quite well in private LB, that I forgot to mention in the original post.
", "rawMarkdown": "Thanks to Kaggle and Google India for hosting this competition and Congratulations to all the winners. That there are many medal winners who don't know either or both languages is very good news for low-resource languages in general. \n\n## Data Augmentation and Data fixes\nLike many other competitors observed, adding TydiQA Telugu and Bengali datasets to the mix certainly boosted the model performances both in CV and LB. This shouldn't be surprising at all since most of us are using multilingual models and adding any Indic QA dataset will have a lot more real-world context overlap. \n\nOne other thing I usually do for all single-answer datasets is to fine-tune on Squad2 or other multiple-answer datasets first, instead of directly fine-tuning on an MLM checkpoint. This gives the models a better intuition on *not ignoring* missed labels. XLMR already comes with a model we all fine-tuned further on. I did these for Muril and Rembert, both of which gave a consistent 0.8 - 0.81 across the LB. Private LB scores were also in the higher range for Muril and Rembert fine-tuned on top of Squad2. (0.76 - 0.77) \n\nChaii dataset is a bit noisy on its own. I didn't fix any incorrect labels. But I [fixed the answer-start values](https://www.kaggle.com/jdoesv/chaii-realigned-answers-single) by picking the first matching answer candidate from @thedrcat 's fine-tuned version (that he published on the first week of this competition). \n\n## Models\nI trained XLMR, Muril-Large, and Rembert over their Squad2 checkpoints, with Chaii dataset, MLQA Hindi, XQuaD Hindi, TydiQA Telugu and Bengali datasets. Didn't do anything wrt the model perse. Just playing around with hyperparameters, etc. But because this competition provides an entire passage as a context, this is as much as a passage selection task as it is a question-answering task. So I maintained a 1:2 ratio of relevant vs irrelevant contexts, by sampling on Chaii's train dataset and using the remaining external data as-is (they were smaller passages) for all models' training.\n\nRembert is a real humdinger of a model. It had the most NLU capabilities among the three model-types (at least the way I trained them). Case in point: \n\nThere was a question about `the largest mountain` in Chaii Dataset and the human annotator marked `Mount Everest` as the answer. But Rembert (Squad2 checkpoint. Not the Chaii checkpoint) predicted `Mauna Loa` because the context read `The tallest mountain doesn't necessarily mean the largest. For instance, Mauna Loa has a base area of ...`. \n\nI picked the most interesting example. But it consistently out-performed both Muril-Large and XLMR when it came to picking the right context. It performed well on extracting outputs from Wikipedia's trivia boxes too. It didn't mark the exact word boundaries for predicted answers. But it almost always had the right answer. \n\n## Ensembling\nThis is where I messed up with selecting the submissions. My best internal submission came from a 3-fold Rembert (0.778). But since I assumed that Rembert wasn't right in generating answer boundaries, I fed its passage outputs to Muril-Large to generate word-boundaries. This reduced the scores to 0.757 :P It was a bit of a gamble between relevant context selection vs answer selection, and I picked an approach that weaves both. This didn't work out so well in retrospect :)\n\nWill cleanup the notebooks and publish them later today. But for now, here are the Squad2 checkpoints that I fine-tuned on Rembert and Muril-Large.\n- [Rembert Squad2 Finetuned checkpoint](https://huggingface.co/Sindhu/rembert-squad2)\n- [Muril-Large Squad2 Finetuened checkpoint](https://huggingface.co/Sindhu/muril-large-squad2) \n\n## Some other interesting approaches\nThe low-resource nature, combined with the morphological richness of Hindi and Tamil in NLP scene restricted the usage of better pre-training approaches like MPNet and Electra, or even SOTA QA models like LUKE. Hindi is a fusional language. So a word can have multiple morphemes depending on gender, tense, etc (Like `go` vs `went` in English). The current fix (that seems to be working well) to compensate for this is to train on a large amount of data. \n\nAgglutinative languages like Tamil are a different kind of reality altogether. One thumb rule I learnt during this competition to evaluate models for agglutinative languages is to look for models' metrics on Turkish datasets (Turkish is an agglutinative beast). Where it isn't available, metrics on Japanese datasets will work just as fine. Character-based transformers like [Canine](https://huggingface.co/google/canine-c) and [Charformers](https://arxiv.org/pdf/2106.12672.pdf) sound like promising approaches for this type of language. They aren't mature/stable yet. But this is NLP. They might very well be the norm this time next year. \n\nThis competition served as an exponential learning curve for me. So much so that, during the start of the competition, I look for metrics in published NLP papers. By the end of the competition, it was more about ablation studies. It's also the first time I was a bit active in forums, GitHub, etc. So the confidence boost helped a lot too. I might be a bit more active on Kaggle going forward, and invest in a decent GPU source too :) \n\n**Edit:** [Final submission inference notebook](https://www.kaggle.com/jdoesv/clean-rembert-feat-selector-xlmr-muril-ans-gen). Even though this approach didn't work in this particular competition, the usage of a QA model for passage selection might still benefit someone in a better-labelled dataset. All inputs are the three types of the models. Of the two utility scripts, `hf-qa-utils` is mostly boilerplate code from the HuggingFace example notebooks to tokenize and map back datasets. `passage-qa-utils` does the sub-context selection from a Rembert QA model. \n\n[A second approach](https://www.kaggle.com/jdoesv/custom-ensemble-outputs?scriptVersionId=77964669) of using 3 individual model outputs and passing them through an LGBM Ranker also faired quite well in private LB, that I forgot to mention in the original post.", "votes": 18 }, { "id": 1584848, "postDate": "2021-11-16T20:03:32.810Z", "content": "Congrats @jdoesv you did great on this competition, and sorry for the bad luck with the shake down… In the long run, it's the learning and knowledge progress that matters, and that was a definite win! Looking forward to see you in the next Kaggle comp!
", "rawMarkdown": "Congrats @jdoesv you did great on this competition, and sorry for the bad luck with the shake down... In the long run, it's the learning and knowledge progress that matters, and that was a definite win! Looking forward to see you in the next Kaggle comp!", "votes": 1, "replies": [ { "id": 1585155, "postDate": "2021-11-17T05:59:51.963Z", "content": "Thank you for your kind words @thedrcat !😄
", "rawMarkdown": "Thank you for your kind words @thedrcat !😄" } ] }, { "id": 1586742, "postDate": "2021-11-18T08:36:49.617Z", "content": "Congrats on a great finish! Thanks for introducing Muril on this competition for the first time and nice to hear that Rembert does well too!
", "rawMarkdown": "Congrats on a great finish! Thanks for introducing Muril on this competition for the first time and nice to hear that Rembert does well too!" }, { "id": 1584255, "postDate": "2021-11-16T11:46:54.320Z", "content": "Also for people who know their Tamil pop culture: https://tinyurl.com/otherbanana Couldn't help, now that the competition is over! :)
", "rawMarkdown": "Also for people who know their Tamil pop culture: https://tinyurl.com/otherbanana Couldn't help, now that the competition is over! :)" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:23:26Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7258863, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/filter-coffee-16th-place-solution-data-augmentatio" }