I would like to thank Kaggle and Chaii (Challenge in AI for India) for setting up this interesting Multilingual Question Answering Challenge.
\nIt turns out we have a couple of solutions that would put us Gold Position but that is how these competitions go sometimes!
\nAcknowledgements
\nA big shout out to @kishalmandal, @abhishek and @nbroad for their wonderful kernels and models that we could leverage.
\nAlso thanks to my wonderful teammates @khyeh0719 and @maxjeblick for their contributions to our solution. This was a team effort!
\nExternal Datasets
\nThe models in the final pipeline used a mix of the baseline dataset and the MLQA and Squad datasets that were disclosed early on in the competition in the datasets page here: https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/264581
\nModels
\nAll models below were trained with doc_stride=135, max_length=400
\n| Model | \nLB Score | \nComments | \n# Folds Used | \n
|---|---|---|---|
| deepset/xlm-roberta-large-squad2 | \n0.792 | \n5 fold Public Model | \n5 | \n
| deepset/xlm-roberta-large-squad2 | \n0.781 | \n10 fold Tez model Trained Locally | \n3 | \n
| google/rembert | \n0.777 | \n5 Fold Trained Locally | \n4 | \n
| google/muril-base-cased | \n0.772 | \n5 Fold Trained Locally | \n4 | \n
Apart from the models above we tried various different architectures including but not limited to electra, muril-base, multilingual BERT, span BERT, tamillion, indic-bert but none of them were as good as the architectures in the table for this task.
\nThere was very little CV -> LB correlation in the competition for us. High CV scores did poorly on LB while decent but not too poor CV results gave higher LB scores. I suspect that this is due to the noise in the dataset and also that private dataset is cleaner with multiple annotators labeling the data.
\nSpeeding up calculations
\n - model.half() - FP16 for slight improvement in speed and no impact to LB score
\n - Reusing features for same archs
\n - Torchscript/jit - Experiments resulted in out of memory
\n - Tried joblib and multiprocessing to do parallel calculations - All experiments worked on train set but out of memory on test. Maybe because test set had longer contexts?
Ensemble Strategies
\nOne of the main challenges was the 5 hr kernel run limit. With different architectures and possibly different max_len/doc_stride it became pretty evident that we needed a good way to ensemble models. Averaging logits only works for same archs and if same doc_stride/max len is used.
\n I read up best practises on how do to this including top solution in the TensorFlow Question Answering Challenge here: https://www.kaggle.com/c/tensorflow2-question-answering/discussion/128832 . I came across majority/hard voting in the solution descriptions and decided to implement a very simple voting algorithm that landed up giving a significant boost to LB score.
Voting
\nThe voter took the cleaned predictions (after basic post processing) as inputs. We had 4 architectures in the final pipeline and the first stage checked for a majority vote among the 4 candidate predictions based on the top answer from each architecture. If there was no majority, the voting pool was expanded to top N predictions among all architectures. We found that taking top 2 votes from each arch gave good results. To further improve on the 2nd stage of the voting, if there was still a deadlock in voting, the votes were compared for similarity using the Levenshtein Ratio and if votes were within a certain threshold, one of those answers were selected as the final since the metric is Jaccard and having part of the answer correct would still be useful.
Score progression
\n - Public LB kernel -> LB 0.792
\n - Voting ensemble with 3 different architectures -> LB 0.798 (+0.006)
\n - Improving models in voting -> LB 0.801 (+0.003)
\n - Adding another architecture to voting and selecting best models -> LB 0.809 (+0.003)
\n - Fine tuning voting logic (see above) -> LB 0.811
Code can be found here: https://www.kaggle.com/trushk/qa-ensemble-voter-top-n-votes-similarity
\nDescription of voting logic: https://www.kaggle.com/trushk/ensemble-using-voting-13th-place-solution
", "comments": [ { "id": 1583913, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-11-16T06:26:25.200000", "content": "Congratulations @trushk and team and thanks for sharing the writeup
", "votes": 3, "replies": [ { "id": 1583960, "author_name": "Trushant Kalyanpur", "author_url": "", "post_date": "2021-11-16T07:05:31.637000", "content": "Thanks @duykhanh99 !
", "votes": 0, "replies": [] } ] }, { "id": 1584211, "author_name": "Max Jeblick", "author_url": "", "post_date": "2021-11-16T10:45:04.583000", "content": "Things which didn't work (many 😄):
\nMy general impression after this competition (and having worked with quite a few NLP datasets) is that the default settings (AdamW, lr~O(1e-5)) are near optimal for many problem types.
answer_start being always the first occurrence which might not be true). As I'm not a speaker of Hindi nor Tamil, I tried to find an automated way how to fix these labeling issues. What I tried is to mark all occurrences of answer_text in the first context[:N] as a target. That is, if the model's input is <Question> <Context Chunk_i> and <Context Chunk_i> is from the beginning of the text and contains answer_text, the corresponding target will be the start and end position of that answer.deepset/xlm-roberta-squad2) don't use token_type_ids. Yet, they may be important, as the input is given by <Question> <Context Chunk_i>. I tried adding token_type_ids back to xlm-roberta models, with 0 denoting the question and 1 denoting the answer. Embedding was initialized with the token_type_id==0 embedding vector of the respective pretrained model (token_type_ids embedding layer is still used by many models, the lookup is always for 0). Surprisingly, OOF scores didn't improve.deepset/xlm-roberta-squad2 on translated squad2 first, scores where slightly worsescore = start_logits + end_logits). For us, it improved oof scores, but lowered public LB.Congratulations @trushk !😄
", "votes": 1, "replies": [] }, { "id": 1590698, "author_name": "MINGOOK KIM", "author_url": "", "post_date": "2021-11-21T15:08:37.657000", "content": "Congratulations on your achievements.
\nCongratulations~
", "votes": 1, "replies": [] }, { "id": 1586348, "author_name": "Ayush Singh", "author_url": "", "post_date": "2021-11-18T02:23:04.973000", "content": "Congratulations @trushk very nice approach. and thanks for sharing the code as well. It is very helpful.
", "votes": 1, "replies": [] }, { "id": 1585924, "author_name": "Tsuno", "author_url": "", "post_date": "2021-11-17T16:44:25.987000", "content": "Congratulations
", "votes": 1, "replies": [] }, { "id": 1584860, "author_name": "Udbhav Bamba", "author_url": "", "post_date": "2021-11-16T20:16:07.630000", "content": "Thanks for the overview. The voting logic seems very fascinating.
", "votes": 1, "replies": [] }, { "id": 1584837, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-11-16T19:54:33.327000", "content": "Congratulations, an interesting solution! I also used majority voting across 3 models blends but in case of no agreement I just stuck defaulted to the best model (MURIL in my case) predictions. Need to try out your method!
", "votes": 1, "replies": [ { "id": 1584841, "author_name": "Trushant Kalyanpur", "author_url": "", "post_date": "2021-11-16T19:57:52.983000", "content": "Thanks @thedrcat. Yes, that is what I had started off with but I found that increasing the voting pool increased the LB score by +0.004 and further checking for similar answers improved the score by +.001
", "votes": 1, "replies": [] } ] }, { "id": 1584516, "author_name": "Mathurin Ache", "author_url": "", "post_date": "2021-11-16T15:12:48.620000", "content": "Great Overview of Your Solution. Thank you
", "votes": 1, "replies": [] }, { "id": 1584034, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-11-16T08:16:22.567000", "content": "I also notice that in the end you used 16 folds :-)
", "votes": 1, "replies": [] }, { "id": 1584033, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-11-16T08:15:54.460000", "content": "We struggled with voting (just used most voted solution, breaking ties by length - shorter favored) and I like very much your voting solution (If there was no majority, the voting pool was expanded to top N predictions among all architectures / votes were compared for similarity using the Levenshtein Ratio). Great solution! I will surely use that on the next QA competition/problem :-)
", "votes": 1, "replies": [] }, { "id": 1583663, "author_name": "Kiran Chaudhary", "author_url": "", "post_date": "2021-11-16T01:42:42.700000", "content": "Congratulations!!!!
", "votes": 1, "replies": [] }, { "id": 1583613, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T01:03:24.763000", "content": "Congratulations! When ensembling different models did you need to convert to a common tokenization? Do the different models use different tokenization?
", "votes": 1, "replies": [ { "id": 1583621, "author_name": "Trushant Kalyanpur", "author_url": "", "post_date": "2021-11-16T01:09:45.320000", "content": "Yes different archs used different tokenizations. That coupled with different doc stride and max-len means that start/end logits have different shapes. So I just ensembled based on multi stage voting of count of predictions- Kernel is here: https://www.kaggle.com/trushk/qa-ensemble-voter-top-n-votes-similarity#Change-vote-count-if-votes-are-similar
", "votes": 1, "replies": [] }, { "id": 1583738, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-11-16T03:14:12.387000", "content": "Great solution for ensemble. Well done!
", "votes": 1, "replies": [] } ] } ], "comment_count": 17, "messages": [ { "id": 1583570, "postDate": "2021-11-16T00:24:24.297Z", "content": "I would like to thank Kaggle and Chaii (Challenge in AI for India) for setting up this interesting Multilingual Question Answering Challenge.
\nIt turns out we have a couple of solutions that would put us Gold Position but that is how these competitions go sometimes!
\nAcknowledgements
\nA big shout out to @kishalmandal, @abhishek and @nbroad for their wonderful kernels and models that we could leverage.
\nAlso thanks to my wonderful teammates @khyeh0719 and @maxjeblick for their contributions to our solution. This was a team effort!
\nExternal Datasets
\nThe models in the final pipeline used a mix of the baseline dataset and the MLQA and Squad datasets that were disclosed early on in the competition in the datasets page here: https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/264581
\nModels
\nAll models below were trained with doc_stride=135, max_length=400
\n| Model | \nLB Score | \nComments | \n# Folds Used | \n
|---|---|---|---|
| deepset/xlm-roberta-large-squad2 | \n0.792 | \n5 fold Public Model | \n5 | \n
| deepset/xlm-roberta-large-squad2 | \n0.781 | \n10 fold Tez model Trained Locally | \n3 | \n
| google/rembert | \n0.777 | \n5 Fold Trained Locally | \n4 | \n
| google/muril-base-cased | \n0.772 | \n5 Fold Trained Locally | \n4 | \n
Apart from the models above we tried various different architectures including but not limited to electra, muril-base, multilingual BERT, span BERT, tamillion, indic-bert but none of them were as good as the architectures in the table for this task.
\nThere was very little CV -> LB correlation in the competition for us. High CV scores did poorly on LB while decent but not too poor CV results gave higher LB scores. I suspect that this is due to the noise in the dataset and also that private dataset is cleaner with multiple annotators labeling the data.
\nSpeeding up calculations
\n - model.half() - FP16 for slight improvement in speed and no impact to LB score
\n - Reusing features for same archs
\n - Torchscript/jit - Experiments resulted in out of memory
\n - Tried joblib and multiprocessing to do parallel calculations - All experiments worked on train set but out of memory on test. Maybe because test set had longer contexts?
Ensemble Strategies
\nOne of the main challenges was the 5 hr kernel run limit. With different architectures and possibly different max_len/doc_stride it became pretty evident that we needed a good way to ensemble models. Averaging logits only works for same archs and if same doc_stride/max len is used.
\n I read up best practises on how do to this including top solution in the TensorFlow Question Answering Challenge here: https://www.kaggle.com/c/tensorflow2-question-answering/discussion/128832 . I came across majority/hard voting in the solution descriptions and decided to implement a very simple voting algorithm that landed up giving a significant boost to LB score.
Voting
\nThe voter took the cleaned predictions (after basic post processing) as inputs. We had 4 architectures in the final pipeline and the first stage checked for a majority vote among the 4 candidate predictions based on the top answer from each architecture. If there was no majority, the voting pool was expanded to top N predictions among all architectures. We found that taking top 2 votes from each arch gave good results. To further improve on the 2nd stage of the voting, if there was still a deadlock in voting, the votes were compared for similarity using the Levenshtein Ratio and if votes were within a certain threshold, one of those answers were selected as the final since the metric is Jaccard and having part of the answer correct would still be useful.
Score progression
\n - Public LB kernel -> LB 0.792
\n - Voting ensemble with 3 different architectures -> LB 0.798 (+0.006)
\n - Improving models in voting -> LB 0.801 (+0.003)
\n - Adding another architecture to voting and selecting best models -> LB 0.809 (+0.003)
\n - Fine tuning voting logic (see above) -> LB 0.811
Code can be found here: https://www.kaggle.com/trushk/qa-ensemble-voter-top-n-votes-similarity
\nDescription of voting logic: https://www.kaggle.com/trushk/ensemble-using-voting-13th-place-solution
", "rawMarkdown": "I would like to thank Kaggle and Chaii (Challenge in AI for India) for setting up this interesting Multilingual Question Answering Challenge. \n\nIt turns out we have a couple of solutions that would put us Gold Position but that is how these competitions go sometimes! \n\n\n**Acknowledgements**\n\nA big shout out to @kishalmandal, @abhishek and @nbroad for their wonderful kernels and models that we could leverage. \n\nAlso thanks to my wonderful teammates @khyeh0719 and @maxjeblick for their contributions to our solution. This was a team effort! \n\n\n**External Datasets**\n\nThe models in the final pipeline used a mix of the baseline dataset and the MLQA and Squad datasets that were disclosed early on in the competition in the datasets page here: https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/264581 \n\n\n**Models**\n\nAll models below were trained with doc_stride=135, max_length=400\n\n| Model | LB Score | Comments | # Folds Used |\n| --- | --- | --- | --- |\n| deepset/xlm-roberta-large-squad2 | 0.792 | 5 fold Public Model | 5 |\n| deepset/xlm-roberta-large-squad2 | 0.781 | 10 fold Tez model Trained Locally | 3 |\n|google/rembert | 0.777 | 5 Fold Trained Locally | 4 |\n| google/muril-base-cased | 0.772 | 5 Fold Trained Locally | 4 |\n\n\nApart from the models above we tried various different architectures including but not limited to **electra, muril-base, multilingual BERT, span BERT, tamillion, indic-bert** but none of them were as good as the architectures in the table for this task. \n\nThere was very little CV -> LB correlation in the competition for us. High CV scores did poorly on LB while decent but not too poor CV results gave higher LB scores. I suspect that this is due to the noise in the dataset and also that private dataset is cleaner with multiple annotators labeling the data. \n\n\n**Speeding up calculations**\n\t- `model.half()` - FP16 for slight improvement in speed and no impact to LB score\n\t- Reusing features for same archs\n\t- Torchscript/jit - Experiments resulted in out of memory\n\t- Tried joblib and multiprocessing to do parallel calculations - All experiments worked on train set but out of memory on test. Maybe because test set had longer contexts? \n\n**Ensemble Strategies**\nOne of the main challenges was the 5 hr kernel run limit. With different architectures and possibly different max_len/doc_stride it became pretty evident that we needed a good way to ensemble models. Averaging logits only works for same archs and if same doc_stride/max len is used.\n I read up best practises on how do to this including top solution in the TensorFlow Question Answering Challenge here: https://www.kaggle.com/c/tensorflow2-question-answering/discussion/128832 . I came across majority/hard voting in the solution descriptions and decided to implement a very simple voting algorithm that landed up giving a significant boost to LB score. \n\n**Voting**\nThe voter took the cleaned predictions (after basic post processing) as inputs. We had 4 architectures in the final pipeline and the first stage checked for a majority vote among the 4 candidate predictions based on the top answer from each architecture. If there was no majority, the voting pool was expanded to top N predictions among all architectures. We found that taking top 2 votes from each arch gave good results. To further improve on the 2nd stage of the voting, if there was still a deadlock in voting, the votes were compared for similarity using the Levenshtein Ratio and if votes were within a certain threshold, one of those answers were selected as the final since the metric is Jaccard and having part of the answer correct would still be useful. \n\n\n\n**Score progression** \n\t- Public LB kernel -> **LB 0.792** \n\t- Voting ensemble with 3 different architectures -> **LB 0.798** (+0.006)\n\t- Improving models in voting -> **LB 0.801** (+0.003)\n\t- Adding another architecture to voting and selecting best models -> **LB 0.809** (+0.003)\n\t- Fine tuning voting logic (see above) -> **LB 0.811**\n\nCode can be found here: https://www.kaggle.com/trushk/qa-ensemble-voter-top-n-votes-similarity\n\nDescription of voting logic: https://www.kaggle.com/trushk/ensemble-using-voting-13th-place-solution", "votes": 29 }, { "id": 1583913, "postDate": "2021-11-16T06:26:25.200Z", "content": "Congratulations @trushk and team and thanks for sharing the writeup
", "rawMarkdown": "Congratulations @trushk and team and thanks for sharing the writeup ", "votes": 3, "replies": [ { "id": 1583960, "postDate": "2021-11-16T07:05:31.637Z", "content": "Thanks @duykhanh99 !
", "rawMarkdown": "Thanks @duykhanh99 !" } ] }, { "id": 1584211, "postDate": "2021-11-16T10:45:04.583Z", "content": "Things which didn't work (many 😄):
\nMy general impression after this competition (and having worked with quite a few NLP datasets) is that the default settings (AdamW, lr~O(1e-5)) are near optimal for many problem types.
answer_start being always the first occurrence which might not be true). As I'm not a speaker of Hindi nor Tamil, I tried to find an automated way how to fix these labeling issues. What I tried is to mark all occurrences of answer_text in the first context[:N] as a target. That is, if the model's input is <Question> <Context Chunk_i> and <Context Chunk_i> is from the beginning of the text and contains answer_text, the corresponding target will be the start and end position of that answer.deepset/xlm-roberta-squad2) don't use token_type_ids. Yet, they may be important, as the input is given by <Question> <Context Chunk_i>. I tried adding token_type_ids back to xlm-roberta models, with 0 denoting the question and 1 denoting the answer. Embedding was initialized with the token_type_id==0 embedding vector of the respective pretrained model (token_type_ids embedding layer is still used by many models, the lookup is always for 0). Surprisingly, OOF scores didn't improve.deepset/xlm-roberta-squad2 on translated squad2 first, scores where slightly worsescore = start_logits + end_logits). For us, it improved oof scores, but lowered public LB.Congratulations @trushk !😄
", "rawMarkdown": "Congratulations @trushk !😄", "votes": 1 }, { "id": 1590698, "postDate": "2021-11-21T15:08:37.657Z", "content": "Congratulations on your achievements.
\nCongratulations~
", "rawMarkdown": "Congratulations on your achievements.\n\nCongratulations~", "votes": 1 }, { "id": 1586348, "postDate": "2021-11-18T02:23:04.973Z", "content": "Congratulations @trushk very nice approach. and thanks for sharing the code as well. It is very helpful.
", "rawMarkdown": "Congratulations @trushk very nice approach. and thanks for sharing the code as well. It is very helpful. ", "votes": 1 }, { "id": 1585924, "postDate": "2021-11-17T16:44:25.987Z", "content": "Congratulations
", "rawMarkdown": "Congratulations", "votes": 1 }, { "id": 1584860, "postDate": "2021-11-16T20:16:07.630Z", "content": "Thanks for the overview. The voting logic seems very fascinating.
", "rawMarkdown": "Thanks for the overview. The voting logic seems very fascinating. ", "votes": 1 }, { "id": 1584837, "postDate": "2021-11-16T19:54:33.327Z", "content": "Congratulations, an interesting solution! I also used majority voting across 3 models blends but in case of no agreement I just stuck defaulted to the best model (MURIL in my case) predictions. Need to try out your method!
", "rawMarkdown": "Congratulations, an interesting solution! I also used majority voting across 3 models blends but in case of no agreement I just stuck defaulted to the best model (MURIL in my case) predictions. Need to try out your method!", "votes": 1, "replies": [ { "id": 1584841, "postDate": "2021-11-16T19:57:52.983Z", "content": "Thanks @thedrcat. Yes, that is what I had started off with but I found that increasing the voting pool increased the LB score by +0.004 and further checking for similar answers improved the score by +.001
", "rawMarkdown": "Thanks @thedrcat. Yes, that is what I had started off with but I found that increasing the voting pool increased the LB score by +0.004 and further checking for similar answers improved the score by +.001", "votes": 1 } ] }, { "id": 1584516, "postDate": "2021-11-16T15:12:48.620Z", "content": "Great Overview of Your Solution. Thank you
", "rawMarkdown": "Great Overview of Your Solution. Thank you", "votes": 1 }, { "id": 1584034, "postDate": "2021-11-16T08:16:22.567Z", "content": "I also notice that in the end you used 16 folds :-)
", "rawMarkdown": "I also notice that in the end you used 16 folds :-)", "votes": 1 }, { "id": 1584033, "postDate": "2021-11-16T08:15:54.460Z", "content": "We struggled with voting (just used most voted solution, breaking ties by length - shorter favored) and I like very much your voting solution (If there was no majority, the voting pool was expanded to top N predictions among all architectures / votes were compared for similarity using the Levenshtein Ratio). Great solution! I will surely use that on the next QA competition/problem :-)
", "rawMarkdown": "We struggled with voting (just used most voted solution, breaking ties by length - shorter favored) and I like very much your voting solution (If there was no majority, the voting pool was expanded to top N predictions among all architectures / votes were compared for similarity using the Levenshtein Ratio). Great solution! I will surely use that on the next QA competition/problem :-)", "votes": 1 }, { "id": 1583663, "postDate": "2021-11-16T01:42:42.700Z", "content": "Congratulations!!!!
", "rawMarkdown": "Congratulations!!!!", "votes": 1 }, { "id": 1583613, "postDate": "2021-11-16T01:03:24.763Z", "content": "Congratulations! When ensembling different models did you need to convert to a common tokenization? Do the different models use different tokenization?
", "rawMarkdown": "Congratulations! When ensembling different models did you need to convert to a common tokenization? Do the different models use different tokenization?", "votes": 1, "replies": [ { "id": 1583621, "postDate": "2021-11-16T01:09:45.320Z", "content": "Yes different archs used different tokenizations. That coupled with different doc stride and max-len means that start/end logits have different shapes. So I just ensembled based on multi stage voting of count of predictions- Kernel is here: https://www.kaggle.com/trushk/qa-ensemble-voter-top-n-votes-similarity#Change-vote-count-if-votes-are-similar
", "rawMarkdown": "Yes different archs used different tokenizations. That coupled with different doc stride and max-len means that start/end logits have different shapes. So I just ensembled based on multi stage voting of count of predictions- Kernel is here: https://www.kaggle.com/trushk/qa-ensemble-voter-top-n-votes-similarity#Change-vote-count-if-votes-are-similar", "votes": 1 }, { "id": 1583738, "postDate": "2021-11-16T03:14:12.387Z", "content": "Great solution for ensemble. Well done!
", "rawMarkdown": "Great solution for ensemble. Well done!", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:22:06Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7262703, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/chaii-paani-13th-place-solution" }