First of all thanks to Kaggle and the organizers for this competition.
\nThe competition data was not up to the mark and thus, there was no way you could build a good CV that correlates with the leaderboard. Unclean public training data and clean hidden private test data only meant we could use either different models and combine them together or use a single strong model like muril-large and train it on a variety of datasets shared by others. We did both.
\nThe first step was to divide the training data into 5 folds. After that, we trained our final models on subset of SQuAD v2, Google-translated SQuAD v2 shared by @tkm2261, all TyDi QA, hindi parts of MLQA, XQUAD and 5-10x oversampled Chaii dataset for each fold. We found that oversampling Chaii dataset helped us to improve CV and LB score. Please note that for CV calculation we used only the original validation folds.
\nAll Models were trained for 1-2 epochs with early stopping based on CV Jaccard score with max_length = 384 and doc_stride = 128. We did try other lengths such as 400, 512 and other strides like 192 but they didn’t perform very well with our cross-validation strategy.
\nSince different models have different tokenizations, simple average ensembling was not possible so tried using prediction scores to select best predictions from different models. This gave us a little improvement.
\nThe next thing we tried was to normalize the scores of all models to the same scale. We saw a little improvement using this too but we knew that this is not the best way to combine models.
\nThe vanilla version failed as different models had different scales for logits. So to overcome this I came up with a custom softmax which could normalize the logits across different context splits. Base code for this is attached below:
\ndef CustomSoftmax(examples, features, raw_predictions):\n all_start_logits, all_end_logits = raw_predictions\n\n # Build a map example to its corresponding features.\n example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n features_per_example = collections.defaultdict(list)\n for i, feature in enumerate(features):\n features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n\n for example_index, example in enumerate(tqdm(examples)):\n feature_indices = features_per_example[example_index]\n start_logits_sum = 0\n end_logits_sum = 0\n\n logCs = None\n logCe = None\n\n # Find maximum logit value across context\n for feature_index in feature_indices:\n if logCs is None:\n logCs = all_start_logits[feature_index].max()\n else:\n logCs = max(logCs, all_start_logits[feature_index].max())\n\n if logCe is None:\n logCe = all_end_logits[feature_index].max()\n else:\n logCe = max(logCe, all_end_logits[feature_index].max())\n\n # Subtract maximum value among all logits before applying exponentiation for numerical stability\n # And take sum across all context splits\n for feature_index in feature_indices:\n all_start_logits[feature_index] -= logCs\n all_end_logits[feature_index] -= logCe\n start_logits_sum += np.exp(all_start_logits[feature_index]).sum()\n end_logits_sum += np.exp(all_end_logits[feature_index]).sum()\n\n # Softmax\n for feature_index in feature_indices:\n all_start_logits[feature_index] = np.exp(all_start_logits[feature_index])/start_logits_sum\n all_end_logits[feature_index] = np.exp(all_end_logits[feature_index])/end_logits_sum\n\n return all_start_logits, all_end_logits\nThis was the final idea we had in the last two days and it boosted our public score from 0.809 to 0.821!
\nWe followed a similar preprocessing technique as used in public notebooks. Though we observed that removing a few of the symbols were hurting the performance so we decided to use a custom punctuations list ('!\"#$&\\'()*+,-/:;<=>?@[\\\\]^_`{|}~').
We also tried to solve the issue of tamil predictions as indicated by @jdoesv in https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/264831 with the help of stemming tamil words in the answers it helped to boost Muril performance from 0.802 -> 0.804 on the public LB.
\nIn the end, we didn’t add many rule-based post-processing as we thought this might overfit the public leaderboard.
\nOur best private submission would have scored 0.779 but we did not select it. It’s funny why we didn’t select that submission. It had a small bug. However, even with that bug, that submission was our best private submission :)
\nWe also had a single 5 fold muril-large model that would have placed us “in the money”.
\nOur best inference kernel can be found here: https://www.kaggle.com/abhishek/chaii-xlm-roberta-x-muril-x-rembert-score-based
\nYou can also try one of our best muril-large models on Hugging Face Hub: https://huggingface.co/abhishek/muril-large-chaii
\nWe will also publish our training code soon.
\nIt was a fun competition but we feel that it could have been better if the provided training data was better or at least similar to the hidden test dataset. We gave up many times.
\nWe also tried a meta-model to re-rank the answers generated by different models but it was quite late in the competition (on the final day) and we didn’t have enough time to experiment with it further.
\nP.S. @abhishek still feels that this competition was all about data collection from hundreds of participants.
\nP.S.S Training code can be found at: https://github.com/abhishekkrthakur/chaii-hindi-tamil-question-answering
", "comments": [ { "id": 1584826, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-11-16T19:39:03.570000", "content": "Congrats @ubamba98 and @abhishek, need to spend some time to understand your ensembling method, my mind is still dumb since yesterday 😂It was great to see your climb in the last days and defending the top position on private!
", "votes": 3, "replies": [] }, { "id": 1584377, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2021-11-16T13:47:19.590000", "content": "I had high hopes for re-ranking model. Too bad it didn't work out and we tried it towards the end and had no time to experiment :D
", "votes": 3, "replies": [] }, { "id": 1593954, "author_name": "Kiran Sharma", "author_url": "", "post_date": "2021-11-24T12:21:31.910000", "content": "Congretzz
", "votes": 1, "replies": [] }, { "id": 1591244, "author_name": "Peter", "author_url": "", "post_date": "2021-11-22T05:47:01.107000", "content": "Congratulations!
", "votes": 1, "replies": [] }, { "id": 1585865, "author_name": "Dhinahar P", "author_url": "", "post_date": "2021-11-17T16:01:41.730000", "content": "Well Done brother nice work
", "votes": 1, "replies": [] }, { "id": 1585166, "author_name": "adldotori", "author_url": "", "post_date": "2021-11-17T06:12:01.077000", "content": "Congrats. i'm envious of gold medal.
", "votes": 2, "replies": [] }, { "id": 1585126, "author_name": "Ankit Maurya", "author_url": "", "post_date": "2021-11-17T05:30:21.223000", "content": "you guys are amazing
", "votes": 2, "replies": [] }, { "id": 1584646, "author_name": "ManogyaK", "author_url": "", "post_date": "2021-11-16T16:46:56.493000", "content": "This is really amazing Udbhav! Congratulations and truly well deserved.
", "votes": 2, "replies": [] }, { "id": 1584635, "author_name": "Raghawendra Singh", "author_url": "", "post_date": "2021-11-16T16:36:36.073000", "content": "Excellent solution,Congrats for strong finish and gold … well deserved.
", "votes": 2, "replies": [] }, { "id": 1584609, "author_name": "Kiran Chaudhary", "author_url": "", "post_date": "2021-11-16T16:16:20.823000", "content": "Congratulations and thanks for solutions…
", "votes": 2, "replies": [] }, { "id": 1584567, "author_name": "Trushant Kalyanpur", "author_url": "", "post_date": "2021-11-16T15:45:47.547000", "content": "Congratulations @ubamba98 and @abhishek. Great idea using the custom softmax - the output was of size max_len? So you could ensemble across models with same len?
", "votes": 2, "replies": [ { "id": 1584581, "author_name": "Udbhav Bamba", "author_url": "", "post_date": "2021-11-16T15:50:37.023000", "content": "No the output shape will remain as that of the input only difference is that now they are normalised between 0-1 so that we could now use the scores from postprocess_qa_predictions to chose the best predictions across models.
Thanks for sharing. Muril Model seems a nice part of your solution.Interesting.
", "votes": 2, "replies": [] }, { "id": 1584471, "author_name": "Ayush Singh", "author_url": "", "post_date": "2021-11-16T14:59:15.247000", "content": "Well deserved, excellent solution Abhishek and Ubdhav
", "votes": 2, "replies": [] }, { "id": 1624314, "author_name": "Durgesh Rao", "author_url": "", "post_date": "2021-12-20T18:20:13.917000", "content": "Excellent Work
\nkeep it up !!
Thanks for sharing @ubamba98 !😄
", "votes": 1, "replies": [] }, { "id": 1590696, "author_name": "MINGOOK KIM", "author_url": "", "post_date": "2021-11-21T15:08:05.623000", "content": "Congratulations !
\nThanks for sharing~
", "votes": 1, "replies": [] }, { "id": 1586157, "author_name": "Mintwater", "author_url": "", "post_date": "2021-11-17T20:33:43.207000", "content": "Nice work! Thanks for sharing: )
", "votes": 1, "replies": [] }, { "id": 1592219, "author_name": "Leon_and_hard", "author_url": "", "post_date": "2021-11-23T01:30:40.133000", "content": "Thanks for sharing~
", "votes": 0, "replies": [] }, { "id": 1830213, "author_name": "Rishu Raj", "author_url": "", "post_date": "2022-06-23T09:09:50.433000", "content": "Thanks for sharing
", "votes": 0, "replies": [] } ], "comment_count": 20, "messages": [ { "id": 1584368, "postDate": "2021-11-16T13:42:10.607Z", "content": "First of all thanks to Kaggle and the organizers for this competition.
\nThe competition data was not up to the mark and thus, there was no way you could build a good CV that correlates with the leaderboard. Unclean public training data and clean hidden private test data only meant we could use either different models and combine them together or use a single strong model like muril-large and train it on a variety of datasets shared by others. We did both.
\nThe first step was to divide the training data into 5 folds. After that, we trained our final models on subset of SQuAD v2, Google-translated SQuAD v2 shared by @tkm2261, all TyDi QA, hindi parts of MLQA, XQUAD and 5-10x oversampled Chaii dataset for each fold. We found that oversampling Chaii dataset helped us to improve CV and LB score. Please note that for CV calculation we used only the original validation folds.
\nAll Models were trained for 1-2 epochs with early stopping based on CV Jaccard score with max_length = 384 and doc_stride = 128. We did try other lengths such as 400, 512 and other strides like 192 but they didn’t perform very well with our cross-validation strategy.
\nSince different models have different tokenizations, simple average ensembling was not possible so tried using prediction scores to select best predictions from different models. This gave us a little improvement.
\nThe next thing we tried was to normalize the scores of all models to the same scale. We saw a little improvement using this too but we knew that this is not the best way to combine models.
\nThe vanilla version failed as different models had different scales for logits. So to overcome this I came up with a custom softmax which could normalize the logits across different context splits. Base code for this is attached below:
\ndef CustomSoftmax(examples, features, raw_predictions):\n all_start_logits, all_end_logits = raw_predictions\n\n # Build a map example to its corresponding features.\n example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n features_per_example = collections.defaultdict(list)\n for i, feature in enumerate(features):\n features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n\n for example_index, example in enumerate(tqdm(examples)):\n feature_indices = features_per_example[example_index]\n start_logits_sum = 0\n end_logits_sum = 0\n\n logCs = None\n logCe = None\n\n # Find maximum logit value across context\n for feature_index in feature_indices:\n if logCs is None:\n logCs = all_start_logits[feature_index].max()\n else:\n logCs = max(logCs, all_start_logits[feature_index].max())\n\n if logCe is None:\n logCe = all_end_logits[feature_index].max()\n else:\n logCe = max(logCe, all_end_logits[feature_index].max())\n\n # Subtract maximum value among all logits before applying exponentiation for numerical stability\n # And take sum across all context splits\n for feature_index in feature_indices:\n all_start_logits[feature_index] -= logCs\n all_end_logits[feature_index] -= logCe\n start_logits_sum += np.exp(all_start_logits[feature_index]).sum()\n end_logits_sum += np.exp(all_end_logits[feature_index]).sum()\n\n # Softmax\n for feature_index in feature_indices:\n all_start_logits[feature_index] = np.exp(all_start_logits[feature_index])/start_logits_sum\n all_end_logits[feature_index] = np.exp(all_end_logits[feature_index])/end_logits_sum\n\n return all_start_logits, all_end_logits\nThis was the final idea we had in the last two days and it boosted our public score from 0.809 to 0.821!
\nWe followed a similar preprocessing technique as used in public notebooks. Though we observed that removing a few of the symbols were hurting the performance so we decided to use a custom punctuations list ('!\"#$&\\'()*+,-/:;<=>?@[\\\\]^_`{|}~').
We also tried to solve the issue of tamil predictions as indicated by @jdoesv in https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/264831 with the help of stemming tamil words in the answers it helped to boost Muril performance from 0.802 -> 0.804 on the public LB.
\nIn the end, we didn’t add many rule-based post-processing as we thought this might overfit the public leaderboard.
\nOur best private submission would have scored 0.779 but we did not select it. It’s funny why we didn’t select that submission. It had a small bug. However, even with that bug, that submission was our best private submission :)
\nWe also had a single 5 fold muril-large model that would have placed us “in the money”.
\nOur best inference kernel can be found here: https://www.kaggle.com/abhishek/chaii-xlm-roberta-x-muril-x-rembert-score-based
\nYou can also try one of our best muril-large models on Hugging Face Hub: https://huggingface.co/abhishek/muril-large-chaii
\nWe will also publish our training code soon.
\nIt was a fun competition but we feel that it could have been better if the provided training data was better or at least similar to the hidden test dataset. We gave up many times.
\nWe also tried a meta-model to re-rank the answers generated by different models but it was quite late in the competition (on the final day) and we didn’t have enough time to experiment with it further.
\nP.S. @abhishek still feels that this competition was all about data collection from hundreds of participants.
\nP.S.S Training code can be found at: https://github.com/abhishekkrthakur/chaii-hindi-tamil-question-answering
", "rawMarkdown": "First of all thanks to Kaggle and the organizers for this competition. \n\n### Data Strategy\n\nThe competition data was not up to the mark and thus, there was no way you could build a good CV that correlates with the leaderboard. Unclean public training data and clean hidden private test data only meant we could use either different models and combine them together or use a single strong model like muril-large and train it on a variety of datasets shared by others. We did both.\n\nThe first step was to divide the training data into 5 folds. After that, we trained our final models on subset of SQuAD v2, Google-translated SQuAD v2 shared by @tkm2261, all TyDi QA, hindi parts of MLQA, XQUAD and 5-10x oversampled Chaii dataset for each fold. We found that oversampling Chaii dataset helped us to improve CV and LB score. Please note that for CV calculation we used only the original validation folds.\n\nAll Models were trained for 1-2 epochs with early stopping based on CV Jaccard score with max_length = 384 and doc_stride = 128. We did try other lengths such as 400, 512 and other strides like 192 but they didn’t perform very well with our cross-validation strategy.\n\n### Models\n\n- XLM Roberta Large (Public LB - 0.800)\n- Muril Large (Public LB - 0.802)\n- RemBERT (Public LB - 0.803)\n\n### Ensemble\n\nSince different models have different tokenizations, simple average ensembling was not possible so tried using prediction scores to select best predictions from different models. This gave us a little improvement.\n\nThe next thing we tried was to normalize the scores of all models to the same scale. We saw a little improvement using this too but we knew that this is not the best way to combine models.\n\nThe vanilla version failed as different models had different scales for logits. So to overcome this I came up with a custom softmax which could normalize the logits across different context splits. Base code for this is attached below:\n\n```python\ndef CustomSoftmax(examples, features, raw_predictions):\n all_start_logits, all_end_logits = raw_predictions\n\n # Build a map example to its corresponding features.\n example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n features_per_example = collections.defaultdict(list)\n for i, feature in enumerate(features):\n features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n\n for example_index, example in enumerate(tqdm(examples)):\n feature_indices = features_per_example[example_index]\n start_logits_sum = 0\n end_logits_sum = 0\n \n logCs = None\n logCe = None\n\n # Find maximum logit value across context\n for feature_index in feature_indices:\n if logCs is None:\n logCs = all_start_logits[feature_index].max()\n else:\n logCs = max(logCs, all_start_logits[feature_index].max())\n \n if logCe is None:\n logCe = all_end_logits[feature_index].max()\n else:\n logCe = max(logCe, all_end_logits[feature_index].max())\n \n # Subtract maximum value among all logits before applying exponentiation for numerical stability\n # And take sum across all context splits\n for feature_index in feature_indices:\n all_start_logits[feature_index] -= logCs\n all_end_logits[feature_index] -= logCe\n start_logits_sum += np.exp(all_start_logits[feature_index]).sum()\n end_logits_sum += np.exp(all_end_logits[feature_index]).sum()\n \n # Softmax\n for feature_index in feature_indices:\n all_start_logits[feature_index] = np.exp(all_start_logits[feature_index])/start_logits_sum\n all_end_logits[feature_index] = np.exp(all_end_logits[feature_index])/end_logits_sum\n \n return all_start_logits, all_end_logits\n```\n\nThis was the final idea we had in the last two days and it boosted our public score from 0.809 to 0.821!\n\n\n### Post-processing\n\nWe followed a similar preprocessing technique as used in public notebooks. Though we observed that removing a few of the symbols were hurting the performance so we decided to use a custom punctuations list (```'!\"#$&\\'()*+,-/:;<=>?@[\\\\]^_`{|}~'```). \n\nWe also tried to solve the issue of tamil predictions as indicated by @jdoesv in https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/264831 with the help of stemming tamil words in the answers it helped to boost Muril performance from 0.802 -> 0.804 on the public LB.\n\nIn the end, we didn’t add many rule-based post-processing as we thought this might overfit the public leaderboard. \n\nOur best private submission would have scored 0.779 but we did not select it. It’s funny why we didn’t select that submission. It had a small bug. However, even with that bug, that submission was our best private submission :)\n\nWe also had a single 5 fold muril-large model that would have placed us “in the money”.\n\nOur best inference kernel can be found here: https://www.kaggle.com/abhishek/chaii-xlm-roberta-x-muril-x-rembert-score-based\n\nYou can also try one of our best muril-large models on Hugging Face Hub: https://huggingface.co/abhishek/muril-large-chaii\n\nWe will also publish our training code soon.\n\n### Final thoughts\n\nIt was a fun competition but we feel that it could have been better if the provided training data was better or at least similar to the hidden test dataset. We gave up many times.\n\nWe also tried a meta-model to re-rank the answers generated by different models but it was quite late in the competition (on the final day) and we didn’t have enough time to experiment with it further. \n\nP.S. @abhishek still feels that this competition was all about data collection from hundreds of participants.\n\nP.S.S Training code can be found at: https://github.com/abhishekkrthakur/chaii-hindi-tamil-question-answering", "votes": 47 }, { "id": 1584826, "postDate": "2021-11-16T19:39:03.570Z", "content": "Congrats @ubamba98 and @abhishek, need to spend some time to understand your ensembling method, my mind is still dumb since yesterday 😂It was great to see your climb in the last days and defending the top position on private!
", "rawMarkdown": "Congrats @ubamba98 and @abhishek, need to spend some time to understand your ensembling method, my mind is still dumb since yesterday 😂It was great to see your climb in the last days and defending the top position on private!", "votes": 3 }, { "id": 1584377, "postDate": "2021-11-16T13:47:19.590Z", "content": "I had high hopes for re-ranking model. Too bad it didn't work out and we tried it towards the end and had no time to experiment :D
", "rawMarkdown": "I had high hopes for re-ranking model. Too bad it didn't work out and we tried it towards the end and had no time to experiment :D ", "votes": 3 }, { "id": 1593954, "postDate": "2021-11-24T12:21:31.910Z", "content": "Congretzz
", "rawMarkdown": "Congretzz ", "votes": 1 }, { "id": 1591244, "postDate": "2021-11-22T05:47:01.107Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 1585865, "postDate": "2021-11-17T16:01:41.730Z", "content": "Well Done brother nice work
", "rawMarkdown": "Well Done brother nice work", "votes": 1 }, { "id": 1585166, "postDate": "2021-11-17T06:12:01.077Z", "content": "Congrats. i'm envious of gold medal.
", "rawMarkdown": "Congrats. i'm envious of gold medal.", "votes": 2 }, { "id": 1585126, "postDate": "2021-11-17T05:30:21.223Z", "content": "you guys are amazing
", "rawMarkdown": "you guys are amazing", "votes": 2 }, { "id": 1584646, "postDate": "2021-11-16T16:46:56.493Z", "content": "This is really amazing Udbhav! Congratulations and truly well deserved.
", "rawMarkdown": "This is really amazing Udbhav! Congratulations and truly well deserved. ", "votes": 2 }, { "id": 1584635, "postDate": "2021-11-16T16:36:36.073Z", "content": "Excellent solution,Congrats for strong finish and gold … well deserved.
", "rawMarkdown": "Excellent solution,Congrats for strong finish and gold ... well deserved.", "votes": 2 }, { "id": 1584609, "postDate": "2021-11-16T16:16:20.823Z", "content": "Congratulations and thanks for solutions…
", "rawMarkdown": "Congratulations and thanks for solutions...", "votes": 2 }, { "id": 1584567, "postDate": "2021-11-16T15:45:47.547Z", "content": "Congratulations @ubamba98 and @abhishek. Great idea using the custom softmax - the output was of size max_len? So you could ensemble across models with same len?
", "rawMarkdown": "Congratulations @ubamba98 and @abhishek. Great idea using the custom softmax - the output was of size max_len? So you could ensemble across models with same len?", "votes": 2, "replies": [ { "id": 1584581, "postDate": "2021-11-16T15:50:37.023Z", "content": "No the output shape will remain as that of the input only difference is that now they are normalised between 0-1 so that we could now use the scores from postprocess_qa_predictions to chose the best predictions across models.
Thanks for sharing. Muril Model seems a nice part of your solution.Interesting.
", "rawMarkdown": "Thanks for sharing. Muril Model seems a nice part of your solution.Interesting.", "votes": 2 }, { "id": 1584471, "postDate": "2021-11-16T14:59:15.247Z", "content": "Well deserved, excellent solution Abhishek and Ubdhav
", "rawMarkdown": "Well deserved, excellent solution Abhishek and Ubdhav", "votes": 2 }, { "id": 1624314, "postDate": "2021-12-20T18:20:13.917Z", "content": "Excellent Work
\nkeep it up !!
Thanks for sharing @ubamba98 !😄
", "rawMarkdown": "Thanks for sharing @ubamba98 !😄", "votes": 1 }, { "id": 1590696, "postDate": "2021-11-21T15:08:05.623Z", "content": "Congratulations !
\nThanks for sharing~
", "rawMarkdown": "Congratulations !\n\nThanks for sharing~", "votes": 1 }, { "id": 1586157, "postDate": "2021-11-17T20:33:43.207Z", "content": "Nice work! Thanks for sharing: )
", "rawMarkdown": "Nice work! Thanks for sharing: )", "votes": 1 }, { "id": 1592219, "postDate": "2021-11-23T01:30:40.133Z", "content": "Thanks for sharing~
", "rawMarkdown": "Thanks for sharing~" }, { "id": 1830213, "postDate": "2022-06-23T09:09:50.433Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:23:05Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7250918, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/abhishek-udbhav-5th-place-solution" }