[UPDATE] source code is now available at https://github.com/Zacchaeus14/chaii-4th-place-solution
\nThank Kaggle and Google Research for hosting this fantastic competition. The CV and LB are really hard and I cannot believe we moved into the money zone. Thank my teammate @zzy. Here I share our solution. Please feel free to leave comments!
There are 5 models involved in the final submissions:
\nmodel 1. Xlm-Roberta-large 5 folds
\nmodel 2. Rembert 5 folds
\nmodel 3. Xlm-Roberta-large 3 seeds
\nmodel 4. Muril-large 4 seeds
\nmodel 4'. Muril-large 5 folds
model 5. Infoxlm 5 folds
\nOur submission A is an ensemble of 125 (public 0.817 private 0.767) and B 2345 (public 0.812 private 0.777). We find the best private submission is 24'5 (public 0.812 private 0.784).
\nI try pre-training on hi/ta wiki with vanilla MLM, hoping these data will cover the hidden test set. Doesn't work. I find that Infoxlm / Rembert / XLM-Roberta benefits from finetuning on the squad 2.0 dataset for 1 -3 epochs. Then, I translate the dataset into Tamil with Google API, but the translated dataset does not bring significant improvements. After that, I train the models on chaii + mlqa + xquad dataset with the latter two downsampled for 3 epochs. The validation Jaccard score was quite unstable, so I adapted \"non-zero Jaccard percentage\" as an early stopping criterion\".
\nXlm-Roberta and Infoxlm share the same tokenizer, while Rembert and Muril have their own. I ensemble models with different tokenizers by mapping the token-level logits back to characters and then average with equal weights. After mapping and blending, I replace consecutive same numbers with -1e5 and finally do the beam search.
\nAs the training data is inconsistent. I make various post-processing heuristics to mitigate the bias. Some major methods:
\nThank NYU Shanghai faculty for granting me the GPU resources.
\nHere is the inference kernel: https://www.kaggle.com/zacchaeus/chaii-infer-blend-postpro-4models. I will publish my github repository after cleaning up my code.
", "comments": [ { "id": 1583727, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-11-16T02:59:59.997000", "content": "congrats @zacchaeus and team on 4th place and thanks for sharing solution
", "votes": 6, "replies": [] }, { "id": 1584821, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-11-16T19:30:47.790000", "content": "Congrats @zacchaeus and @zzy990106 ! I was great competing with you, glad you finished in the money zone! I had the same experience with Google Translate 😂
", "votes": 1, "replies": [ { "id": 1584862, "author_name": "Zacchaeus", "author_url": "", "post_date": "2021-11-16T20:17:06.387000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 1584466, "author_name": "Ayush Singh", "author_url": "", "post_date": "2021-11-16T14:56:57.240000", "content": "Great work, congratulations !!!
\nWell deserved
Congratulations, Nice work :)
", "votes": 1, "replies": [] }, { "id": 1584405, "author_name": "Mathurin Ache", "author_url": "", "post_date": "2021-11-16T14:04:53.460000", "content": "Well done @zacchaeus , it's well documented. according to you, PostProcessing have played a good part of your ranking ?
", "votes": 1, "replies": [ { "id": 1584416, "author_name": "Zacchaeus", "author_url": "", "post_date": "2021-11-16T14:12:48.233000", "content": "@mathurinache you are right, pp boosted much :)
", "votes": 0, "replies": [] } ] }, { "id": 1583845, "author_name": "joejeo1", "author_url": "", "post_date": "2021-11-16T05:22:44.553000", "content": "Great work, congratulations !!!
", "votes": 1, "replies": [] }, { "id": 1583777, "author_name": "Mintwater", "author_url": "", "post_date": "2021-11-16T04:00:54.233000", "content": "Congrats, nice work!
", "votes": 2, "replies": [] }, { "id": 1583669, "author_name": "van73", "author_url": "", "post_date": "2021-11-16T01:50:05.330000", "content": "Congratulations! Thanks for sharing. I have some questions about your solution:
\nAfter that, I train the models on chaii + mlqa + xquad dataset with the latter two undersampled for 3 epochs.by mapping the token-level logits back to characters and then average with equal weights.@vanle73 thanks for the comment.
\n-1. I made a custom sampler. In each epoch, I use the whole chaii data and sample part of the external data.
class ChaiiRandomSampler(Sampler):\n def __init__(self, data_source, downsample=1.0):\n self.data_source = data_source\n self.downsample = downsample\n self.chaii_indices = []\n self.ext_indices = []\n for i, item in enumerate(self.data_source):\n if item['src'] == 0:\n self.chaii_indices.append(i)\n else:\n self.ext_indices.append(i)\n self.chaii_indices = torch.tensor(self.chaii_indices, dtype=torch.long)\n self.ext_indices = torch.tensor(self.ext_indices, dtype=torch.long)\n self.n_chaii = len(self.chaii_indices)\n self.n_ext = int(len(self.ext_indices)*self.downsample)\n\n def __iter__(self):\n sampled_ext_indices_indices = torch.randperm(len(self.ext_indices))[:self.n_ext].long()\n# print(type(sampled_ext_indices_indices))\n sampled_ext_indices = self.ext_indices[sampled_ext_indices_indices]\n indices = torch.cat([self.chaii_indices, sampled_ext_indices])\n indices_indices = torch.randperm(self.n_chaii+self.n_ext).long()\n return iter(indices[indices_indices].tolist())\n\n def __len__(self):\n return len(self.data_source)\n-2. The mapping brings a best single 0.796 to 0.817 on public.
", "votes": 3, "replies": [] }, { "id": 1583723, "author_name": "van73", "author_url": "", "post_date": "2021-11-16T02:58:18.393000", "content": "@zacchaeus Nice job! I have tried character mapping in the another competition CCKS2021, but I doesn't work. Thanks for your reply!
", "votes": 2, "replies": [] } ] }, { "id": 1583656, "author_name": "DrHB", "author_url": "", "post_date": "2021-11-16T01:37:30.423000", "content": "Wow congrats amazing finish =)
", "votes": 2, "replies": [] }, { "id": 1583652, "author_name": "Luca Massaron", "author_url": "", "post_date": "2021-11-16T01:33:49.970000", "content": "By the way, could you elaborate more on \"non-zero Jaccard percentage\" as an early stopping criterion?
", "votes": 2, "replies": [ { "id": 1583684, "author_name": "Zacchaeus", "author_url": "", "post_date": "2021-11-16T02:02:48", "content": "@lucamassaron Becuase the annotation is noisy, I refer to totally wrong (jaccard = 0) answers as a metric.
\n$metric = 1-\\frac{# total wrong}{# samples}$
Very interesting solution, compliments!
", "votes": 2, "replies": [] }, { "id": 1583582, "author_name": "atfujita", "author_url": "", "post_date": "2021-11-16T00:33:38.467000", "content": "Congratulations!
", "votes": 2, "replies": [] }, { "id": 1583581, "author_name": "maze508", "author_url": "", "post_date": "2021-11-16T00:33:31.763000", "content": "Congrats 😊🎉
", "votes": 2, "replies": [] }, { "id": 1583558, "author_name": "Kiran Chaudhary", "author_url": "", "post_date": "2021-11-16T00:10:21.167000", "content": "Congrats for gold and thanks for sharing your approach.
", "votes": 2, "replies": [] }, { "id": 1583555, "author_name": "Raghawendra Singh", "author_url": "", "post_date": "2021-11-16T00:08:08.840000", "content": "congratulations!!!!
", "votes": 2, "replies": [] }, { "id": 1590697, "author_name": "MINGOOK KIM", "author_url": "", "post_date": "2021-11-21T15:08:23.537000", "content": "Congratulations on your achievements.
\nCongratulations~
", "votes": 0, "replies": [] }, { "id": 1585167, "author_name": "adldotori", "author_url": "", "post_date": "2021-11-17T06:12:11.517000", "content": "Congrats. i'm envious of gold medal.
", "votes": 0, "replies": [] }, { "id": 1591226, "author_name": "", "author_url": "", "post_date": "2021-11-22T05:28:38.150000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 23, "messages": [ { "id": 1583553, "postDate": "2021-11-16T00:05:44.743Z", "content": "[UPDATE] source code is now available at https://github.com/Zacchaeus14/chaii-4th-place-solution
\nThank Kaggle and Google Research for hosting this fantastic competition. The CV and LB are really hard and I cannot believe we moved into the money zone. Thank my teammate @zzy. Here I share our solution. Please feel free to leave comments!
There are 5 models involved in the final submissions:
\nmodel 1. Xlm-Roberta-large 5 folds
\nmodel 2. Rembert 5 folds
\nmodel 3. Xlm-Roberta-large 3 seeds
\nmodel 4. Muril-large 4 seeds
\nmodel 4'. Muril-large 5 folds
model 5. Infoxlm 5 folds
\nOur submission A is an ensemble of 125 (public 0.817 private 0.767) and B 2345 (public 0.812 private 0.777). We find the best private submission is 24'5 (public 0.812 private 0.784).
\nI try pre-training on hi/ta wiki with vanilla MLM, hoping these data will cover the hidden test set. Doesn't work. I find that Infoxlm / Rembert / XLM-Roberta benefits from finetuning on the squad 2.0 dataset for 1 -3 epochs. Then, I translate the dataset into Tamil with Google API, but the translated dataset does not bring significant improvements. After that, I train the models on chaii + mlqa + xquad dataset with the latter two downsampled for 3 epochs. The validation Jaccard score was quite unstable, so I adapted \"non-zero Jaccard percentage\" as an early stopping criterion\".
\nXlm-Roberta and Infoxlm share the same tokenizer, while Rembert and Muril have their own. I ensemble models with different tokenizers by mapping the token-level logits back to characters and then average with equal weights. After mapping and blending, I replace consecutive same numbers with -1e5 and finally do the beam search.
\nAs the training data is inconsistent. I make various post-processing heuristics to mitigate the bias. Some major methods:
\nThank NYU Shanghai faculty for granting me the GPU resources.
\nHere is the inference kernel: https://www.kaggle.com/zacchaeus/chaii-infer-blend-postpro-4models. I will publish my github repository after cleaning up my code.
", "rawMarkdown": "[UPDATE] source code is now available at https://github.com/Zacchaeus14/chaii-4th-place-solution\nThank Kaggle and Google Research for hosting this fantastic competition. The CV and LB are really hard and I cannot believe we moved into the money zone. Thank my teammate @zzy. Here I share our solution. Please feel free to leave comments!\n# TLDR:\n1. Ensembling models with different tokenizers\n2. Finetuning on squad\n3. Post-processing\n# Models:\nThere are 5 models involved in the final submissions:\n## Zacchaeus:\nmodel 1. Xlm-Roberta-large 5 folds\nmodel 2. Rembert 5 folds\nmodel 3. Xlm-Roberta-large 3 seeds\nmodel 4. Muril-large 4 seeds\nmodel 4'. Muril-large 5 folds\n## zzy:\nmodel 5. Infoxlm 5 folds\n\n**Our submission A is an ensemble of 125 (public 0.817 private 0.767) and B 2345 (public 0.812 private 0.777). We find the best private submission is 24'5 (public 0.812 private 0.784).**\n\n# Training:\nI try pre-training on hi/ta wiki with vanilla MLM, hoping these data will cover the hidden test set. Doesn't work. I find that Infoxlm / Rembert / XLM-Roberta benefits from finetuning on the squad 2.0 dataset for 1 -3 epochs. Then, I translate the dataset into Tamil with Google API, but the translated dataset does not bring significant improvements. After that, I train the models on chaii + mlqa + xquad dataset with the latter two downsampled for 3 epochs. The validation Jaccard score was quite unstable, so I adapted \"non-zero Jaccard percentage\" as an early stopping criterion\".\n\n# Ensembling:\nXlm-Roberta and Infoxlm share the same tokenizer, while Rembert and Muril have their own. I ensemble models with different tokenizers by mapping the token-level logits back to characters and then average with equal weights. After mapping and blending, I replace consecutive same numbers with -1e5 and finally do the beam search.\n\n# Post-processing:\nAs the training data is inconsistent. I make various post-processing heuristics to mitigate the bias. Some major methods:\n1. Recursively strip left and right punctuations.\n2. Add back some punctuations if necessary (inspired by @nbroad 's notebook).\n3. Extract years from mm/yyyy/dd answers using regular expression if the question asks for a year specifically.\nAs a non-native speaker, going through all pieces of bad cases and copying/pasting in Google Translate can take forever.\n\n# Other things interesting but not working:\n1. Second-stage reranking model. Thanks @zzy for attempting on this. \n2. SWA.\n3. KL loss (mentioned in the VAULT paper).\n4. torchscript and ONNX\n\n# Acknowledgement\nThank NYU Shanghai faculty for granting me the GPU resources.\n\nHere is the inference kernel: https://www.kaggle.com/zacchaeus/chaii-infer-blend-postpro-4models. I will publish my github repository after cleaning up my code.\n\n\n", "votes": 33 }, { "id": 1583727, "postDate": "2021-11-16T02:59:59.997Z", "content": "congrats @zacchaeus and team on 4th place and thanks for sharing solution
", "rawMarkdown": "congrats @zacchaeus and team on 4th place and thanks for sharing solution", "votes": 6 }, { "id": 1584821, "postDate": "2021-11-16T19:30:47.790Z", "content": "Congrats @zacchaeus and @zzy990106 ! I was great competing with you, glad you finished in the money zone! I had the same experience with Google Translate 😂
", "rawMarkdown": "Congrats @zacchaeus and @zzy990106 ! I was great competing with you, glad you finished in the money zone! I had the same experience with Google Translate 😂", "votes": 1, "replies": [ { "id": 1584862, "postDate": "2021-11-16T20:17:06.387Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 1584466, "postDate": "2021-11-16T14:56:57.240Z", "content": "Great work, congratulations !!!
\nWell deserved
Congratulations, Nice work :)
", "rawMarkdown": "Congratulations, Nice work :)", "votes": 1 }, { "id": 1584405, "postDate": "2021-11-16T14:04:53.460Z", "content": "Well done @zacchaeus , it's well documented. according to you, PostProcessing have played a good part of your ranking ?
", "rawMarkdown": "Well done @zacchaeus , it's well documented. according to you, PostProcessing have played a good part of your ranking ?", "votes": 1, "replies": [ { "id": 1584416, "postDate": "2021-11-16T14:12:48.233Z", "content": "@mathurinache you are right, pp boosted much :)
", "rawMarkdown": "@mathurinache you are right, pp boosted much :)" } ] }, { "id": 1583845, "postDate": "2021-11-16T05:22:44.553Z", "content": "Great work, congratulations !!!
", "rawMarkdown": "Great work, congratulations !!!", "votes": 1 }, { "id": 1583777, "postDate": "2021-11-16T04:00:54.233Z", "content": "Congrats, nice work!
", "rawMarkdown": "Congrats, nice work!", "votes": 2 }, { "id": 1583669, "postDate": "2021-11-16T01:50:05.330Z", "content": "Congratulations! Thanks for sharing. I have some questions about your solution:
\nAfter that, I train the models on chaii + mlqa + xquad dataset with the latter two undersampled for 3 epochs.by mapping the token-level logits back to characters and then average with equal weights.@vanle73 thanks for the comment.
\n-1. I made a custom sampler. In each epoch, I use the whole chaii data and sample part of the external data.
class ChaiiRandomSampler(Sampler):\n def __init__(self, data_source, downsample=1.0):\n self.data_source = data_source\n self.downsample = downsample\n self.chaii_indices = []\n self.ext_indices = []\n for i, item in enumerate(self.data_source):\n if item['src'] == 0:\n self.chaii_indices.append(i)\n else:\n self.ext_indices.append(i)\n self.chaii_indices = torch.tensor(self.chaii_indices, dtype=torch.long)\n self.ext_indices = torch.tensor(self.ext_indices, dtype=torch.long)\n self.n_chaii = len(self.chaii_indices)\n self.n_ext = int(len(self.ext_indices)*self.downsample)\n\n def __iter__(self):\n sampled_ext_indices_indices = torch.randperm(len(self.ext_indices))[:self.n_ext].long()\n# print(type(sampled_ext_indices_indices))\n sampled_ext_indices = self.ext_indices[sampled_ext_indices_indices]\n indices = torch.cat([self.chaii_indices, sampled_ext_indices])\n indices_indices = torch.randperm(self.n_chaii+self.n_ext).long()\n return iter(indices[indices_indices].tolist())\n\n def __len__(self):\n return len(self.data_source)\n-2. The mapping brings a best single 0.796 to 0.817 on public.
", "rawMarkdown": "@vanle73 thanks for the comment.\n-1. I made a custom sampler. In each epoch, I use the whole chaii data and sample part of the external data.\n```\nclass ChaiiRandomSampler(Sampler):\n def __init__(self, data_source, downsample=1.0):\n self.data_source = data_source\n self.downsample = downsample\n self.chaii_indices = []\n self.ext_indices = []\n for i, item in enumerate(self.data_source):\n if item['src'] == 0:\n self.chaii_indices.append(i)\n else:\n self.ext_indices.append(i)\n self.chaii_indices = torch.tensor(self.chaii_indices, dtype=torch.long)\n self.ext_indices = torch.tensor(self.ext_indices, dtype=torch.long)\n self.n_chaii = len(self.chaii_indices)\n self.n_ext = int(len(self.ext_indices)*self.downsample)\n\n def __iter__(self):\n sampled_ext_indices_indices = torch.randperm(len(self.ext_indices))[:self.n_ext].long()\n# print(type(sampled_ext_indices_indices))\n sampled_ext_indices = self.ext_indices[sampled_ext_indices_indices]\n indices = torch.cat([self.chaii_indices, sampled_ext_indices])\n indices_indices = torch.randperm(self.n_chaii+self.n_ext).long()\n return iter(indices[indices_indices].tolist())\n\n def __len__(self):\n return len(self.data_source)\n```\n-2. The mapping brings a best single 0.796 to 0.817 on public.", "votes": 3 }, { "id": 1583723, "postDate": "2021-11-16T02:58:18.393Z", "content": "@zacchaeus Nice job! I have tried character mapping in the another competition CCKS2021, but I doesn't work. Thanks for your reply!
", "rawMarkdown": "@zacchaeus Nice job! I have tried character mapping in the another competition CCKS2021, but I doesn't work. Thanks for your reply!", "votes": 2 } ] }, { "id": 1583656, "postDate": "2021-11-16T01:37:30.423Z", "content": "Wow congrats amazing finish =)
", "rawMarkdown": "Wow congrats amazing finish =)", "votes": 2 }, { "id": 1583652, "postDate": "2021-11-16T01:33:49.970Z", "content": "By the way, could you elaborate more on \"non-zero Jaccard percentage\" as an early stopping criterion?
", "rawMarkdown": "By the way, could you elaborate more on \"non-zero Jaccard percentage\" as an early stopping criterion?", "votes": 2, "replies": [ { "id": 1583684, "postDate": "2021-11-16T02:02:48Z", "content": "@lucamassaron Becuase the annotation is noisy, I refer to totally wrong (jaccard = 0) answers as a metric.
\n$metric = 1-\\frac{# total wrong}{# samples}$
Very interesting solution, compliments!
", "rawMarkdown": "Very interesting solution, compliments!", "votes": 2 }, { "id": 1583582, "postDate": "2021-11-16T00:33:38.467Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 2 }, { "id": 1583581, "postDate": "2021-11-16T00:33:31.763Z", "content": "Congrats 😊🎉
", "rawMarkdown": "Congrats 😊🎉", "votes": 2 }, { "id": 1583558, "postDate": "2021-11-16T00:10:21.167Z", "content": "Congrats for gold and thanks for sharing your approach.
", "rawMarkdown": "Congrats for gold and thanks for sharing your approach.", "votes": 2 }, { "id": 1583555, "postDate": "2021-11-16T00:08:08.840Z", "content": "congratulations!!!!
", "rawMarkdown": "congratulations!!!!", "votes": 2 }, { "id": 1590697, "postDate": "2021-11-21T15:08:23.537Z", "content": "Congratulations on your achievements.
\nCongratulations~
", "rawMarkdown": "Congratulations on your achievements.\n\nCongratulations~" }, { "id": 1585167, "postDate": "2021-11-17T06:12:11.517Z", "content": "Congrats. i'm envious of gold medal.
", "rawMarkdown": "Congrats. i'm envious of gold medal." }, { "id": 1591226, "postDate": "2021-11-22T05:28:38.150Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:22:52Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7253814, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/zacchaeus-zzy-4th-place-solution" }