Thank you to Kaggle, the hosts, the discussion forums, and all the rivals!
\nI really enjoyed my first Kaggle thanks to all of you!
I got 20th place at the private LB (0.755). (although was 9th at the public LB (0.811)!)
\nMy solution was the ensemble of 9 XLM-RoBERTa-large-based models (from 5 different ways of model construction) and some hacky postprocessing for inference.
Because I haven't evaluated them in detail, each model construction is dirty and may contain many meaningless (or even harmful) tricks and differences. Note that I describe them simply as what I did and what I guess.
\nThe starting point is the famous 0.792 notebooks.
(I mimicked the format of 1st place Darek's solution.)
\nSimply used sklearn's model_selection.StratifiedKFold of 5-fold and y = data['language']
\nIt is used for early stopping and model (seed) selection.
Basically, it is a standard extraction QA BERT model (AutoModelForQuestionAnswering), i.e., encoding by a transformer and predicting the logits for (start_index, end_index) via a linear layer.
\nThe score of each phrase is calculated by start_logit + end_logit. (If both indicate CLS token's position, it represents 'no-answer.')
I tried some variants for enhancing the diversity of models.
\nwhere h_{*} indicate *-th layer's output at the token and h_last_of_CLS is the final output at CLS token.
\nI did it almost just for fun! I expected that they possibly could contribute to the ensemble.
\nSome external datasets are sometiems used or sometimes not.
\nFinetuning
\nAdditional pretraining
\nIn total, I used five settings for the ensemble.
\nThey may have different settings:
([ ] shows scores at the public LB)
\n(1) [0.774, 0.775]
\n(2) [0.778, 0.782]
\n(3) [0.769, 0.770]
\n(4) [0.763]
\n(5) [0.765, 0.774]
\nI expected that adding external data and regularizations (dropout, weight decay, early stopping, consistency regularization) may work sufficiently. So, I didn't use special data augmentation.
\nInstead, at each epoch, I sometimes simply used different 'stride' hyperparameters (e.g., +Uniform[-10, 10]) for constructing the training dataset of 'strode chunks'.
\nThis produces (a little) different examples at each epoch.
It uses tricks in the famous 0.792 notebooks.
\nIn addition, I originally used some tricks after analyzing errors (in CV evaluation) by my models.
\nI guessed that some preferences or tendencies could happen in annotation processes.
I observed some cases, where a model prediction and GT indicate the same numerical expression but the prediction is written in Hindi but GT is written in Arabic (and both expressions are in the text). This is an ambiguity of answers due to orthographical variants. Although I'm not familiar with Hindi, GT answers seem more frequently written in Arabic. Therefore, I performed a simple replacement (Hindi -> Arabic) by
\n{'०':\"0\", '१':\"1\", '२':\"2\", '३':\"3\", '४':\"4\", '५':\"5\", '६':\"6\", '७':\"7\", '८':\"8\", '९':\"9\"} if the model predicts Hindi number but the same Arabic number also exists in the text.
(Maybe the same as the tkm kh's processing)
\nI tried the same trick for Tamil but not effective.
\nIt seems that a phrase at earlier positions can be an answer a little more frequently than the others.
\nI added a simple bias for the score by - (2.0 * start_char_index / len(context).
BERT-based QA by huggingface implementation split long text into many subtext chunks (with some stride setup), each of which is shorter than max_seq_length of the model.
\nI guess that prediction on the positions near the boundaries of chunks might be unreliable because some important surrounding context could be missing by chunking.
\nTherefore, I added a simple bias for the score by
if [it is NOT the beginning of the whole text] and [start_index_in_chunk is earlier than 10]:\n bias = -3\nelif [it is NOT the end of the whole text] and [end_index_in_chunk is later than max_length-10]:\n bias -= 3\nelse:\n bias = 0\nEven if a phrase gets a penalty due to the bias, the phrase may occur in another chunk without penalty because stridden chunks have overlapped tokens.
\nThanks!
", "comments": [ { "id": 1590709, "author_name": "MINGOOK KIM", "author_url": "", "post_date": "2021-11-21T15:14:07.920000", "content": "Thanks for the good summary.
\nIt helped a lot.
", "votes": 1, "replies": [ { "id": 1591524, "author_name": "soskek", "author_url": "", "post_date": "2021-11-22T12:35:57.977000", "content": "Thank you! I feel so happy to hear that!
", "votes": 0, "replies": [] } ] }, { "id": 1587726, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-11-18T22:22:58.667000", "content": "Congratulations, that's an impressive result for a first Kaggle competition!
", "votes": 1, "replies": [ { "id": 1591529, "author_name": "soskek", "author_url": "", "post_date": "2021-11-22T12:42:10.323000", "content": "Thank you!!
\n(BTW, your posts in discussion helped me a lot! In addition, your higher rankings in pubLB made me keep motivated!)
I also tried SAM, but it was very unstable and I didn't have good intuition about what parameters to change. I'll keep experimenting though, because it does seem like it would make models generalize better.
\nWhat was your experience with SAM?
", "votes": 2, "replies": [ { "id": 1591515, "author_name": "soskek", "author_url": "", "post_date": "2021-11-22T12:31:01.023000", "content": "Thank you for asking!
\nI found that my logs of SAM exp were missing… I'm also still interested in SAM.
\nSo, I re-run a simple setting using https://huggingface.co/deepset/xlm-roberta-large-squad2 as the pretrained model and compared (1) baseline, (2) SAM from the beginning, (3) SAM only after 200 updates.
\n(hyperparameters-> max_seq_length=400, doc_stride=160, epochs=5, train_batch_size * gradient_accumulation_steps=32, AdamW, lr=1.5e-05, weight_decay=0.005, warmup_ratio=0.1, all used the same seed) (fp32 was used because my fp16+SAM seemed to be somehow broken.)
Jaccard comparison on the average of 5-fold CV with early stopping.
\n(1) Baseline: 0.678
\n(2) SAMfromBegin: 0.686
\n(3) SAMfrom200it: 0.677
\nIn the results, (2) SAM seems a little better performance, though less conclusive.
That was a little different from my feeling (= vague memory…) during the competition. It might depend on settings. Anyway, I believe that SAM is still promising in the next tasks!
", "votes": 1, "replies": [] } ] } ], "comment_count": 7, "messages": [ { "id": 1584321, "postDate": "2021-11-16T13:16:56.633Z", "content": "Thank you to Kaggle, the hosts, the discussion forums, and all the rivals!
\nI really enjoyed my first Kaggle thanks to all of you!
I got 20th place at the private LB (0.755). (although was 9th at the public LB (0.811)!)
\nMy solution was the ensemble of 9 XLM-RoBERTa-large-based models (from 5 different ways of model construction) and some hacky postprocessing for inference.
Because I haven't evaluated them in detail, each model construction is dirty and may contain many meaningless (or even harmful) tricks and differences. Note that I describe them simply as what I did and what I guess.
\nThe starting point is the famous 0.792 notebooks.
(I mimicked the format of 1st place Darek's solution.)
\nSimply used sklearn's model_selection.StratifiedKFold of 5-fold and y = data['language']
\nIt is used for early stopping and model (seed) selection.
Basically, it is a standard extraction QA BERT model (AutoModelForQuestionAnswering), i.e., encoding by a transformer and predicting the logits for (start_index, end_index) via a linear layer.
\nThe score of each phrase is calculated by start_logit + end_logit. (If both indicate CLS token's position, it represents 'no-answer.')
I tried some variants for enhancing the diversity of models.
\nwhere h_{*} indicate *-th layer's output at the token and h_last_of_CLS is the final output at CLS token.
\nI did it almost just for fun! I expected that they possibly could contribute to the ensemble.
\nSome external datasets are sometiems used or sometimes not.
\nFinetuning
\nAdditional pretraining
\nIn total, I used five settings for the ensemble.
\nThey may have different settings:
([ ] shows scores at the public LB)
\n(1) [0.774, 0.775]
\n(2) [0.778, 0.782]
\n(3) [0.769, 0.770]
\n(4) [0.763]
\n(5) [0.765, 0.774]
\nI expected that adding external data and regularizations (dropout, weight decay, early stopping, consistency regularization) may work sufficiently. So, I didn't use special data augmentation.
\nInstead, at each epoch, I sometimes simply used different 'stride' hyperparameters (e.g., +Uniform[-10, 10]) for constructing the training dataset of 'strode chunks'.
\nThis produces (a little) different examples at each epoch.
It uses tricks in the famous 0.792 notebooks.
\nIn addition, I originally used some tricks after analyzing errors (in CV evaluation) by my models.
\nI guessed that some preferences or tendencies could happen in annotation processes.
I observed some cases, where a model prediction and GT indicate the same numerical expression but the prediction is written in Hindi but GT is written in Arabic (and both expressions are in the text). This is an ambiguity of answers due to orthographical variants. Although I'm not familiar with Hindi, GT answers seem more frequently written in Arabic. Therefore, I performed a simple replacement (Hindi -> Arabic) by
\n{'०':\"0\", '१':\"1\", '२':\"2\", '३':\"3\", '४':\"4\", '५':\"5\", '६':\"6\", '७':\"7\", '८':\"8\", '९':\"9\"} if the model predicts Hindi number but the same Arabic number also exists in the text.
(Maybe the same as the tkm kh's processing)
\nI tried the same trick for Tamil but not effective.
\nIt seems that a phrase at earlier positions can be an answer a little more frequently than the others.
\nI added a simple bias for the score by - (2.0 * start_char_index / len(context).
BERT-based QA by huggingface implementation split long text into many subtext chunks (with some stride setup), each of which is shorter than max_seq_length of the model.
\nI guess that prediction on the positions near the boundaries of chunks might be unreliable because some important surrounding context could be missing by chunking.
\nTherefore, I added a simple bias for the score by
if [it is NOT the beginning of the whole text] and [start_index_in_chunk is earlier than 10]:\n bias = -3\nelif [it is NOT the end of the whole text] and [end_index_in_chunk is later than max_length-10]:\n bias -= 3\nelse:\n bias = 0\nEven if a phrase gets a penalty due to the bias, the phrase may occur in another chunk without penalty because stridden chunks have overlapped tokens.
\nThanks!
", "rawMarkdown": "Thank you to Kaggle, the hosts, the discussion forums, and all the rivals!\nI really enjoyed my first Kaggle thanks to all of you!\n\nI got 20th place at the private LB (0.755). (although was 9th at the public LB (0.811)!)\nMy solution was the ensemble of 9 XLM-RoBERTa-large-based models (from 5 different ways of model construction) and some hacky postprocessing for inference.\n\nBecause I haven't evaluated them in detail, each model construction is dirty and may contain many meaningless (or even harmful) tricks and differences. Note that I describe them simply as what I did and what I guess.\nThe starting point is the famous [0.792 notebook](https://www.kaggle.com/tkm2261/reproduction-of-0-792-notebook)s.\n\n(I mimicked the format of [1st place Darek's solution](https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/287923).)\n\n\n#### CV Setup\n\nSimply used sklearn's model_selection.StratifiedKFold of 5-fold and y = data['language']\nIt is used for early stopping and model (seed) selection.\n\n\n#### Architecture\n\nBasically, it is a standard extraction QA BERT model (AutoModelForQuestionAnswering), i.e., encoding by a transformer and predicting the logits for (start_index, end_index) via a linear layer.\nThe score of each phrase is calculated by start_logit + end_logit. (If both indicate CLS token's position, it represents 'no-answer.')\n\nI tried some variants for enhancing the diversity of models.\n- transformer -> linear (baseline)\n- transformer -> cat([h_last, h_19, h_15, h_11, h_last_of_CLS]) -> linear -> LeaklyReLU -> [k=3 depthwise conv -> linear] x2 with residual connection -> linear (variant A)\n- transformer -> cat([h_last, h_19, h_15, h_11, h_last_of_CLS]) -> linear -> [new BERTLayer] x 2 -> linear (variant B)\n\nwhere h_{*} indicate *-th layer's output at the token and h_last_of_CLS is the final output at CLS token.\n\nI did it almost just for fun! I expected that they possibly could contribute to the ensemble.\n\n\n#### Data Recipes\n\nSome external datasets are sometiems used or sometimes not.\n\nFinetuning\n- [XQuAD-hindi](https://www.kaggle.com/rhtsingh/mlqa-hindi-processed), [MLQA-hindi](https://www.kaggle.com/rhtsingh/mlqa-hindi-processed) (thanks, @rhtsingh)\n- [SQuAD-english](https://rajpurkar.github.io/SQuAD-explorer/)\n- [MLQA-translate-hindi](https://github.com/facebookresearch/MLQA#translate-train-and-translate-test-data) with some filtering like \"context length <= 2000\"\n- [xtune's training datasets for xquad](https://github.com/microsoft/unilm/tree/master/xtune)\n\n\nAdditional pretraining\n- [IndicCorp in Hindi and Tamil](https://indicnlp.ai4bharat.org/corpora/), Wikipedia in [Hindi](https://dumps.wikimedia.org/hiwiki/latest/) and [Tamil](https://dumps.wikimedia.org/tawiki/latest/)\n\n\n#### Fine-tuning\n\nIn total, I used five settings for the ensemble.\nThey may have different settings:\n- Model architectures.\n- Pretrained models. When additional pretraining is performed, the initialization is from XLM-RoBERTa-Large. In addition, two-stage finetuning is sometimes used.\n- Finetuning datasets. The chaii dataset + some externals.\n- Hyperparameters. Maybe, less important. epoch={1, 4}, lr=1.5e-5, weight decay={5e-3, 1e-2}, total_batch_size={8, 16}, max_seq_length={384, 400}, doc_stride={128, 135}\n\n([ ] shows scores at the public LB)\n\n(1) [0.774, 0.775]\n- Variant A-arch; using CNN\n- MLM pretraining with 30% corpus\n- MLQA, XQuAD\n\n(2) [0.778, 0.782]\n- Baseline Arch.\n- MLM pretraining with 30% corpus\n- MLQA, XQuAD, MLQA-translated, SQuAD\n\n(3) [0.769, 0.770]\n- Baseline arch.\n- MLM pretraining with only Wikipedia x 4 epochs\n- MLQA, XQuAD, SQuAD\n\n(4) [0.763]\n- Variant B-arch; using attention\n- MLM pretraining with 100% corpus\n- 1st stage: SQuAD + [consistency regularization](https://arxiv.org/abs/2106.08226) (repro. of xtune)\n- 2nd stage: MLQA, XQuAD, SQuAD\n\n(5) [0.765, 0.774]\n- Baseline arch.\n- XLM-RoBERTa\n- 1st stage: xtune's xquad setting 1\n- 2nd stage: xtune's xquad setting 2 without augumentation.\n- 3rd stage: MLQA, XQuAD\n- (BTW, [xtune](https://github.com/microsoft/unilm/tree/master/xtune)(+ finetuning) was very helpful for training a best single model (0.790))\n\n\n#### Data Augmentations\n\nI expected that adding external data and regularizations (dropout, weight decay, early stopping, consistency regularization) may work sufficiently. So, I didn't use special data augmentation.\nInstead, at each epoch, I sometimes simply used different 'stride' hyperparameters (e.g., +Uniform[-10, 10]) for constructing the training dataset of 'strode chunks'.\nThis produces (a little) different examples at each epoch.\n\n\n\n#### Post-processing\n\nIt uses tricks in the famous 0.792 notebooks.\nIn addition, I originally used some tricks after analyzing errors (in CV evaluation) by my models.\nI guessed that some preferences or tendencies could happen in annotation processes.\n\n##### Arabic Numerical Expressions\n\nI observed some cases, where a model prediction and GT indicate the same numerical expression but the prediction is written in Hindi but GT is written in Arabic (and both expressions are in the text). This is an ambiguity of answers due to orthographical variants. Although I'm not familiar with Hindi, GT answers seem more frequently written in Arabic. Therefore, I performed a simple replacement (Hindi -> Arabic) by\n`{'०':\"0\", '१':\"1\", '२':\"2\", '३':\"3\", '४':\"4\", '५':\"5\", '६':\"6\", '७':\"7\", '८':\"8\", '९':\"9\"}` if the model predicts Hindi number but the same Arabic number also exists in the text.\n\n(Maybe the same as the [tkm kh](https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/287917)'s processing)\n\nI tried the same trick for Tamil but not effective.\n\n\n##### Earlier Phrases\n\nIt seems that a phrase at earlier positions can be an answer a little more frequently than the others.\nI added a simple bias for the score by `- (2.0 * start_char_index / len(context)`.\n\n\n##### Stride Boundaries\n\nBERT-based QA by huggingface implementation split long text into many subtext chunks (with some stride setup), each of which is shorter than max_seq_length of the model.\nI guess that prediction on the positions near the boundaries of chunks might be unreliable because some important surrounding context could be missing by chunking.\nTherefore, I added a simple bias for the score by\n```\nif [it is NOT the beginning of the whole text] and [start_index_in_chunk is earlier than 10]:\n bias = -3\nelif [it is NOT the end of the whole text] and [end_index_in_chunk is later than max_length-10]:\n\tbias -= 3\nelse:\n\tbias = 0\n```\n\nEven if a phrase gets a penalty due to the bias, the phrase may occur in another chunk without penalty because stridden chunks have overlapped tokens.\n\n\n##### Things that Didn't Work (for My Ensembles)\n\n- I thought mT5s could be the best model. But, I failed to get good performances with them; even reproducing mT5 papers' results in SQuAD.\n- Label smoothing. It could help but almost seem meaningless. I also tried something similar to Jaccard-based Soft Labels, but less helpful.\n- MURIL Large. Many winners of chaii seem to use MURIL too. But, my MURIL was somehow not good.\n- info-xlm-large. It was very good as a single model. But, finally not helpful in my ensemble...\n- Tagging the questions by dataset-specific special tokens for discrimination, as used in NMT with [backtranslation](https://aclanthology.org/2020.acl-main.532.pdf) or for [multilingual modeling](https://aclanthology.org/2021.findings-acl.264.pdf). It seemed to alleviate the performance change due to dataset selection but seems not very helpful when selecting appropriate external datasets only.\n- I tried to use SQuAD-translated-tamil and XQA-tamil but were not helpful in my final submission.\n- In NaturalQA, [it](https://arxiv.org/abs/1901.08634) is reported that using CLS-token scores for calibrations.\n- [SAM-based optimization](https://arxiv.org/abs/2110.08529).\n- [xtune](https://github.com/microsoft/unilm/tree/master/xtune) was very helpful for training a best single model. But, often less helpful in my ensembles.\n\n\nThanks!", "votes": 8 }, { "id": 1590709, "postDate": "2021-11-21T15:14:07.920Z", "content": "Thanks for the good summary.
\nIt helped a lot.
", "rawMarkdown": "Thanks for the good summary.\n\nIt helped a lot.", "votes": 1, "replies": [ { "id": 1591524, "postDate": "2021-11-22T12:35:57.977Z", "content": "Thank you! I feel so happy to hear that!
", "rawMarkdown": "Thank you! I feel so happy to hear that!" } ] }, { "id": 1587726, "postDate": "2021-11-18T22:22:58.667Z", "content": "Congratulations, that's an impressive result for a first Kaggle competition!
", "rawMarkdown": "Congratulations, that's an impressive result for a first Kaggle competition!", "votes": 1, "replies": [ { "id": 1591529, "postDate": "2021-11-22T12:42:10.323Z", "content": "Thank you!!
\n(BTW, your posts in discussion helped me a lot! In addition, your higher rankings in pubLB made me keep motivated!)
I also tried SAM, but it was very unstable and I didn't have good intuition about what parameters to change. I'll keep experimenting though, because it does seem like it would make models generalize better.
\nWhat was your experience with SAM?
", "rawMarkdown": "I also tried SAM, but it was very unstable and I didn't have good intuition about what parameters to change. I'll keep experimenting though, because it does seem like it would make models generalize better.\n\nWhat was your experience with SAM?", "votes": 2, "replies": [ { "id": 1591515, "postDate": "2021-11-22T12:31:01.023Z", "content": "Thank you for asking!
\nI found that my logs of SAM exp were missing… I'm also still interested in SAM.
\nSo, I re-run a simple setting using https://huggingface.co/deepset/xlm-roberta-large-squad2 as the pretrained model and compared (1) baseline, (2) SAM from the beginning, (3) SAM only after 200 updates.
\n(hyperparameters-> max_seq_length=400, doc_stride=160, epochs=5, train_batch_size * gradient_accumulation_steps=32, AdamW, lr=1.5e-05, weight_decay=0.005, warmup_ratio=0.1, all used the same seed) (fp32 was used because my fp16+SAM seemed to be somehow broken.)
Jaccard comparison on the average of 5-fold CV with early stopping.
\n(1) Baseline: 0.678
\n(2) SAMfromBegin: 0.686
\n(3) SAMfrom200it: 0.677
\nIn the results, (2) SAM seems a little better performance, though less conclusive.
That was a little different from my feeling (= vague memory…) during the competition. It might depend on settings. Anyway, I believe that SAM is still promising in the next tasks!
", "rawMarkdown": "That was a little different from my feeling (= vague memory...) during the competition. It might depend on settings. Anyway, I believe that SAM is still promising in the next tasks!", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:22:23Z", "competition_id": 30060, "competition_slug": "chaii-hindi-and-tamil-question-answering", "team_id": 7504622, "solution_url": "/competitions/chaii-hindi-and-tamil-question-answering/writeups/soskek-20th-place-solution-xlm-rs-xtune-hacky-post" }