# 20th Place Solution; XLM-Rs, xtune, hacky postprocessing

Thank you to Kaggle, the hosts, the discussion forums, and all the rivals!
I really enjoyed my first Kaggle thanks to all of you!
I got 20th place at the private LB (0.755). (although was 9th at the public LB (0.811)!)
My solution was the ensemble of 9 XLM-RoBERTa-large-based models (from 5 different ways of model construction) and some hacky postprocessing for inference.
Because I haven't evaluated them in detail, each model construction is dirty and may contain many meaningless (or even harmful) tricks and differences. Note that I describe them simply as what I did and what I guess.
The starting point is the famous
[0.792 notebook](https://www.kaggle.com/tkm2261/reproduction-of-0-792-notebook)
s.
(I mimicked the format of
[1st place Darek's solution](https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/287923)
.)
CV Setup
Simply used sklearn's model_selection.StratifiedKFold of 5-fold and y = data['language']
It is used for early stopping and model (seed) selection.
Architecture
Basically, it is a standard extraction QA BERT model (AutoModelForQuestionAnswering), i.e., encoding by a transformer and predicting the logits for (start_index, end_index) via a linear layer.
The score of each phrase is calculated by start_logit + end_logit. (If both indicate CLS token's position, it represents 'no-answer.')
I tried some variants for enhancing the diversity of models.
transformer -> linear (baseline)
transformer -> cat([h_last, h_19, h_15, h_11, h_last_of_CLS]) -> linear -> LeaklyReLU -> [k=3 depthwise conv -> linear] x2 with residual connection -> linear (variant A)
transformer -> cat([h_last, h_19, h_15, h_11, h_last_of_CLS]) -> linear -> [new BERTLayer] x 2 -> linear (variant B)
where h_{*} indicate *-th layer's output at the token and h_last_of_CLS is the final output at CLS token.
I did it almost just for fun! I expected that they possibly could contribute to the ensemble.
Data Recipes
Some external datasets are sometiems used or sometimes not.
Finetuning
[XQuAD-hindi](https://www.kaggle.com/rhtsingh/mlqa-hindi-processed)
,
[MLQA-hindi](https://www.kaggle.com/rhtsingh/mlqa-hindi-processed)
(thanks,
[@rhtsingh](https://www.kaggle.com/rhtsingh)
)
[SQuAD-english](https://rajpurkar.github.io/SQuAD-explorer/)
[MLQA-translate-hindi](https://github.com/facebookresearch/MLQA#translate-train-and-translate-test-data)
with some filtering like "context length <= 2000"
[xtune's training datasets for xquad](https://github.com/microsoft/unilm/tree/master/xtune)
Additional pretraining
[IndicCorp in Hindi and Tamil](https://indicnlp.ai4bharat.org/corpora/)
, Wikipedia in
[Hindi](https://dumps.wikimedia.org/hiwiki/latest/)
and
[Tamil](https://dumps.wikimedia.org/tawiki/latest/)
Fine-tuning
In total, I used five settings for the ensemble.
They may have different settings:
Model architectures.
Pretrained models. When additional pretraining is performed, the initialization is from XLM-RoBERTa-Large. In addition, two-stage finetuning is sometimes used.
Finetuning datasets. The chaii dataset + some externals.
Hyperparameters. Maybe, less important. epoch={1, 4}, lr=1.5e-5, weight decay={5e-3, 1e-2}, total_batch_size={8, 16}, max_seq_length={384, 400}, doc_stride={128, 135}
([ ] shows scores at the public LB)
(1) [0.774, 0.775]
Variant A-arch; using CNN
MLM pretraining with 30% corpus
MLQA, XQuAD
(2) [0.778, 0.782]
Baseline Arch.
MLM pretraining with 30% corpus
MLQA, XQuAD, MLQA-translated, SQuAD
(3) [0.769, 0.770]
Baseline arch.
MLM pretraining with only Wikipedia x 4 epochs
MLQA, XQuAD, SQuAD
(4) [0.763]
Variant B-arch; using attention
MLM pretraining with 100% corpus
1st stage: SQuAD +
[consistency regularization](https://arxiv.org/abs/2106.08226)
(repro. of xtune)
2nd stage: MLQA, XQuAD, SQuAD
(5) [0.765, 0.774]
Baseline arch.
XLM-RoBERTa
1st stage: xtune's xquad setting 1
2nd stage: xtune's xquad setting 2 without augumentation.
3rd stage: MLQA, XQuAD
(BTW,
[xtune](https://github.com/microsoft/unilm/tree/master/xtune)
(+ finetuning) was very helpful for training a best single model (0.790))
Data Augmentations
I expected that adding external data and regularizations (dropout, weight decay, early stopping, consistency regularization) may work sufficiently. So, I didn't use special data augmentation.
Instead, at each epoch, I sometimes simply used different 'stride' hyperparameters (e.g., +Uniform[-10, 10]) for constructing the training dataset of 'strode chunks'.
This produces (a little) different examples at each epoch.
Post-processing
It uses tricks in the famous 0.792 notebooks.
In addition, I originally used some tricks after analyzing errors (in CV evaluation) by my models.
I guessed that some preferences or tendencies could happen in annotation processes.
Arabic Numerical Expressions
I observed some cases, where a model prediction and GT indicate the same numerical expression but the prediction is written in Hindi but GT is written in Arabic (and both expressions are in the text). This is an ambiguity of answers due to orthographical variants. Although I'm not familiar with Hindi, GT answers seem more frequently written in Arabic. Therefore, I performed a simple replacement (Hindi -> Arabic) by
{'०':"0", '१':"1", '२':"2", '३':"3", '४':"4", '५':"5", '६':"6", '७':"7", '८':"8", '९':"9"}
if the model predicts Hindi number but the same Arabic number also exists in the text.
(Maybe the same as the
[tkm kh](https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/287917)
's processing)
I tried the same trick for Tamil but not effective.
Earlier Phrases
It seems that a phrase at earlier positions can be an answer a little more frequently than the others.
I added a simple bias for the score by
- (2.0 * start_char_index / len(context)
.
Stride Boundaries
BERT-based QA by huggingface implementation split long text into many subtext chunks (with some stride setup), each of which is shorter than max_seq_length of the model.
I guess that prediction on the positions near the boundaries of chunks might be unreliable because some important surrounding context could be missing by chunking.
Therefore, I added a simple bias for the score by
if [it is NOT the beginning of the whole text] and [start_index_in_chunk is earlier than 10]:
    bias = -3
elif [it is NOT the end of the whole text] and [end_index_in_chunk is later than max_length-10]:
    bias -= 3
else:
    bias = 0
Even if a phrase gets a penalty due to the bias, the phrase may occur in another chunk without penalty because stridden chunks have overlapped tokens.
Things that Didn't Work (for My Ensembles)
I thought mT5s could be the best model. But, I failed to get good performances with them; even reproducing mT5 papers' results in SQuAD.
Label smoothing. It could help but almost seem meaningless. I also tried something similar to Jaccard-based Soft Labels, but less helpful.
MURIL Large. Many winners of chaii seem to use MURIL too. But, my MURIL was somehow not good.
info-xlm-large. It was very good as a single model. But, finally not helpful in my ensemble…
Tagging the questions by dataset-specific special tokens for discrimination, as used in NMT with
[backtranslation](assets/a979b018cf937262a8e174d1.pdf)
or for
[multilingual modeling](assets/e903b08f096b497e671c4e40.pdf)
. It seemed to alleviate the performance change due to dataset selection but seems not very helpful when selecting appropriate external datasets only.
I tried to use SQuAD-translated-tamil and XQA-tamil but were not helpful in my final submission.
In NaturalQA,
[it](https://arxiv.org/abs/1901.08634)
is reported that using CLS-token scores for calibrations.
[SAM-based optimization](https://arxiv.org/abs/2110.08529)
.
[xtune](https://github.com/microsoft/unilm/tree/master/xtune)
was very helpful for training a best single model. But, often less helpful in my ensembles.
Thanks!