# Tentative 2nd place solution

Hi all,
Congratulations
[@thedrcat](https://www.kaggle.com/thedrcat)
! Seems like we trusted LB too much. But, we wrer able to remain in 2nd place. I am happy and I can be GM! Thanks a lot,
[@kanbehmw](https://www.kaggle.com/kanbehmw)
. His knowledge of NLP has boosted our score significantly.
Here is our tentative solution write-up. We will publish all our codes and datasets. Please post any questions in this thread if you have any.
Resources
Our inference notebook is available here:
[https://www.kaggle.com/kanbehmw/tkm-tydi-rem-info-muril-top35](https://www.kaggle.com/kanbehmw/tkm-tydi-rem-info-muril-top35)
Training code is in each dataset
External data
We trained our models with the competition data, MLQA, and TyDi QA (only Bengali and Telugu). In particular, TyDi QA pushed our score largely (0.787 -> 0.799 in public LB). We think most of the top teams achieving >0.81 use the TyDi QA dataset. It may be due to the language similarity among  Bengali, Telugu, Hindi, and Tamil. However, we suspect some possibility that some questions are shared even though they are translated to Hindi or Tamil since TyDi QA is owned by Google. This is just a guess.
Model Training
We just did 2 epochs of training with the whole data (no fold out) on the huggingface trainer. We do not distinguish between Hindi and Tamil. For more details, The training code should be included in each dataset. Please have a look.
Loss function
While the original cross-entropy score has the best performance, the loss function described in
[https://github.com/heartkilla/kaggle_tweet](https://github.com/heartkilla/kaggle_tweet)
(used in the previous competition) worked to obtain more diverse models. In our kernel whose name ends "exp33" is trained with the "Jaccard-based Soft Labels Since Cross-Entropy".
Local Evaluation (CV) Strategy
Nothing. We tune everything based on Public LB. I did not even track local scores. This is the first time since I started Kaggle that I have not tracked local scores at all.
Model Ensembling
As in the CommonLit competition (
[https://www.kaggle.com/c/commonlitreadabilityprize/overview)](https://www.kaggle.com/c/commonlitreadabilityprize/overview))
, model ensembling seems to be the key technique to win NLP competitions. We ensemble the following models:
XLM-R (7 models)
deepset/xlm-roberta-large-squad2
One model trained with Google-translated SQuAD
[https://www.kaggle.com/tkm2261/google-translated-squad20-to-hindi-and-tamil](https://www.kaggle.com/tkm2261/google-translated-squad20-to-hindi-and-tamil)
AlexKay/xlm-roberta-large-qa-multilingual-finedtuned-ru
RemBert (3 models)
InfoXLM (3 models)
MuRIL (2 models)
We don't know why the models fine-tuned in Russian had a good performance. We stacked the models gradually and our scores gradually improved. (0.799 -> 0.816 -> 0.821 -> 0.827 -> 0.829). The MuRIL models pushed our score from 0.827 to 0.829.
Misc.:
To handle the difference of the token size in a context, we took the softmax function on the entire text before ensembling different types of models.
I do not use the 0.792 models.
I do not use CV models. All models are trained with the entire data. I think changing seeds and ensembling them should be better than ensembling CV models.
Effect of max_seq_length and doc_stride
As in my notebook
[https://www.kaggle.com/tkm2261/reproduction-of-0-792-notebook](https://www.kaggle.com/tkm2261/reproduction-of-0-792-notebook)
, max_seq_length and doc_stride seem to have a large impact. Even though I explored a lot of values of these two parameters, it did not help to improve our score. So, we just use max_length = 384 and doc_stride = 128.  Maybe, this part is not important since we ensemble models with different token sizes.
Post-processing
As shown in this kernel (
[https://www.kaggle.com/nbroad/chaii-qa-torch-5-fold-with-post-processing-765)](https://www.kaggle.com/nbroad/chaii-qa-torch-5-fold-with-post-processing-765))
, post-processing has a large impact on this competition. Since the test data is annotated by 4 people, they are not very likely to forget the trailing periods like “​​ई.पू.”, i.e., answers should be in the most formal format. Besides, we processed the Devanagari numbers (०१२३४५६७८९). We replace them with Arabic numerals (0-9) if the selected answer consists only of Devanagari numbers. It seems like the annotators agreed that they use Arabic numerals when the answer is a year. Without the postprocessing, our score dropped to 0.806. So, it pushed our score by around 0.02.
Public LB score timeline
0.787: Teamed up
0.799: Trained with TyDi QA (only Bengali and Telugu)
0.816: Ensembled with Rembert and InfoXLM
0.821: Ensemble with more XLM-R, Rembert, and InfoXLM
0.827: Tune ensemble weights
0.829: Ensembled with MuRIL
0.831: Tune parameters in post-processing
Did not work
After we found TyDi QA helps a lot, we explored a lot of datasets. But, almost all datasets did not work. In our case, Only MLQA and TiDy QA worked. XQuAD was not bad, but I did not use it.
#
Name
URL
1
Bengali Question Answering Dataset
[https://www.kaggle.com/mayeesha/bengali-question-answering-dataset](https://www.kaggle.com/mayeesha/bengali-question-answering-dataset)
2
XQuad
[https://github.com/deepmind/xquad](https://github.com/deepmind/xquad)
3
UQuAD
[https://t.co/OP6FYHbuju?amp=1](https://t.co/OP6FYHbuju?amp=1)
4
TQuAD
[https://github.com/TQuad/turkish-nlp-qa-dataset](https://github.com/TQuad/turkish-nlp-qa-dataset)
5
PersianQA
[https://github.com/sajjjadayobi/PersianQA](https://github.com/sajjjadayobi/PersianQA)
6
MMQA dataset: Hindi QA dataset
[https://github.com/deepaknlp/MMQA](https://github.com/deepaknlp/MMQA)
7
XQA: testset for Tamil QA
[https://github.com/thunlp/XQA](https://github.com/thunlp/XQA)
8
bAbi 1.2 dataset: Has Hindi version of bAbi tasks in romanized Hindi.
[http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz](http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz)
9
HiQuAD
[https://www.cse.iitb.ac.in/~ganesh/HiQuAD/clqg/](https://www.cse.iitb.ac.in/~ganesh/HiQuAD/clqg/)
10
ARCD
[https://github.com/husseinmozannar/SOQAL/tree/master/data](https://github.com/husseinmozannar/SOQAL/tree/master/data)
11
FQuAD
[https://fquad.illuin.tech/](https://fquad.illuin.tech/)
12
Squad_Translated_to_Tamil for Chaii
[https://www.kaggle.com/msafi04/squad-translated-to-tamil-for-chaii](https://www.kaggle.com/msafi04/squad-translated-to-tamil-for-chaii)
13
RuBQ: A Russian Knowledge Base Question Answering and Machine Reading Comprehension Data Set
[https://github.com/vladislavneon/RuBQ](https://github.com/vladislavneon/RuBQ)
14
SberQuAD
[https://t.me/s/opendatascience?q=%23SberQuAD](https://t.me/s/opendatascience?q=%23SberQuAD)
15
XOR-TyDi QA
[https://github.com/AkariAsai/XORQA#additional-resources](https://github.com/AkariAsai/XORQA#additional-resources)