# 13th Place Solution

I would like to thank Kaggle and Chaii (Challenge in AI for India) for setting up this interesting Multilingual Question Answering Challenge.
It turns out we have a couple of solutions that would put us Gold Position but that is how these competitions go sometimes!
Acknowledgements
A big shout out to
[@kishalmandal](https://www.kaggle.com/kishalmandal)
,
[@abhishek](https://www.kaggle.com/abhishek)
and
[@nbroad](https://www.kaggle.com/nbroad)
for their wonderful kernels and models that we could leverage.
Also thanks to my wonderful teammates
[@khyeh0719](https://www.kaggle.com/khyeh0719)
and
[@maxjeblick](https://www.kaggle.com/maxjeblick)
for their contributions to our solution. This was a team effort!
External Datasets
The models in the final pipeline used a mix of the baseline dataset and the MLQA and Squad datasets that were disclosed early on in the competition in the datasets page here:
[https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/264581](https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/264581)
Models
All models below were trained with doc_stride=135, max_length=400
Model
LB Score
Comments
# Folds Used
deepset/xlm-roberta-large-squad2
0.792
5 fold Public Model
5
deepset/xlm-roberta-large-squad2
0.781
10 fold Tez model Trained Locally
3
google/rembert
0.777
5 Fold Trained Locally
4
google/muril-base-cased
0.772
5 Fold Trained Locally
4
Apart from the models above we tried various different architectures including but not limited to
electra, muril-base, multilingual BERT, span BERT, tamillion, indic-bert
but none of them were as good as the architectures in the table for this task.
There was very little CV -> LB correlation in the competition for us. High CV scores did poorly on LB while decent but not too poor CV results gave higher LB scores. I suspect that this is due to the noise in the dataset and also that private dataset is cleaner with multiple annotators labeling the data.
Speeding up calculations
-
model.half()
- FP16 for slight improvement in speed and no impact to LB score
- Reusing features for same archs
- Torchscript/jit - Experiments resulted in out of memory
- Tried joblib and multiprocessing to do parallel calculations - All experiments worked on train set but out of memory on test. Maybe because test set had longer contexts?
Ensemble Strategies
One of the main challenges was the 5 hr kernel run limit. With different architectures and possibly different max_len/doc_stride it became pretty evident that we needed a good way to ensemble models. Averaging logits only works for same archs and if same doc_stride/max len is used.
I read up best practises on how do to this including top solution in the TensorFlow Question Answering Challenge here:
[https://www.kaggle.com/c/tensorflow2-question-answering/discussion/128832](https://www.kaggle.com/c/tensorflow2-question-answering/discussion/128832)
. I came across majority/hard voting in the solution descriptions and decided to implement a very simple voting algorithm that landed up giving a significant boost to LB score.
Voting
The voter took the cleaned predictions (after basic post processing) as inputs. We had 4 architectures in the final pipeline and the first stage checked for a majority vote among the 4 candidate predictions based on the top answer from each architecture. If there was no majority, the voting pool was expanded to top N predictions among all architectures. We found that taking top 2 votes from each arch gave good results. To further improve on the 2nd stage of the voting, if there was still a deadlock in voting, the votes were compared for similarity using the Levenshtein Ratio and if votes were within a certain threshold, one of those answers were selected as the final since the metric is Jaccard and having part of the answer correct would still be useful.
Score progression
- Public LB kernel ->
LB 0.792
-  Voting ensemble with 3 different architectures  ->
LB 0.798
(+0.006)
-  Improving models in voting ->
LB 0.801
(+0.003)
-  Adding another architecture to voting and selecting best models  ->
LB 0.809
(+0.003)
- Fine tuning voting logic (see above) ->
LB 0.811
Code can be found here:
[https://www.kaggle.com/trushk/qa-ensemble-voter-top-n-votes-similarity](https://www.kaggle.com/trushk/qa-ensemble-voter-top-n-votes-similarity)
Description of voting logic:
[https://www.kaggle.com/trushk/ensemble-using-voting-13th-place-solution](https://www.kaggle.com/trushk/ensemble-using-voting-13th-place-solution)