# 28th Place Solution(Contains additional reading material)

Congratulations to the winners in the medal place! Also, thanks for everyone contributing to this competition!
Then, I'm happy that I get my second silver medal in this cometition which is my first time to participate a multilingual  languge competition (maybe I have some chance to get a gold one 😆). My effective solution almost come from new translated data, ensembling method and diversity model with different trained data.
translated data :
[https://www.kaggle.com/vanle73/nq-translated-data](https://www.kaggle.com/vanle73/nq-translated-data)
It contains two version data with different number samples. I translated with Google API before I truncated the span keeping the maxium length less than 5000.
ensembling method
logits average (like almost everyone do)
logits sum vote:
This method is designed for different pretrained model with different tokenizer. I choose the top 3 different predicted answers for a single sample, keeping span score, then sum all span score from different model, finally get the top 1 ranking answer as the final result.
The two methods have the similar socre in private /publie LB.
diversity model with different trained data
At the beginning, I regard this competition as a weakly supervised problem, So I try to denoise the data using HardEm algorithm iteratively, it actually have a obvious improvement relative to just using origin data, but still below the baseline. Then I try to do neg-samlping to filter some samples, it still have some improvement, but a low score. Then I found this competition is a "data competition" after I add xquad/mlqa data.
So I just translated the similar data ---NQ which I joined last two years. It actually bring more benefit. My finally ensembling method also contain all model mentioned above because of diversity.
Besides, I also tried some other methods, having not get a better result. beacure I have not my own training GPU and not do enough experiment, my teammates does😨. So I list it for your reference hoping helpful:
Pretained model :
[indic-bert](https://indicnlp.ai4bharat.org/indic-bert/)
[hindiBert](https://huggingface.co/mrm8488/HindiBERTa)
[roberta-hindi](https://huggingface.co/flax-community/roberta-hindi)
[info-xlm](https://github.com/microsoft/unilm/tree/master/infoxlm)
[xlm-align](https://github.com/CZWin32768/XLM-Align)
Different model structure:
[FILTER](https://github.com/yuwfan/FILTER)
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Ensembleing:
[BERT for Question Answering on SQuAD 2.0](assets/dc155b7b9021bf2bdaad7c13.pdf)
[Gestalt: a Stacking Ensemble for SQuAD2.0](https://www.linkedin.com/pulse/stacking-ensemble-squad20-mohamed-el-geish/)
[Applying Ensembling Methods to BERT to Boost Model Performance](assets/0e4a7e14e52ec82b863284cb.pdf)
[Ensemble ALBERT and RoBERTa for Span Prediction in Question Answering](assets/f07bc9644f85b72e3a29ffb0.pdf)
offline translated model:
Hi2en:
[https://huggingface.co/Helsinki-NLP/opus-mt-hi-en](https://huggingface.co/Helsinki-NLP/opus-mt-hi-en)
En2hi:
[https://huggingface.co/Helsinki-NLP/opus-mt-en-hi](https://huggingface.co/Helsinki-NLP/opus-mt-en-hi)
Tmail2en:
[https://huggingface.co/Helsinki-NLP/opus-mt-dra-en](https://huggingface.co/Helsinki-NLP/opus-mt-dra-en)
En2tmail:
[https://huggingface.co/Helsinki-NLP/opus-mt-en-dra](https://huggingface.co/Helsinki-NLP/opus-mt-en-dra)
mBART-50:
[https://huggingface.co/facebook/mbart-large-50](https://huggingface.co/facebook/mbart-large-50)
weakly supervised method:
MML& HardEM:
A Discrete Hard EM Approach for Weakly Supervised Question Answering
[https://doi.org/10.18653/v1/D19-1284](https://doi.org/10.18653/v1/D19-1284)
.
Mutual Information Maximization Approach:
A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering
[https://arxiv.org/abs/2106.07174](https://arxiv.org/abs/2106.07174)
Finally,  thank
[@thedrcat](https://www.kaggle.com/thedrcat)
[@abhishek](https://www.kaggle.com/abhishek)
[@nbroad](https://www.kaggle.com/nbroad)
for your notebooks( Extremely agonizing text tokenizing processing🙄 ! ).
Update:
Score vote maybe a better ensembling method in this competition for private LB! My best submissions also come from it but I havn't choosed them😂
[](https://postimg.cc/Y4qWR1PC)