# 22nd Place Solution: Model Architecture

Thanks to Kaggle for hosting an interesting and challenging challenging NLP competition. This was my first serious NLP competition and really enjoyed every bit of it and learnt a lot on the way.
Firstly, a couple of things made this very very challenging competition
Data annotation issues: There were multiple data annotation issues in the train data and although it seemed easy to get frustrated, the challenge of the competition was to handle this kind of real world data and annotation issues that exist all the time.
Lack of CV/LB corelation: Since there was no good co-relation, it meant that we didnt really know what to focus on. This lead to efforts being distributed all over the place. I trained XLM-Roberta Large, MURIL Large, tried multiple loss functions, and different kinds of heads and only after the results were announced did we know that Muril and Mean/Max head really worked (more on this below)
Architecture
XLM-Roberta: I used the Trainer models from Hugging Face
Muril-Large: These had a really good CV/LB co-relation, but the training was really unstable. I couldnt get it work unreliably across a number of experiments. This is something I will seek to improve after the competition.
XLM-Roberta with Mean/Max Head: This worked like a charm in terms of CV/LB co-relation and unfortunately I realised this in the last week. Whenever, CV increased LB score jumped and I would really like to understand why this particular "head" works so well. Credit to
[@rashmibanthia](https://www.kaggle.com/rashmibanthia)
for this idea
[here](https://www.kaggle.com/rashmibanthia/xlm-roberta-jaccard-validation-mean-max-head/notebook)
Muril and Mean/Max really proved once again, that architecting and using great models is really important for a good outcome.
Data
Augmenation: I tried random-crops and was excited about this idea, but the CV didnt really improve. I should have tried checking the Public LB score of this technique, but I think it wouldn't have improved. The reason this probably didn't work is because the model is too powerful to be fooled by this level of augmentation.
Auto-Translated Data: I used Auto-translated data, but this really hurt performance. This makes sense since the translations are not really of great quality.
TyDi Dataset: I experimented with all the languages in the Tydi Dataset and found that that Telugu performed the best, followed by English and Bengali. Other languages didnt really do well.
Post-processing
I fine tune the post-processing shared by
[@nbroad](https://www.kaggle.com/nbroad)
to remove other special characters like ( and ) and also replace k.m with k.m. etc. This gave a boost of 0.002
Things that didn't work
Data Augmenation using Auto-translated data: As described above this didn't work well
Using Median instead of Mean: I had spent sometime in the Google Brain - Ventilator Pressure Prediction and there was an idea by
[@cdeotte](https://www.kaggle.com/cdeotte)
that really worked well when the predictions were far removed from mean. I tried the same idea of using Median here, but that dropped score by 0.005 on LB. Upon further evaluation, it did look like the eventual logits were not as far apart as I thought they were.
End_Logits modification: During evaluation of model predictions, I realised that the start_logits was working well, but the model was placing a low emphasis on end_logits. So I tried fine tuning it with
np.log100(np.sqrt(end_logits))
. This did work well on public LB by about 0.002, but i eventually dropped it since I felt it wouldn't do well in the Private LB (and that did happen).
Things that I wanted to try but didn't get to
Base Tuning on Auto-translated data and then fine-tune on chaii+mlqa+tydi data: I believe this has real potential, but didn't have the time to try
Ensemble Voting: I thought this had potential, but didn't get to it.
Finally, thanks to everyone who contributed to make this competition interesting.