# 9th place solution

Hi, somewhat surprisingly we achieved 9th place. Our aim for this competition was to try models trained with WECHSEL (
[https://openreview.net/forum?id=JcfISE1-u4](https://openreview.net/forum?id=JcfISE1-u4)
, you've probably seen the models
[here](https://www.kaggle.com/c/chaii-hindi-and-tamil-question-answering/discussion/286603)
) as kind of a sanity-check of the method when compared to SOTA multilingual models. To make our submission competitive we combined the WECHSEL models with some others:
deepset/xlm-roberta-large-squad2
with
max_len=384
doc_stride=128
batch_size=32
learning_rate=3e-5
epochs=2
max_grad_norm=1.0
and appending MLQA and XQUAD data to the train set.
muril-large-cased
with
max_len=384
doc_stride=128
batch_size=32
learning_rate=3e-5
epochs=2
max_grad_norm=0.5 (this helped stabilize training for Muril)
and appending MLQA and XQUAD data to the train set.
roberta-large-wechsel-hindi
with
max_len=384
doc_stride=128
batch_size=32
learning_rate=3e-5
epochs=2
and appending MLQA and XQUAD data to the train set.
Tamil data
We did not get
roberta-large-wechsel-tamil
to work properly because the Tamil dataset is very small.
[@cdminix](https://www.kaggle.com/cdminix)
investigated adding Tamil data by scraping questions in Tamil from the web and automatically finding Wikipedia passages where the answer occurs. We ended up with ~200 extra Tamil samples this way. This improved the score of our Tamil WECHSEL model but it was still not on par with multilingual models, and we didn't really have time to investigate further.
Preprocessing
We removed references from the text (all square brackets where only numbers are between the open and closing brackets) since a lot of those occured in the text.
Stacking
We combined models by converting logits to the character level: For every token, we simply repeated the logit for each character in the token. We subtracted
0.01 * character position
for the start logits and
0.01 * (token length - character position)
for the end logits to make sure the longest character span within a token is selected as the answer.
After getting logits to the character level we simply summed up the predictions of different models, without any weighting.
Submission
After disclosing our WECHSEL models and making our submission more or less competitive by stacking with other models, we got frustrated with tuning hyperparameters and not being able to improve our public score that much. So thinking that we would need to tune more parameters and that there where probably some tricks necessary to achieve high scores (like fixing answer start, more external data) we decided not to put any more work into the competition five days ago and didn't choose any final submissions.
So our submission was automatically chosen to be the best submission on the public LB, which was 2x each model trained on the full dataset. Our last submission which we would have expected to be best scored 0.787 on the public LB using 5-fold CV of each model. It scored 0.779 on the private LB but was not automatically selected as one of the final submissions.
![Best model score](https://i.imgur.com/oSryczx.png)
That would have been 2nd or 3rd place so it's a bit frustrating that we didn't select anything, but overall it's still obviously much better than we expected so I can't complain.
My main takeaways are:
It is really, really easy to overfit to the public LB when there is this little data, so it might have been a blessing in disguise that we didn't spend much time on this competition.
Select a final submission, even if you don't expect to do very well :)