# 6th place solution - with a visualization of our Jaccard scores over time

First of all, I would like to thank Kaggle and the competitions hosts for hosting this interesting nlp challenge. I like the trend, that we see more and more nlp competitions. I need a short break but will join Jigsaw for sure 😉.
We realized early in the competition
that the local cv is not really meaningful
. We decided the first time in a competition to focus on the public leaderboard instead on the cv but
keeping diversity all the time
to avoid a shakedown. Diversity was already a key for our gold medal in the CommonLit challenge. As all teams we started with XLM Roberta Large. We pushed it to
0.785
but didn’t manage to improve it further
(1)
.
[](https://ibb.co/wdwXFCp)
After that we had obviously a breakthrough. We managed to
ensemble models of different architectures
. We mapped the logits to the character level to combine outputs of different tokenizer sizes (link to the notebook the end of my post).
After that we ensembled our Roberta Large with Muril Base (0.732) and scored
0.792 (2)
. We implemented a normalization of the logits to take different scales into account and scored
0.795 (3)
.
Now a link to Muril Large was posted in the forums. We struggled to finetune it but finally created a model with 0.784 public score. We ensembled it with our Roberta Large and had a big jump to
0.808 (4)
. An interesting sidenote is that this model would have score 0.776 on the private lb already.
Now we added Rembert (0.774 single score) (5) and XLM Info Large (6). In the last days of the competition, we spent a lot of time finetuning the weights of our ensemble (7). We also added the Wechsel-models with a small share in the end (8) and
ended up with 0.821 public lb score
.
Our final ensemble was:
Rembert Large - 16.8%
Muril Large - 29.4%
Roberta Large - 44.1%
Xlm Info Large - 5.7%
Wechsel - 4%
I'm happy with the result, even though we were out of the top 5 in the end. Because of the small data, the huge gap between cv and lb,
it felt right to focus on the public lb
. I expected a much higher correlation between private lb and public lb than between private lb and cv. I have checked it for our submissions and feel vindicated:
[](https://imgbb.com/)
It was a pleasure to tackle this problem with you guys. I will try to be more active in the forums next time and wish you all a good time.
Thanks to my awesome teammate
[@chamecall](https://www.kaggle.com/chamecall)
Link to our notebook:
[https://www.kaggle.com/benbla/fork-of-chaiiqa-inference-modelsensembling-eba980](https://www.kaggle.com/benbla/fork-of-chaiiqa-inference-modelsensembling-eba980)