# 21 place solution: Trust CV & Differential fusion

Thank kaggle stuff holds this great competition. Thank the great community provides so many fantastic notebooks and discussion. Finally thank my teammate
[@guohey](https://www.kaggle.com/guohey)
and
[@thousandtie](https://www.kaggle.com/thousandtie)
for accompanying me on this unforgettable 10-day India trip.
Our solution is mainly the topic: trust the cv and fuse models with as much different as possible.
We can't find a way to ensemble rembert and xlmr, so we decided to use only Rembert in one final submit and XLMR in the other.
Why do we decide to Violently fusion of different models
Because we found that adding data can't improve performance of single model in both CV and LB.
Why do we decide to trust CV more than LB?
Actually, there's a bit gamble. We can't get a better LB score than the 792 submission before the last 2 days. Besides, no matter how I blend my own model with 792, even if I give my model a small weight, it's worse than 792. I calculate the cv and lb of each model in the 792 submission, finding the fold 0 and fold 1 have poor lb and cv, while the last 3 folds have great cv and great lb. However, without using the poor first 2 fols, its LB score drops greatly. Reading the training notebook of the first 2 folds, I found nothing special.
Taking the above experimental results and analysis into consideration, I would rather believe the 792 submission overfit LB. So instead of trying to beat the public 792 on LB, I chose to use my CV's better model to ensemble and prayed for its shake up on PLB.
Luckily,  this gamble paid off.
How do we make models different?
We take different seed and different data.
For example, the 12 folds rembert submission consists of the following models:
2 fold with
only hindi external data
and
seed 2021
(LB 782)
3 fold with
hindi and tamil external data
which is popular in community and
seed 42
(LB776)
4 fold with the same data of the above one and
seed  1111
(haven't got LB, we didn't have enough submission)
2 fold with the data of the above one
adding a sample of 15k
[@zzy990106](https://www.kaggle.com/zzy990106)
external tamil data
and seed 99.
We tried some other combinations, sampling more external data and replace it with some folds in the seed1111 part. But this one plays best in LB, so we use it for final submision of rembert.
We take the same principle in xlmr, but maybe we use some/all folds of the public 792 models in our xlmr ensemble, this strategy didn't work in xlmr.(No matter how I blend with 792, it is worse in Lb.)
How do we trust our cv?
We train 5 fold of different data and seed each time. After we got weight, we calculate their CV. We only use CV better than 0.68 to ensemble.
What can be improved in the future？
We can't find a way to evaluate the cv of our ensemble submission, because there are different seeds, and each seed only use 20%-80% model.
In fact our best submission has a 0.760 PLB, which is really close to gold. But its LB is worse than our best 795 rembert submission, so we didn't choose it. Maybe with a better cv strategy, we have a chance to get a gold medal.
If you have any ideas to calculate cv of the ensemble output, please discuss with me!
Thank you for your reading, feel free to comment and upvote!