# 🥈 Quick overview of the 26th place solution [LB 0.24838]

Congratulations to winners and my team
[@taniaj](/taniaj)
,
[@dskswu](/dskswu)
,
[@arigion](/arigion)
We didn't manage to fine-tune Bert, but still had a couple of ideas that worked pretty good.
Used Bert-large with concatenated embeddings from -3, -4, -5, -6 layers. As
[@ceshine](/ceshine)
pointed out in his great kernel (
[https://www.kaggle.com/ceshine/pytorch-bert-endpointspanextractor-kfold](https://www.kaggle.com/ceshine/pytorch-bert-endpointspanextractor-kfold)
), it's better not to use the last layer. After some experimentation, we settled on -3, -4, -5, and -6 layers based on CV scores.
Used corrected and original data (
[https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331#latest-503495](https://www.kaggle.com/c/gendered-pronoun-resolution/discussion/81331#latest-503495)
). We couldn't decide which models to choose: the ones that were trained on original or corrected data. Until we tried to blend them and it worked great (0.34-0.35 for original/corrected and 0.33 after the blending for Public LB). Our 2 final submissions were only different in terms of weights in blending. We tried 0.6/0.4 and 0.5/0.5 for corrected/original data. Just like in Public LB, 0.6/0.4 split worked better (0.24838 vs 0.25200 Private LB)
Applied data augmentation inside each fold by simply swapping A and B columns and concatenating with original data.
10 folds, of course.
A simple NN (4 layers with BN, ReLU and dropout) on top of bert embeddings and linguistic features from public kernels. More complicated architectures didn't bring anything to the table.
It's not as sophisticated as
[@boliu0](/boliu0)
solution (great job, btw. Really impressive), but I still hope you find it useful.