#  congrats to Guanshuo Xu! and [14th place] solution 

This was a nice competition with lots of ideas and sharing on the forums - thank you to kaggle and the organisers for  hosting it.
I would like to congratulate my colleague (at
[h2o.ai](https://www.h2o.ai/)
) ,
[Guanshuo Xu](https://www.kaggle.com/wowfattie)
for winning the competition and for becoming the new kaggle #1
. It takes a lot of hard work and dedication to achieve this - well done!
My solution is fairly simple. Where I think I did really well is I never became emotional and never tried things that I did not see them working in validation first.
I used a single holdout (20%)  for validation (stratified by type) . I trained 2 models - an EfficientNet b3 and b4 on the training (80%) part of the data AND 2 more models on 100% of the data. So 4 models in total. Using a cosine learning schedule, all models (small and full ones), had very similar behaviour.
I generated multiple checkpoints predictions (over 10) from each one of these models. I progressively added more augmetations, swapped optimizers , (lowered) learning rates and batch sizes. I would make changes every-time I would see the validation performance getting halted. Every single time, I did get improvements from these swaps. In total, I trained the b3 model for 150 epochs and the b4 120 epochs.
Augmentations in stages:
Vertical and horizontal flips
Vertical and horizontal + transpose + rotate
Vertical and horizontal + transpose  + rotate + cutout (1 hole, 80)
Vertical and horizontal + transpose + rotate + cutout (2 holes, 64)
Vertical and horizontal + transpose + rotate  + cutout (4 holes, 64)
For TTA I used  Vertical , horizontal and  Vertical + horizontal
my models are all in pytorch . At some point, I tried to run EfficientNetb3 in keras (TF) with same batch size, optimmizers and augmentations and performance was significantly worse (not sure why) .
I got a lot of gain from stacking - around +0.003-4 in LB. For test predictions in stacking, I had 25% the predictions generated from the small(80%) model and 75% of the model trained with 100% of the data.
NN with 2 hidden layers, leakyrelu and a bit of l2 regularization had the best cv for me (0.936).
[Lightgbm](https://lightgbm.readthedocs.io/en/latest/Parameters.html)
with dart was a close second.
[EextraTreesClassificer](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html)
added a little bit too.