# 25th place solution

Sorry for the late writeup 😅
First of all, I would like to thank you to Kaggle for organizing this competition and great Kaggle community. Special thanks to
[@khyeh0719](https://www.kaggle.com/khyeh0719)
,
[@piantic](https://www.kaggle.com/piantic)
, and many other great notebooks that I didn't mention.
This is my first competition and it's been a wild ride of learning.
Summary
My final submission two submissions are stacking ensembles of efficientnet b4 and ViT models (1st submission) and only ViT models (2nd submission). My solution is relatively simple compared to other competitors.
Datset
Efficiennet only used 2020 data. I tried to use both but it didn't score well. I think it has something to do with the 2019 image sizes, as I had to resize the image first in augmenetation.
ViT, Deit, R50+ViT I used 2020 + 2019 (data only for class 0,1,2,4). Class 3 already has many images (6x higher than other classes).
Architectures
Model
Public LB
Priave LB
Efficientnetb4
0.9023
0.8977
Deit_base_patch32_38
0.9018
0.8980
Vit_base_patch32_38
0.9061
0.8978
R50+ViT-B
0.9009
0.8975
Augmentation
I spent few days trying to find the best augmentations. This is my final augmentation strategy.
RandomResizedCrop
Transpose
HorizontalFlip
VerticalFlip
ColorJitter (brightness=0.2, contrast=0.2, saturation=0.2, hue=0.2, always_apply=False, p=0.5),
OneOf([MotionBlur(blur_limit=3), MedianBlur(blur_limit=3), GaussianBlur(blur_limit=3),], p=0.5,),
Normalize
CoarseDropout
Cutout
I tried cutmix, mixup, fmix snapmix but they didn't work for my models. I think coarsedropout and cutout already quite heavy augmentation.
Training parameters
I tried all the loss functions available in
[@piantic](https://www.kaggle.com/piantic)
shared notebook. Also, some basic mix of loss functions (active and passive). Labelsmoothing =0.3 give me the best results.
Scheduler: CosineAnnealingWarmRestarts
Optimizer: AdamP
Train for 10 epoch + 4 epochs of fine tunning.
Ensemble
I tried different combinations of  my models ranging from 2-4 models. Only ViT and efficientnet have good synergy.
My two submissions are:
ViT base 16 (10 folds) of 2 learning rate 1e-2 and 1e-6 = Public LB 0.906 Private 0.8984. Many kagglers reported that ViT achieved high public LB but low CV, so I'm not that suprised.
Efficientnet (0.4) and ViT (0.6) Public LB = 0.9068, Private LB =0.9011.
My best private score is 0.9017, which is a combination of efficientnet (0.4) and ViT (0.6) but with ViT model that trained with only 2020 dataset.
I still have a long way to learn, so any suggestions are welcome.
Thank you for reading.
PS. Sorry for my bad grammar.