# Private 606th / Public 34th Solution

Hi everyone!
First of all, congratulations to all the winners and did a great job on all participants' hard works!
Overview
I selected two versions based on best CV & LB scores because I'm really worried about a huge shakeup. (However, I did not make it out :cool_crying:)
Public  LB:   34th, 0.907
Private LB: 606th, 0.898
Base Model
I only got Kaggle GPU, so I couldn't experiment with many models with various recipes.
I tried 6 models (ResNeSt50,  ResNeSt50 4s2x40d, EffNet-B3, 4, ViT-L/16, DeiT-B/16) and decided to use
ResNeSt50 4s2x40d
,
EffNet-B4
based on LB & CV scores.
arch
CV
Public LB
Private LB
ResNeSt50-fast-4s2x40d (tta4)
0.891
0.898
0.895
EfficientNet-B3 NS (tta4)
0.895
0.896
0.895
EfficientNet-B4 NS (tta4)
0.891
0.900
0.893
To increase the robustness, I trained the model on different training recipes with the same architecture. I chose to use
focal cosine loss
&
cross-entropy w/ label smoothing 0.2
.
arch
loss
CV
Public LB
Private LB
EfficientNet-B4 NS (tta3)
focal cosine
0.890
0.900
0.895
EfficientNet-B4 NS (tta4)
cross entropy w/ label smooth
0.891
0.900
0.893
Lastly, the pseudo label boosts the CV score +0.007. (I generated a pseudo label based on my best CV score, 0.9051)
arch
pseudo
CV
Public LB
Private LB
EfficientNet-B4 NS (tta4)
no
0.891
0.900
0.893
EfficientNet-B4 NS (tta4)
yes
0.898
0.898
0.893
Ensemble
Weights
Single model's CV & LB scores seem not good, But, ensembling raises the CV & LB scores to 0.005. Also, TTA increases LB scores +0.002.
I tuned the ensemble weights with
Optuna
&
[Simple](https://www.kaggle.com/daisukelab/optimizing-ensemble-weights-using-simple)
on CV.
Here is the script :
[optimive cv](https://www.kaggle.com/kozistr/valid-optimize-cv)
Combinations
best LB : 2 x ResNeSt50-fast-4s2x40d + 2 x EffNet-B4
best CV : 3 x ResNeSt50-fast-4s2x40d + 3 x EffNet-B4 + 1 x EffNet-B3
baseline
CV
Public LB
Private LB
best LB
0.9003
0.907
0.897
best CV
0.9051
0.905
0.896
Here is the script :
[inference](https://www.kaggle.com/kozistr/inference-rns50-effnet-b4)
TTA
Simply using
CenterCrop + horizontal & vertical flop
achieves better CV & LB scores.
Training Recipe
Data : 2019 + 2020 datasets, only 2020, w/ pseudo labels
Validation : stratified kfold - 5 folds
Augmentations : FMix, CutMix, SnapMix, MixUp, Cutout, etc…
Optimizer : AdamW, AdamP, RAdam
Loss : focal cosine, cross-entropy w/ label smoothing, bi-tempered
Scheduler : Cosine Anealling w/o warm-up (1e-4 to 1e-6)
Epochs : 10 ~ 20
Resolution : 512x512
Auxiliary : freeze BN layer
TTA : n_iters 4 is best in my case.
Experiment :
[note](https://www.kaggle.com/kozistr/cassava-leaf-disease-experiments)
Works
Label Smoothing (alpha = 0.2)
TTA (n_iters 4 is best)
Adam, AdamW to RAdam (slight improvement)
external dataset (2019 dataset)
Not Works
noise-tolerant(?) loss like bi-tempered loss
Reflections
ensemble weights
I guess tuning ensemble weights might be one of the reasons, especially where there're noisy labels. Usually, the tuned versions tend to have a lower score than the average one. (-0.001 ~ -0.002 gap on Private LB score). But, Ironically best Private score is the tuned version (not the selected one).
multiple scales (image resolution)
I think training & inferencing with multiple scales would be helpful to improve the score. On several experiments, a lower resolution (384x384) works better than a higher resolution (512x512).
Thank you very much!