# 16th place solution with detailed and readable code for beginner

Thanks to
Makerere University AI Lab
and
Kaggle
for organizing this competition.
Here is
[our solution with code](https://github.com/freedom1810/kaggle-cassava)
, our approach is based on a
good CV splitting strategy
and better augmentation + loss + optimization to face the
noisy-label problem
.
1. Data Preprocessing + Augmentation:
Dataset: 2019 + 2020 (512 input size).
For training:
Simple augmentation
: RandomResizedCrop, Transpose, HorizontalFlip, VerticalFlip, ShiftScaleRotate, ShiftScaleRotate, HueSaturationValue, RandomBrightnessContrast, Normalize, CoarseDropout, Cutout (we referred to this
[kernel](https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug)
(many thanks to
[@khyeh0719](https://www.kaggle.com/khyeh0719)
), with a little bit change in the RandomResizedCrop and Normalize params).
Advance augmentation
: We tried with mixup, cutmix, fmix, snapmix. With our model, snapmix worked best with alpha = 5 (we refer to this
[kernel](https://www.kaggle.com/sachinprabhu/pytorch-resnet50-snapmix-train-pipeline)
for snapmix, many thanks to
[@sachinprabhu](https://www.kaggle.com/sachinprabhu)
).
For validation:
We remove RandomResizedCrop by using CenterCrop + Resize with cv2.INTER_AREA. Snapmix was also not used. We also applied TTA = 5.
2. Traning parameters
Loss function
: We tried focal cosine loss, bi-tempered loss, SCE loss (with label smoothing, of course). Bi-tempered worked best with t1 = 0.6, t2 = 1.2, label_smoothing=0.1, num_iters=5(we refered to this
[kernel](https://github.com/mlpanda/bi-tempered-loss-pytorch/blob/master/bi_tempered_loss.py)
for bi-tempered loss).
Optimization
: We tried Adam, Ranger, SAM. SAM worked best on our CV (0.2-0.3% higher than Adam on CV). However, it did not show good results on Public LB. Finally, we chose Adam.
LR Scheduler
: We used the LR scheduler based on the Yolov5
[code](https://github.com/ultralytics/yolov5/blob/master/train.py)
and
[tricks](assets/18311988b7d9debf0d51bef0.pdf)
on this paper with init LR = 1e-4 -> 5e-4, lrf = 1e-2.
Training strategy
: We unfroze the backbone after 5 epochs. The total training epochs were 50.
Backbone model
: All the pre-trained models are noisy-student models. We tried with B0 (using B0 as a baseline), B3, B4, and resnext50 using
[Timm](https://github.com/rwightman/pytorch-image-models)
. With limited resources (up to 2x2080Ti and Colab Pro), we were not able to obtain a better result with B4 compared to B3 on the CV.  Also, it took a lot of time for some experiments with ViT, but we did not get any prospects.
Some tricks
: We applied EMA on yolov5 with fp16. Using fp16 helped us to save a lot of time and computing resources.
3. Splitting strategy
It sounds really simple. We separately divided the 2020 and 2019 data into 5folds. After each trial, we mixed the best and the worst folds on the CV and resplitted them until the gap CV is not large than 0.4% (it took 4-5 days using Colab Pro with B0). It helped us to boost our result at the very last submits.
4. Experiments
Here are some of our outstanding experiments.
Model
Note
CV
Public LB
Private LB
B0
SAM + 10 TTA
0.8993
0.8969
0.8989
B3
Adam + 10 TTA
0.9011
0.9036
0.8984
B3
SAM + 10 TTA
0.9029
0.9027
0.8981
B3
SAM + newest splitting  + 5 TTA
0.9033
0.9007
0.8992
B3 + B4 + resnext50
5 TTA
…
0.9045
0.9014
B3 + B4 + B0
5 TTA
…
0.9032
0.8990
B3 + B0 + resnext50
5 TTA
…
0.9030
0.9001
The inference notebook is not included in our code, inference TTA is similar to validation TTA.
We are working to modify your code as readable and usable as possible. I hope it could help the newbie to get familiar with other Kaggle competitions.
P/S:
We are a Vietnamese team (me and
[@gdvipbb258](https://www.kaggle.com/gdvipbb258)
), thank you in advance for reading!