# 58 place solution

Hi everybody and congratulations to all of the participants!
Data
We used resized images from 2015 competition to pretrain our models.
Also, we added IDRID and MESSIDOR data to cross-validation while finetuning.
Models
We tried a lot of models from the start of competition, but got no luck with either small (ResNet34/DenseNet121) nor very large models (EfficientNet-B6/B7, ResNe(x)t101, DenseNet201)
Best scoring modes were of medium size and medium resolution:
EfficientNet-B4: 380x380
EfficientNet-B5: 456x456
SE-ResNeXt50: 512x512
SE-ResNeXt50: 380x380
We got no improvement after 512x512 images.
Preprocessing
At first, we used circle cropping and Ben's preprocessing fro a while, but then started to experiment with tight cropping and corners masking, which was worse on every model we tried (both CV and LB).
A week before the competition deadline we realized, that we lose too much information and switched to no preprocessing and hard augmentations instead.
Augmentations
We used a lot of augmentations, all from Albumentations library:
OpticalDistortion, GridDistortion, PiecewiseAffine, CLAHE, HorizontalFlip, VerticalFlip, RandomRotate90, ShiftScaleRotate, RGBShift, RandomBrightnessContrast, AdditiveGaussianNoise, GaussNoise, MotionBlur, MedianBlur, Blur, Sharpen, Emboss, RandomGamma, ToGray, CoarseDropout, Cutout.
Training
We used multi-target learning (classification, ordinal regression and regression), and their result was transformed to regression combined with a linear layer (initialized with 0.3333 weights) to get regression output.
At first, we trained only on the current data, but then switched to pretraining on 2015 dataset.
The flow is following:
Pretrain on 2015 dataset (20 epochs, SGD+Nesterov+CosineLR, Label smoothing)
Train 5-fold CV on 2019 data, IDRID and MESSIDOR combined (75 epochs, RAdam, CosineLR, Label smoothing)
Every training stage was made of three substages:
- Freeze model body and head combiner, train for 5 epochs.
- Unfreeze body, train N epochs.
- Freeze everything, unfreeze head combiner, train for 5 epochs.
If we trained everything together, we've  always got uneven training of heads.
Also, we added random noise in range (-0.3, 0.3) to labels for regression, that greatly reduced overfitting.
Hardware
We used server from FastGPU.net with 4xV100, that greatly reduced our experiment cycle length.
Ensembling
We picked model by their performance on CV and holdout and did not orient on LB at all.
Our best performing solution was an ensemble of 20 models (4 models x 5 folds) with 10xTTA (hflip, vflip, transpose, rotate, zoom) combined with 0.25-trimmed mean. This ensemble performed worse both on holdout and LB, but we decided to choose with hope that it will generalize better (what it did!).