# 13th place solution

Congrats all the prize winner and everyone finished in gold zone! And I really want to say thank you to my great teammate
[@mhiro2](/mhiro2)
and
[@yasufuminakama](/yasufuminakama)
, I'm pretty confident that I couldn't get gold medal if I'm alone...
This is my first time to write solution, but I'll try to open up what we did as much as possible.
OVERVIEW
ensemble 4 models, 2 SeResNext101 and 2 EfficientNet B5
2 kinds of pseudo labeling
use old competition data
CHALENGE
severe leakage with image size
difference between train / test distribution
unstable metric, QWK
DETAIL
preprocess
The most important part of this competition is preprocess. As everyone realized, this competition had a severe leakage with image size and shape. At first I tried many kinds of preprocess (ex. zoom, circle crop or corner cut etc.) to eliminate
this leakage, but almost all failed. When I believed train images and test images looked very similar, that makes LB score significantly dropped. Only rotate worked good for me, but it was very unstable.(ex I used 120 degree. If I changed to 90 or 180, LB score significantly dropped.) 
And the most interesting image shape was (480,640), which consist of over 70% of test set. That makes me come up with below hypothesis: 
After admin made the dataset, they realized there was strong correlation between image size/shape and diagnosis score. They want to prevent all kagglers from submitting LGBM model which use only image meta features. But they also want to the model which can detect retinopathy, so they decided to crop only some part of public test set....
That means the image size and shape information is still useful, so I stopped focusing on preprocess. I'm now confident that this thoughts was somehow right after checking private score because it's incredibly higher than previous competition. And you can see that the evidence of this in
[this great kernel](https://www.kaggle.com/taindow/be-careful-what-you-train-on)
's private score.
What we used are below. I believe this difference between preprocess might diversify our models.
```python
for SeResNext
Compose([
        BensCrop(img_size), #from 2015 solution
        RandomHorizontalFlip(),
        RandomVerticalFlip(),
        RandomRotation((-120, 120)),
        ),
    ])
for EfficientNet B5
Compose([
        Resize(img_size),
        HorizontalFlip(),
        VerticalFlip(),
        Rotate(),
        RandomContrast(0.5),
        IAAAdditiveGaussianNoise(p=0.25),
    ])
```
dataset
The 2nd important part of our result was how to split and select dataset. I tried many combination, and finally I found below setting works well for me.
1. split current train dataset to 5 folds by stratified k fold.
2. train on 4/5 train set and previous competition train set
3. validate 1/5 train set
When I pre-trained on 2015 dataset and fine-tuned on 2019 dataset, my single model(SeResNext)'s score was LB0.811, but after I changed to above method, my score was boost to LB0.827.
model and training method
We used 2 kinds of model, SeResNext101 and EfficientNet B5. Each models used a differnt training method.
For SeResNext101
- image size:320
- loss:RMSE
- train on 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5 of 2019 dataset)
For EfficientNet B5
- image size:256 -> 320
- loss:MSE
- 2 step training
    1. pre-train on 4/5folds of 2015 dataset (validate with 2019 dataset) (img_size=256)
    2. fine-tune on 1/5fold of 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5fold)(img_size=320)
Before starting ensemble, SeResNext scored LB0.827 and EfficientNet scored LB0.830.
pseudo labeling and ensemble
We used 2 kinds of pseudo labeling.
1. make soft label for test set and re-train from scratch(ImageNet pre-trained model).
2. make hard label for test set and fine-tune from trained model
Both method boost LB score a lot, so we decided used both method to diversify our models. Sometimes we also use external data (Messidor and IEEE) to give some fresh information to our model, but I'm not sure it's necessary or not.
By combining pseudo labeling and ensemble, our both single models reach LB0.846. Below is the result of what we did.(I know it's not beautiful, but we had no time to go back and start from 1st stage model...)
![](assets/9eee21d91870a057e2c9d91f.png)
(I deleted the row which didn't contribute to our final model)
Finally we submitted below 2 models.
- No.2+No.5+No.18+No.19(public0.930)
- No.16(public0.930)
UPDATED
Here is my single model's scores that shows how pseudo labeling was effective on private dataset.
model: SeResNext101
- before applied pseudo labeling:
private LB 0.922
- pseudo labeling with only SeResNext101's labels:
private LB 0.925
- pseudo labeling with both SeResNext101 and EfficientNet's labels:
private LB 0.931
Interestingly last model is better than our final submission and enough to win gold medal :)
Thank you for reading, please feel free to ask me if you have a question!