# 1st place solution summary

Thanks to APTOS and Kaggle for hosting this interesting competition. 
I also would like to thank those who generously contributed in the kernels and discussions in this competition, and to the top teams that shared solutions and findings in the 2015 competition. Those findings and solutions have greatly impacted my strategy.
Validation Strategy
One of the most popular topics in every competition are proper validation strategies. During the early stage, I tried using 2015 data (both train and test) as train set, and 2019 train data as validation set. Unfortunately, the validation results and public LB were very different, I was not able to make their performance correlate well. In some discussions and kernels, other participants were also reporting inconsistent performance between CV and LB. Because of this, I did not know how to move forward, so I shifted to some other competitions for a few weeks. When I came back to this competition, I made the decision to combine the whole 2015 and 2019 data as train set, and solely relied on public LB for validation.
Preprocessing
I don't think it's necessary to preprocess images to help with the modelling, the image qualities are perfect as input for deep neural networks. So, no special preprocessing, just plain resizing.
Models and Input sizes
My final submission was a simple average of the following eight models. Inceptions and ResNets usually blend well. If I could have two more weeks I would definitely add some EfficientNets.
2 x inception_resnet_v2, input size 512
2 x inception_v4, input size 512
2 x seresnext50, input size 512
2 x seresnext101, input size 384
The input size was mainly determined by observations in the 2015 competition that larger input size brought better performance. Even though I did not find a lot beneficial to go beyond 384 based on the public LB feedback, I still push to the extreme the input size because the private set might benefit.
Loss, Augmentations, Pooling
I used only
nn.SmoothL1Loss()
as the loss function. Other loss functions may work well too. I sticked to this single loss just to simplify the emsembling process.
For augmentations, the following were helpful
contrast_range=0.2,
brightness_range=20.,
hue_range=10.,
saturation_range=20.,
blur_and_sharpen=True,
rotate_range=180.,
scale_range=0.2,
shear_range=0.2,
shift_range=0.2,
do_mirror=True,
For the last pooling layer, I found the generalized mean pooling (
[assets/20d3245b562c6db41b467cbf.pdf](assets/20d3245b562c6db41b467cbf.pdf)
) better than the original average pooling. Code copied from
[https://github.com/filipradenovic/cnnimageretrieval-pytorch](https://github.com/filipradenovic/cnnimageretrieval-pytorch)
.
from torch.nn.parameter import Parameter
def gem(x, p=3, eps=1e-6):
    return F.avg_pool2d(x.clamp(min=eps).pow(p), (x.size(-2), x.size(-1))).pow(1./p)
class GeM(nn.Module):
    def __init__(self, p=3, eps=1e-6):
        super(GeM,self).__init__()
        self.p = Parameter(torch.ones(1)*p)
        self.eps = eps
    def forward(self, x):
        return gem(x, p=self.p, eps=self.eps)
def __repr__(self):
        return self.__class__.__name__ + '(' + 'p=' + '{:.4f}'.format(self.p.data.tolist()[0]) + ', ' + 'eps=' + str(self.eps) + ')'
model = se_resnet50(num_classes=1000, pretrained='imagenet')
model.avg_pool = GeM()
Training and Testing
The training process can be divided into two stages. In the first stage, I routinely trained the eight models and validated each of them on the public LB. To get more stable results, models were evaluated in pairs (with different seeds), that's why I have 2x for each type of model. When probing LB, I tried to reduce the degree of freedom of hyperparemeters to alleviate overfitting, for example, to determine the best number of epochs for training I used a step size of five. The following are the optimized results after stage1 training:
inception_resnet_v2   public: 0.831 private: 0.927
inception_v4            public: 0.826 private: 0.924
seresnext50            public: 0.826 private: 0.931
seresnext101          public: 0.819 private: 0.923   (the 2nd best result, missing the best)
ensemble                public: 0.844 private: 0.934
In the second stage of training, I added pseudo-labelled (soft version) public test data and two additional external data - the Idrid and the Messidor dataset, to the stage1 trainset. The labels of Idrid also have five levels, to mitigate the labeling bias (my guess), the Idrid labels were smoothed by averaging the provided labels with the predicted labels from stage1 models. For the Messidor dataset which we only have four levels, I grouped the stage1 predicted soft labels by the provided groundtruth labels, calculated the mean of each group, and bounded the outliers using the mean values. For example, if the mean value of a group is 2.2, and within the same group an image has stage1 prediction of 1.1, then the label is adjusted to 2.2-0.5=1.7. After the preparation of all the labels and data, each stage1 model were trained for 10 more epoch. Finally, the LB improved from public:0.850 and private:0.935. Hours before the deadline, I made the last shot by changing the qwk thresholds from [0.5, 1.5, 2.5, 3.5] to [0.7, 1.5, 2.5, 3.5] and private improved to 0.936 ...