# 18th Place Brief Solution Overview

I would like to thank Kaggle for organizing such an interesting competition. Unfortunately, I had not paid much attention to it until just a couple of weeks ago, and am regretting not spending more time on it. Especially since models for the competition took
really
long time to train.
Like many others, I decided to seriously join the competition after discovering the wonderful
[Alex Shonenkov](https://www.kaggle.com/shonenkov)
starter
[kernel](https://www.kaggle.com/shonenkov/train-inference-gpu-baseline)
. Most of my work was building upon his work, and experimenting with different network backbones, training schedules, ensembles, and to a lesser degree augmentations.
My final ensemble consists of the following three EfficientNet model architectures:
B1 - I have only trained one of these networks successfully. The best result can be found in
[this kernel](https://www.kaggle.com/tunguz/best-b1-inference/)
, and all the weights are in
[this dataset](https://www.kaggle.com/tunguz/eb1-weights)
. The best single model achieves 0.918 locally, 0.925 on public LB and 0.911 on private LB.
B2 - I have trained several of these networks, and managed to get quite a bit of an improvement over Alex's original model. However, these networks have only made a minor contribution to my final blend. The best result can be found in
[this kernel](https://www.kaggle.com/tunguz/best-b2-inference)
, and all the weights are in
[this dataset](https://www.kaggle.com/tunguz/eb2-weights/)
. The best single model achieves 0.923 locally, 0.929 on public LB and 0.916 on private LB.
B4 - These networks have been the main workhorses behind my solution. Unfortunately, they are really slow to train, and I pretty much pushed all of my compute resources to get the best solution with them. Did not have any additional capacity (compute + time) to train bigger models, which I suspect would have performed even better. The best result can be found in
[this kernel](https://www.kaggle.com/tunguz/best-b4-inference)
, and all the weights are in
[this dataset](https://www.kaggle.com/tunguz/alaska2-eb4-model-weights)
. The best single model achieves 0.930 locally, 0.940 on public LB and 0.925 on private LB.
Training Schedule
This is probably the biggest source of improvement for my networks over the base one that was found in the public kernel. I started with the same training schedule as in the original kernel, but then I
re-trained
all the weights - twice. The second time I decreased the LR to 0.0005, and increased patience to 2. I added JpegCompression to the augmentations, but otherwise did not change anything. I also increased training to 50 epochs, which took a really long time to go through, since each epoch would take about 2.5 hours on an V100 GPU.
Train/Val Split
For the most of the models I stuck to the original 80/20 split, but have used different seeds. The choice of split seems to have been very important in this competition - the final models varied between 0.933 and 0.940 on LB. I have also trained a couple of models with the 95/5 split, but since they were B2 networks, they did not contribute significantly to the final ensemble.
Final Blend
Since I've used different splits for my training, it was not possible to do a consistent local validation for the ensemble, so I was forced to rely on LB. Again, since I was only able to work on this competition for less than two weeks, that did not leave me a lot of submissions on which to base my validation strategy, so I had to rely on my intuition a lot. For instance, I did not use an equally weighted average for the blend of the same model on different splits, but based the weights on the individual model's performance on LB. It is highly likely that the weights I had finally used are suboptimal, but I feel fortunate that my two submissions were the best ones on private LB.