# 14th Place Solution

Congratulations and Thank You
This has been an interesting, challenging, and very educational experience for us. We would like to congratulate all the winners and everyone else who has worked so hard over the past three months on this competition. Hope that even those of you who have been burned by the shakeup at the end can come away from this competition with a lot of new things that you have learned. We certainly are.
We had lots of fun teaming up with each other. Keeping each other abreast of all that we were trying to do, and encouraging each other, especially in the last phase of this competition, has been an incredible experience for us.
We would like to thank the organizers, Bengali.ai and Kaggle, for putting this competition together. We hope that the results of this competition can be used in a meaningful and productive way for the advancement of the handwritten Bengali language.
Chris and Bojan would like to thank Nvidia for its ongoing generous support in terms of time and resources.
Frameworks
For our final solution, we have used both PyTorch and TensorFlow. We have also worked with Rapids on an alternative solution, which showed lots of promise but was not possible to incorporate with our best solution in the allotted submission time limits.
Datasets
Most of our models used just the competition dataset. One of our models (EfficientNet B7) trained partially with some of the external data. We had also tried using synthetic data, but that did not help our local models. It is possible that it did better on the private dataset, but we have not made any submissions with it yet.
Image Resolutions
Like everyone else, throughout the competition, we had experimented with different resolutions and aspect ratios. Our final solution was a combination of the following different resolutions: 128x128, 224x224, 137x236, 128x256. Our single best model used the 256x256 resolution, but the inference with it was taking too long, so we had to make a difficult decision to leave it out of the final ensemble in favor of a more diverse blend.
Validation Scheme
Our main validation scheme has been a 90/10 train/validation split. For most of the models, we have used the same 90/10 split, but in the latter stages of the competition, we had added a few with different splits for the sake of diversity.
Network Architectures
We have tried several different architectures and discovered that for the cutmix augmentation as we had implemented it, bigger networks were better, but they required a longer time to train and larger batch sizes. All but one network were trained with a single head. In hindsight, this might have seriously limited our post-processing as we had implemented it. Our final blend consisted of the following seven network architectures:
EfficientNet B4
EfficentNet B6
EfficientNet B7
DenseNet 201
SeNet 154
2 SE-ResNext 101
Augmentations
Two main augmentations that we used were CutMix and CoarseDropout. We had also used a mild shift, scale, rotate augmentations. We also used CAM CutMix, Chris describes it
[here](https://www.kaggle.com/c/bengaliai-cv19/discussion/136021)
.
Training Regiment(s)
Our main training consisted of one cycle with starting lr of 0.0001, and a decrease at the plateau. We would train between 100 and 200 epochs, with batch sizes between 150 and 300. After the one main cycle, we would retrain for another 70 epochs, which would usually bump our validation for another 0.001. Depending on the machine, network, and batch sizes, one epoch would take between 10 and 30 minutes to train. Our single best network achieved 0.9980 for local validation, and 0.9892 on public leaderboard.
Ensembling
We’ve tried to build some advanced second level models, but our best ensembling ended up being just a weighted average of our models. Since our models were trained to predict logits and not probabilities, finding a good blend was tricky. In the end, we managed to get a pretty decent set of weights, that was the best for both oof predictions as well as the public leaderboard.
Post Processing
This was one of the most important steps in our model, and most of the models that finished at the top in this competition. It prevented us from dropping like a stone after the final shakeup. In a nutshell, it turns out that the recall metric is very sensitive to the distribution of various classes. In order to exploit this insight, it was necessary to scale various predictions in such a way that the probabilities of less frequent items got proportionally higher weight. The most mathematically reasonable way of doing this is to multiply various probabilities by their inverse frequencies. In practice, we had to deviate slightly from the perfect inverse law. We had spent a lot of time in the last days trying to improve this post processing, but in the end, a very simple set of exponent worked best for our models. You can read more about this and other aspects of our solution in
[Chris Deotte](https://www.kaggle.com/cdeotte)
’s
[wonderful writeup](https://www.kaggle.com/c/bengaliai-cv19/discussion/136021)
.
Hardware
The main workhorses for our model building have been two Nvidia DGX Stations which have been generously allocated by Nvidia for the purposes of Kaggle competitions. We have also relied on our own private resources, including a dual RTX Titan workstation, and several 2080 Ti workstations.