# [RECAP] 41st place solution with code

Hello everyone!
Story
I've joined this competition in the first days of January, with the purpose of learning computer vision from both theoretical and practical sides. I thought that it would be a good idea to finally learn how to write a dl pipeline using config files. I also wanted to dive deeper into Catalyst framework, so Catalyst Config API seemed to fit perfectly.
For the first two weeks, I couldn't get a score higher than 0.955 whilst there were 0.96+ kernels. 
I'd thought that picking a dataset with cropped images is a good idea since I didn't want the model to memorize the position of the character to not overfit on public lb.
Following pieces of advice on forum, I started training with mixup and cutmix. At that moment there was no Cutmix implementation in Catalyst and Mixup implementation didn't support multi-output.
So, I adopted code from forum and wrote MixupCutmixCallback, which later resulted in PR with Cutmix implementation to Catalyst library. It was my first time contributing something to open-source deep learning project.
Mixup + Cutmix had given some improvement and I finally crossed 0.955 border. The next step was to go beyond high scoring public kernels (the best one had a score of 0.966 at that time). I tried different architectures (primarily sticked to Efficient Net series), optimizers, schedulers, lrs, losses, augmentations, but 0.966 was still far from me. I got very close to it after I added finetuning on 224x224 images (the first stage was on 128x128 cropped images).
On the very next day after I got close to 0.966 kernel a new 0.97 kernel came out, so I had to do the climb all over again and at that time I'd tried tweaking almost every possible thing in my pipeline. I
[also](https://www.kaggle.com/c/bengaliai-cv19/discussion/131358)
felt like I was doing some basic thing very wrong, but couldn't identify which exactly. I thought the problem was in misunderstanding some concept in Catalyst, so I decided to rewrite my whole pipeline to Pytorch. This is definitely not something you want to do taking into account there were 3.5 weeks before the end of the competition. Before I started rewriting my pipeline to Pytorch, I decided to train on uncropped images with Catalyst. On the next day I'd finished my Pytorch pipeline and decided to look at the results of training with uncropped images: it was the first time my CV crossed 0.97, resulting in 0.968 LB. Of course, I decided to stay with Catalyst😄 . I took a deeper model and finally beat the 0.97 kernel.
Again, I was out of ideas and again I went on the forum. I skimmed through the main topics and even went through all the posts written by top-scorers individually. I decided to implement the tricks from
[this](https://arxiv.org/abs/1912.11370)
paper posted by
[@drhabib](/drhabib)
, but none of them seem to work.
I started reading other papers, which had been posted on the forum and decided to improve my augmentations. I couldn't fit the geometric ones and thought to tweak
alpha
in Mixup and Cutmix. I decided to go aggressive with Mixup and set
alpha = 4
, which I think is pretty reasonable as there are unseen graphemes in the test set.
In the last week, I'd switched to original images, trained 5-fold B3 and added 3-epoch finetuning with no augmentations, which yielded 0.985. I had a pretty small CV/LB gap and decided to choose my highest scoring model.
Configuration
Efnet B3 with weights from Noisy Student
One model with three heads
opt_level O2
137x236 images
CrossEntropyLoss. I guess I could spend some more time tweaking OHEM, it should give a better result.
Mixup (alpha = 4) + Cutmix (alpha = 1), each with probability of 0.5
loss weights: 7, 1, 2 respectively
100 epochs
AdamW optimizer
OneCycleSchduler
finetuning 3 epochs with no augs also on 137x236
You can find my complete config files in the link to my repo provided below.
Forum
Thanks, to the people who shared actively on the forum. Namely:
[@drhabib](/drhabib)
,
[@pestipeti](/pestipeti)
,
[@hengck23](/hengck23)
,
[@machinelp](/machinelp)
,
[@haqishen](/haqishen)
. You are not just giving hints, but implicitly encourage others to participate in discussions and share as well, as a form of expressing gratitude for what we have learned from your posts and kernels (poor wording, but I hope you get the idea😅 ).
The other important thing related to the forum is I think in this competition many people trapped themselves into hyper-parameters tweaking (myself included). Most posts for this comp were about architecture, number of epochs, augmentations, and even optimizers. I guess some of us can relate that they were jumping back and forth between parameter options after some high-ranked users posted some information about his/her pipeline.
A very few people were discussing validation and strategies to deal with unseen graphemes.
The challenge here is the more information we have the harder to find the truth. So, it is very crucial to focus on things that matter.
Takeaways
Lab book to track your experiments
Cam activations to better understand what is going on with you model thx
[@cdeotte](/cdeotte)
Better error-analysis in general
Data preprocessing (resize, file extension, etc) which allow you to conduct more experiments
Use forum wisely
Don't be afraid to ask questions on forum. The worst question is the question that hasn't' been asked.
And congratulations to everyone who had taken part in this competition regardless of their score!
You can find my code here:
[https://github.com/LightnessOfBeing/kaggle-bengali-classification](https://github.com/LightnessOfBeing/kaggle-bengali-classification)