# 38th Place - ZoneMix - A domain specific augmentation

A really big thanks to all the amazing participants for sharing the ideas. @haqishen @hengck23  @bibek777 @machinelp @quandapro and so many other awesome users without whose suggestions I would have been completely lost. And also big thanks to Bengali.AI organizers for this competition about my native language, the lessons learned from this competition will also be applicable to other languages of the Indian subcontinent like Hindi, Odia, Telegu, etc.
I'll be honest, I joined the competition by at nearly the end of January, and for most of the time, I was trying to improve the score just by simple augmentations/model/hyperparameter tuning to improve the score. And having only Google Colab to train doesn't help with that either. Had I continued on that track I surely wouldn't have been in any medal zone. Thanks a lot to my teammate Ram for pointing out the important discusions about unseen graphemes by the organizers. And subsequently Qishen Ha's post on validating on unseen graphemes.
With only 10 days remaining after realizing how poorly the model will do on unseen graphemes. Particularly - Classes with few combinations - Which is more than half of the grapheme roots, and consonant 3 and 6 - Generalize the worst.  (Check combination counting kernel
[here](https://www.kaggle.com/dipamc77/bengali-dataset-grapheme-combination-counts?scriptVersionId=30348992)
)
I probed for the scores of some of these classes similar to the method described
[here](https://www.kaggle.com/philippsinger/checking-r-c-v-scores)
. From the gaps, I was sure that a huge shakeup was coming.
After reading the top solutions I'm inspired to pursue ML-based  methods to improve generalization rather than a rule-based method like I describe below; I would still like to share it
ZoneMix - Splitting graphemes by zones to apply cutmix
As a native Bengali speaker, I can say that generally graphemes can be split into particular zones based on the combinations. Below is a diagram of the zones.
![](assets/541308bf20336d5cc9d4baed.png)
Some extra rules have to be made for some combinations, which was kind of a pain to write in code and made me question why I'm even working on such a tedious idea 😭 . I'll share the code surely, just right now its a huge mess.
Each combination takes up some of these zones and cutting by the zone's coordinates based on the character's bounding box can roughly give the components.  Here is an example
![](assets/8e7b52e0a33fa348cbbcc2bc.png)
Of course - Real-world data doesn't always follow rules and failure cases are common
![](assets/0f3ca27d870630784b5594ed.png)
Once the zone data is extracted, they can be placed on the other images by resizing based on the new zone coordinates. This can produce very realistic images, but also unrealistic ones.
![](assets/cd53779c7315ff18a6b14efa.png)
![](assets/cef406563f6baab45a6be636.png)
When all the setup was done I had about 3 days left to train with this method. And sadly it didn't work directly, I think its because the mistakes and unrealistic images are too much noise for the model to deal with.
I was able to tone down the noise by doing the following tricks - The grapheme root zones are roughly correct most of the time, while the vowel and consonants are way off. I figured that since grapheme root and consonant 3 and 6 are the main classes to tackle, I should focus on those.
Using zonemix I trained a model to only predict the grapheme root, this reduced the noisy gradients from the vowel and consonants and improved the recall as well as unseen recall.
Consonant 3 and 6 have a particular property that they are basically a combination of Consonant 2 and 4 and Consonant 4 and 5 respectively. Dieter also explains this in his
[solution post](https://www.kaggle.com/c/bengaliai-cv19/discussion/136129)
and I feel he found a much better way to deal with it. Anyway, so my approach was to use zonemix to place consonant 4 onto images with consonant 2 and 5. Examples below. With these I trained a new model for consonants.
I literally trained both models parallely on two colab sessions on the last day and didn't even have time to tune anything, had to manually stop the runs because I had only 3 hours left to the end of the competition.
Final solution
- Zonemix models for grapheme roots with less combinations (<5), and consonant 3, 6, along the old trained ensemble for the rest (A pretty bad one with public LB only 0.982 and private LB 0.931). But I confidently picked the zonemix kernel knowing it will surely be good on private set 😄
![](assets/baed6cebef4ad8176a777297.png)
Maybe with some better tuning and validation it could have probably done much better.
Hyperparameters tuning that worked and didn't work for me
Wasted weeks on cropped and preprocessed images, didn't work at all. I'm still not convinced why.
Did lots of experiments on resnet34 with 128x128 (just resize) - This was practical for fast experimentation on Google Colab.
Balanced sampling - The metric is macro average, so helped to do balanced sampling I guess.
Cutout/Cutmix - Gives better results than mixup.
Interestingly Mixup + Gridmask gives much better results on Unseen CV
Finally trained SeResNeXt50 with 128x128 which gave improvement on CV (Though same pathetic results on Unseen CV)
Well tuned ReduceLRonPlateau converges as much as 1.5x - 2x faster than Cosine Decay. Didn't try other LR schedules.
Some ideas that I didn't get to try
CAM FMix - One of the top solution uses CAM Cutmix
Correct zone segmentation via noisy student training methods on the noisy zones.
Final thoughts and important lessons learned
From the only other competition I took part in
[Severstal: Steel Defect Detection](https://www.kaggle.com/c/severstal-steel-defect-detection)
- I had this lesson:
If there is a gap in CV and LB, trust your CV more and don't overfit LB.
I can change that lesson now - If there is a gap in CV and LB, try to understand why from the discussions, problem statement and the data. Don't dive into coding new ideas trying to reduce the gap without having a proper understanding of what is causing the gap.
Try ML-based approaches for solving the problem, after seeing the top solutions I feel like I understand them very well, but it didn't occur to me to use these ideas. Maybe I'm biased to use rule-based ideas and that needs to change.
Do lots of EDA from time to time, not just at the beginning of the competition. Seeing the results of trained models changes our intuition about the data.
Don't ignore external datasets. Of course, working with them needs time and effort, but with correct usage, external data can work wonders.
Lastly - It is important to keep hacks to a minimum, but at the end of the day, Its a competition, do what you need to do to improve your score, like the awesome metric hacking magic.