Congratulations to all participants. The main goal for me in this competition was checking the effective way of using new optimizes like RAdam and Over9000. Despite those optimizers are promised to work much better than Adam in corresponding studies, my initial tests of these optimizers were quite disappointing: RAdam gave about the same result as Adam while Over9000 worked much worse. In the last competition on Cloud Segmentation I participated in gave quite interesting observation: if RAdam is used there must be no warm-up in cosine annealing. Over9000 worked even better with such scheduler than RAdam as expected from here. With fast.ai it can be done with the following additional arguments learn.fit_one_cycle(36, max_lr=slice(0.2e-2,1e-2), pct_start=0.0, div_factor=100). So, this competition has provided quite small dataset and fast training to experiment with optimizes and schedulers.
In this competition I used 3 models:\n1) Resnet20 from CIFAR-pretrained-models (and the corresponding dataset) on 32x32x1 images (I have replaced first conv summing corresponding pretrained weights).\n2,3) Pretrained DenseNet121 and DenseNet169 on 64x64x1 images (because of the first layer with stride 2 conv followed by 3x3 pooling it is really helpful to upscale small images).
\n\nIn all models ReLU is replaced by Mish. The head is similar to one used by fast.ai by default: Concat pooling + Mish + BN + Dropout(0.5) + Linear + Mish + BN + Dropout(0.5) + Linear.
\n\nOther things used:\n- Over9000 one cycle without warm-up scheduler\n- Categorical Cross Entropy\n- gradient clipping\n- best model selection\n- discriminative lr with split of the model into backbone and head\n- standard fast.ai augmentation without flip get_transforms(do_flip=False,max_zoom=1.2)\n- 5 fold CV and ignore public LB
Things that could work:\n- PL could easily boost the private LB score to 0.995+\n- More models
\n\nThings that didn't work:\n- ArcFace loss (I expected that it will make the class representation more compact and robust to train/test data mismatch)\n- Backbone freezing and training the head first (even without replacement of ReLU by Mish)\n- ResNet44 and ResNet56 from CIFAR-pretrained-models worked worse than ResNet20\n- use Dig-MNIST
\n\nRegarding train/test mismatch: I expect that each person has provided several handwritten digits. The leak happens when digits written by the same person are split between train and val. I tried to assume that each person has provided one or several (fixed number) instances of digits. It didn't work for me because most likely people has provided the different number of handwritten examples, and everything still remains mixed.\nRAdam and Over9000 could be added to your kernels with this utility script.
", "comments": [ { "id": 702036, "author_name": "Iafoss", "author_url": "", "post_date": "2019-12-24T07:32:16.787000", "content": "I have posted a kernel for Handwritten Grapheme Classification competition that implements most of the points mentioned here.
", "votes": 2, "replies": [] }, { "id": 701474, "author_name": "MichaelP", "author_url": "", "post_date": "2019-12-23T14:21:52.760000", "content": "Thanks for sharing. Your approach sounds pretty different to a lot of the other ones described. Well done!
", "votes": 0, "replies": [] }, { "id": 700617, "author_name": "corochann", "author_url": "", "post_date": "2019-12-22T10:06:35.600000", "content": "Congrats! and thaks for sharing.
", "votes": 0, "replies": [] }, { "id": 699145, "author_name": "", "author_url": "", "post_date": "2019-12-20T05:05:47.300000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 699133, "postDate": "2019-12-20T04:44:53.293Z", "content": "Congratulations to all participants. The main goal for me in this competition was checking the effective way of using new optimizes like RAdam and Over9000. Despite those optimizers are promised to work much better than Adam in corresponding studies, my initial tests of these optimizers were quite disappointing: RAdam gave about the same result as Adam while Over9000 worked much worse. In the last competition on Cloud Segmentation I participated in gave quite interesting observation: if RAdam is used there must be no warm-up in cosine annealing. Over9000 worked even better with such scheduler than RAdam as expected from here. With fast.ai it can be done with the following additional arguments learn.fit_one_cycle(36, max_lr=slice(0.2e-2,1e-2), pct_start=0.0, div_factor=100). So, this competition has provided quite small dataset and fast training to experiment with optimizes and schedulers.
In this competition I used 3 models:\n1) Resnet20 from CIFAR-pretrained-models (and the corresponding dataset) on 32x32x1 images (I have replaced first conv summing corresponding pretrained weights).\n2,3) Pretrained DenseNet121 and DenseNet169 on 64x64x1 images (because of the first layer with stride 2 conv followed by 3x3 pooling it is really helpful to upscale small images).
\n\nIn all models ReLU is replaced by Mish. The head is similar to one used by fast.ai by default: Concat pooling + Mish + BN + Dropout(0.5) + Linear + Mish + BN + Dropout(0.5) + Linear.
\n\nOther things used:\n- Over9000 one cycle without warm-up scheduler\n- Categorical Cross Entropy\n- gradient clipping\n- best model selection\n- discriminative lr with split of the model into backbone and head\n- standard fast.ai augmentation without flip get_transforms(do_flip=False,max_zoom=1.2)\n- 5 fold CV and ignore public LB
Things that could work:\n- PL could easily boost the private LB score to 0.995+\n- More models
\n\nThings that didn't work:\n- ArcFace loss (I expected that it will make the class representation more compact and robust to train/test data mismatch)\n- Backbone freezing and training the head first (even without replacement of ReLU by Mish)\n- ResNet44 and ResNet56 from CIFAR-pretrained-models worked worse than ResNet20\n- use Dig-MNIST
\n\nRegarding train/test mismatch: I expect that each person has provided several handwritten digits. The leak happens when digits written by the same person are split between train and val. I tried to assume that each person has provided one or several (fixed number) instances of digits. It didn't work for me because most likely people has provided the different number of handwritten examples, and everything still remains mixed.\nRAdam and Over9000 could be added to your kernels with this utility script.
", "rawMarkdown": "Congratulations to all participants. The main goal for me in this competition was checking the effective way of using new optimizes like [RAdam](https://arxiv.org/pdf/1908.03265.pdf) and [Over9000](https://github.com/mgrankin/over9000). Despite those optimizers are promised to work much better than Adam in corresponding studies, my initial tests of these optimizers were quite disappointing: RAdam gave about the same result as Adam while Over9000 worked much worse. In the last competition on Cloud Segmentation I participated in gave quite interesting observation: **if RAdam is used there must be no warm-up in cosine annealing**. Over9000 worked even better with such scheduler than RAdam as expected from [here](https://github.com/mgrankin/over9000). With fast.ai it can be done with the following additional arguments `learn.fit_one_cycle(36, max_lr=slice(0.2e-2,1e-2), pct_start=0.0, div_factor=100)`. So, this competition has provided quite small dataset and fast training to experiment with optimizes and schedulers.\n\nIn this competition I used 3 models:\n1) **Resnet20** from [CIFAR-pretrained-models](https://github.com/chenyaofo/CIFAR-pretrained-models) (and the corresponding [dataset](https://www.kaggle.com/iafoss/cifarpretrainedmodels)) on 32x32x1 images (I have replaced first conv summing corresponding pretrained weights).\n2,3) Pretrained **DenseNet121** and **DenseNet169** on 64x64x1 images (because of the first layer with stride 2 conv followed by 3x3 pooling it is really helpful to upscale small images).\n\nIn all models ReLU is replaced by **Mish**. The head is similar to one used by fast.ai by default: Concat pooling + Mish + BN + Dropout(0.5) + Linear + Mish + BN + Dropout(0.5) + Linear.\n\n\n**Other things used**:\n- **Over9000 one cycle without warm-up scheduler**\n- Categorical Cross Entropy\n- gradient clipping\n- best model selection\n- discriminative lr with split of the model into backbone and head\n- standard fast.ai augmentation without flip `get_transforms(do_flip=False,max_zoom=1.2)`\n- 5 fold CV and ignore public LB\n\n\n**Things that could work**:\n- PL could easily boost the private LB score to 0.995+\n- More models\n\n\n**Things that didn't work**:\n- ArcFace loss (I expected that it will make the class representation more compact and robust to train/test data mismatch)\n- Backbone freezing and training the head first (even without replacement of ReLU by Mish)\n- ResNet44 and ResNet56 from CIFAR-pretrained-models worked worse than ResNet20\n- use Dig-MNIST\n\nRegarding train/test mismatch: I expect that each person has provided several handwritten digits. The leak happens when digits written by the same person are split between train and val. I tried to assume that each person has provided one or several (fixed number) instances of digits. It didn't work for me because most likely people has provided the different number of handwritten examples, and everything still remains mixed.\nRAdam and Over9000 could be added to your kernels with this [utility script](https://www.kaggle.com/iafoss/radam).", "votes": 33 }, { "id": 702036, "postDate": "2019-12-24T07:32:16.787Z", "content": "I have posted a kernel for Handwritten Grapheme Classification competition that implements most of the points mentioned here.
", "rawMarkdown": "I have posted a [kernel](https://www.kaggle.com/iafoss/grapheme-fast-ai-starter-0-960-lb) for Handwritten Grapheme Classification competition that implements most of the points mentioned here.", "votes": 2 }, { "id": 701474, "postDate": "2019-12-23T14:21:52.760Z", "content": "Thanks for sharing. Your approach sounds pretty different to a lot of the other ones described. Well done!
", "rawMarkdown": "Thanks for sharing. Your approach sounds pretty different to a lot of the other ones described. Well done!" }, { "id": 700617, "postDate": "2019-12-22T10:06:35.600Z", "content": "Congrats! and thaks for sharing.
", "rawMarkdown": "Congrats! and thaks for sharing." }, { "id": 699145, "postDate": "2019-12-20T05:05:47.300Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T11:34:45Z", "competition_id": 16017, "competition_slug": "Kannada-MNIST", "team_id": 3718338, "solution_url": "/competitions/Kannada-MNIST/writeups/iafoss-4-th-place-solution-over9000-optimizer" }