![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1025042%2F2986414371496927cefbe70a07a7d4b7%2Fflowchart.png?generation=1576649337493329&alt=media\")
Thanks a lot to Kaggle and @higgstachyon for hosting this competition, it's such a great experience for me to learn and acquire knowledge from other competitors, congratulations to all the winners!
\n\nCNN (Convolutional Neural Network)
\n\n1. 5-fold cross-validation, average the results (or majority vote)
\n\n2. Image augmentation:\n translation, rotation, shearing and scaling
\n\n3. CNN with Squeeze-and-Excitation Block (SE block)\n some simple explanation from my kernel
\n\n4. General training tricks\n Reduce learning rate (ReduceLROnPlateau), early stopping
\n\n5. Pseudo-labeling on test data (most effective)
\n\n6. Trick when training Dig-MNIST dataset\n As discussed in post1 and post2, you may notice that your model will get worse if you add Dig-MNIST dataset directly to your training dataset. There are some issues in Dig-MNIST dataset such as border cut-off effect, non-native user's style... which may cause poor performance to your original model.
\n\nMy baseline model(trained only with 60k training data) got ~99.8% accuracy on training set but got only ~90% accuracy on Dig-MNIST dataset. It seems difficult to generalize well on that 10% rest of images, but it also means that your model is still able to predict most of the images correctly. So I trained with Dig-MNIST by following steps:\n(1)Train a baseline model(with 60k training data) \n(2)Use model to perform pseudo labeling on test images (now you have 65k training data)\n(3)Use model to predict Dig-MNIST dataset, add the correctly classified data to the training set (says 9200 images, now you have 74200 training data)\n(4)Train a new model on new training dataset, replace baseline model with new model\n(5)Repeat steps (2) to (4), and see if there's improvement on Dig-MNIST prediction
\n\nthe flowchart is illustrated as below:\n
Pseudo-labeling and Dig-MNIST dataset training obviously improved my LB score\nfrom (private/public) : 0.9890/0.9894 to 0.99160/0.99120.
\n\nSome implementation details:\n- Optimizer: Adam\n- Epoch of each fold: maximum 200\n- Initial learning rate: 1e-3\n- Data augmentation (in pytorch): \n torchvision.transform.RandomAffine(degree=15, translate=(0.25,0.25),scale=(0.7,1.1),shear=8)\n- Model training time: One set of training for 5-fold cross validation takes about 4 hours on my PC (RTX 2080+i9-9900KF)
\n\n1. More folds cross-validation (up to 20 folds)
\n\n2. Further image augmentations:\n colorjitter (brightness, contrast... ) will make model worse, also, you might want to \n avoid vertically flip since it will cause wrong classification for similar digits such as 6 and 9.
\n\n3. Critical subclasses classification(0 and 1, 6 and 9)
\n\n4. Classify after \"Native\" classifier\n In order to deal with \"newbie effect\" mentioned in this post, before classifying the digits, I also tried to predict whether the numbers are written by native user or not. Assumes that training set are done by native user, label 1; Dig-MNIST set are done by non-native user, label 0, then train two different models separably for these two kinds of image, but this method doesn't improve in my case.
\n\n5. Several optimizers and learning rate reduce methods:\n RMSProp, Adamax, AdamW, SGD, SGDR(SGD+CosineAnnealingWarmRestarts)...
\n\n6. Several architecture of neural network\n Resnet18, Resnet34, Resnet50, SE_Resnet(SE block +residual network) ...
\n\n7. Transfer learning
\n\n8. Snapshot ensembling
\n\n1. Border cut-off effect augmentation
\n\n2. Different ensemble methods
\n\n3. Tune hyperparameters in more systematically, or in more smarter ways
\n\nMy final models were trained over ~75K images (train set 60k + test set 5K + Dig-MNIST data 10k) and got 0.9924 on the private LB. In order to avoid overfitting on the Dig-MNIST dataset, I also submitted another models which were trained without Dig-MNIST dataset and got 0.9916 on the private LB.
\n\nThanks for reading!
", "comments": [ { "id": 704302, "author_name": "Pham Bao", "author_url": "", "post_date": "2019-12-27T09:49:19.110000", "content": "That's a great approach!. May I ask you 2 questions? First, How many epochs that you train the second model and third model and more...? Second, Every model you retrain again with the new data then you perform pseudo labeling on test images, could you tell me how is the label of test images changing? I think when training model, the model will memorize the label. So that could be a problem when performing pseudo label.
", "votes": 1, "replies": [ { "id": 712483, "author_name": "ccchang", "author_url": "", "post_date": "2020-01-07T09:36:51.673000", "content": "Thank for your reply!\n1. I set maximum 200 epochs for each models, but with early stopping, most of them ended in < 70 epochs.\n2. After 5 stages of repeatedly training, there's only few changes on pseudo label compared to the first stage(about 20 labels are different), but the number of Dig-MNIST data (number N in my figure) increase about 200, much more than previous one. So I think how good can your model deal with DIG-MNIST dataset is more important than test set in this case.
", "votes": 2, "replies": [] } ] }, { "id": 698011, "author_name": "MichaelP", "author_url": "", "post_date": "2019-12-18T17:02:21.303000", "content": "Good work to get some benefit from the dig-mnist data for training. I used a function to clean the borders of the dig-mnist data, not as augmentation just as a preprocessing step. The code just set the outside n number of lines in the reshaped 28x28 image to 0. I found n=2 worked quite well. I didn't use the data for training. The preprocessing step led to an improved holdout validation score. Something like this might have helped your approach a little bit. Not that you needed much help 🙂
\n\nCongratulations.
", "votes": 1, "replies": [ { "id": 698267, "author_name": "ccchang", "author_url": "", "post_date": "2019-12-19T01:56:58.980000", "content": "Thank you! border cleaning is also an option, I didn’t do that cause I think translate and zoom in augmentation sometimes do that for me, interesting to hear that it does help improve the accuracy!\nyour works and shared information in this competition are very useful to me, congrats to you too!
", "votes": 1, "replies": [] }, { "id": 701471, "author_name": "MichaelP", "author_url": "", "post_date": "2019-12-23T14:19:54.643000", "content": "Thanks
", "votes": 0, "replies": [] } ] }, { "id": 712857, "author_name": "Jainmiah Shaik", "author_url": "", "post_date": "2020-01-07T16:36:17.610000", "content": "Very clear explanation. Can I have the entire code to learn?
", "votes": 0, "replies": [] }, { "id": 700619, "author_name": "corochann", "author_url": "", "post_date": "2019-12-22T10:09:22.443000", "content": "Congrats and thanks for sharing.
", "votes": 0, "replies": [ { "id": 701457, "author_name": "ccchang", "author_url": "", "post_date": "2019-12-23T14:07:35.353000", "content": "Thanks you!
", "votes": 0, "replies": [] } ] }, { "id": 697757, "author_name": "", "author_url": "", "post_date": "2019-12-18T11:05:21.963000", "content": "", "votes": 0, "replies": [ { "id": 697805, "author_name": "ccchang", "author_url": "", "post_date": "2019-12-18T12:47:35.887000", "content": "Thank you for the comment!
", "votes": 0, "replies": [] } ] } ], "comment_count": 10, "messages": [ { "id": 697607, "postDate": "2019-12-18T06:38:12.797Z", "content": "Thanks a lot to Kaggle and @higgstachyon for hosting this competition, it's such a great experience for me to learn and acquire knowledge from other competitors, congratulations to all the winners!
\n\nCNN (Convolutional Neural Network)
\n\n1. 5-fold cross-validation, average the results (or majority vote)
\n\n2. Image augmentation:\n translation, rotation, shearing and scaling
\n\n3. CNN with Squeeze-and-Excitation Block (SE block)\n some simple explanation from my kernel
\n\n4. General training tricks\n Reduce learning rate (ReduceLROnPlateau), early stopping
\n\n5. Pseudo-labeling on test data (most effective)
\n\n6. Trick when training Dig-MNIST dataset\n As discussed in post1 and post2, you may notice that your model will get worse if you add Dig-MNIST dataset directly to your training dataset. There are some issues in Dig-MNIST dataset such as border cut-off effect, non-native user's style... which may cause poor performance to your original model.
\n\nMy baseline model(trained only with 60k training data) got ~99.8% accuracy on training set but got only ~90% accuracy on Dig-MNIST dataset. It seems difficult to generalize well on that 10% rest of images, but it also means that your model is still able to predict most of the images correctly. So I trained with Dig-MNIST by following steps:\n(1)Train a baseline model(with 60k training data) \n(2)Use model to perform pseudo labeling on test images (now you have 65k training data)\n(3)Use model to predict Dig-MNIST dataset, add the correctly classified data to the training set (says 9200 images, now you have 74200 training data)\n(4)Train a new model on new training dataset, replace baseline model with new model\n(5)Repeat steps (2) to (4), and see if there's improvement on Dig-MNIST prediction
\n\nthe flowchart is illustrated as below:\n
Pseudo-labeling and Dig-MNIST dataset training obviously improved my LB score\nfrom (private/public) : 0.9890/0.9894 to 0.99160/0.99120.
\n\nSome implementation details:\n- Optimizer: Adam\n- Epoch of each fold: maximum 200\n- Initial learning rate: 1e-3\n- Data augmentation (in pytorch): \n torchvision.transform.RandomAffine(degree=15, translate=(0.25,0.25),scale=(0.7,1.1),shear=8)\n- Model training time: One set of training for 5-fold cross validation takes about 4 hours on my PC (RTX 2080+i9-9900KF)
\n\n1. More folds cross-validation (up to 20 folds)
\n\n2. Further image augmentations:\n colorjitter (brightness, contrast... ) will make model worse, also, you might want to \n avoid vertically flip since it will cause wrong classification for similar digits such as 6 and 9.
\n\n3. Critical subclasses classification(0 and 1, 6 and 9)
\n\n4. Classify after \"Native\" classifier\n In order to deal with \"newbie effect\" mentioned in this post, before classifying the digits, I also tried to predict whether the numbers are written by native user or not. Assumes that training set are done by native user, label 1; Dig-MNIST set are done by non-native user, label 0, then train two different models separably for these two kinds of image, but this method doesn't improve in my case.
\n\n5. Several optimizers and learning rate reduce methods:\n RMSProp, Adamax, AdamW, SGD, SGDR(SGD+CosineAnnealingWarmRestarts)...
\n\n6. Several architecture of neural network\n Resnet18, Resnet34, Resnet50, SE_Resnet(SE block +residual network) ...
\n\n7. Transfer learning
\n\n8. Snapshot ensembling
\n\n1. Border cut-off effect augmentation
\n\n2. Different ensemble methods
\n\n3. Tune hyperparameters in more systematically, or in more smarter ways
\n\nMy final models were trained over ~75K images (train set 60k + test set 5K + Dig-MNIST data 10k) and got 0.9924 on the private LB. In order to avoid overfitting on the Dig-MNIST dataset, I also submitted another models which were trained without Dig-MNIST dataset and got 0.9916 on the private LB.
\n\nThanks for reading!
", "rawMarkdown": "Thanks a lot to Kaggle and @higgstachyon for hosting this competition, it's such a great experience for me to learn and acquire knowledge from other competitors, congratulations to all the winners!\n\n## The overall approach:\n**CNN (Convolutional Neural Network)**\n\n## What worked for me:\n**1. 5-fold cross-validation, average the results (or majority vote)**\n\n**2. Image augmentation:**\n translation, rotation, shearing and scaling\n\n**3. CNN with Squeeze-and-Excitation Block (SE block)**\n some simple explanation from my [kernel](https://www.kaggle.com/ccchang801023/se-net-my-top1-baseline-model-with-pytorch)\n\n**4. General training tricks**\n Reduce learning rate (ReduceLROnPlateau), early stopping\n\n**5. Pseudo-labeling on test data (most effective)**\n\n**6. Trick when training Dig-MNIST dataset**\n As discussed in [post1 ](https://www.kaggle.com/c/Kannada-MNIST/discussion/111462)and [post2](https://www.kaggle.com/c/Kannada-MNIST/discussion/112933), you may notice that your model will get worse if you add Dig-MNIST dataset directly to your training dataset. There are some issues in Dig-MNIST dataset such as border cut-off effect, non-native user's style... which may cause poor performance to your original model.\n \nMy baseline model(trained only with 60k training data) got ~99.8% accuracy on training set but got only ~90% accuracy on Dig-MNIST dataset. It seems difficult to generalize well on that 10% rest of images, but it also means that your model is still able to predict most of the images correctly. So I trained with Dig-MNIST by following steps:\n(1)Train a baseline model(with 60k training data) \n(2)Use model to perform pseudo labeling on test images (now you have 65k training data)\n(3)Use model to predict Dig-MNIST dataset, add the correctly classified data to the training set (says 9200 images, now you have 74200 training data)\n(4)Train a new model on new training dataset, replace baseline model with new model\n(5)Repeat steps (2) to (4), and see if there's improvement on Dig-MNIST prediction\n\nthe flowchart is illustrated as below:\n\n\n\nPseudo-labeling and Dig-MNIST dataset training obviously improved my LB score\nfrom (private/public) : 0.9890/0.9894 to 0.99160/0.99120.\n\n**Some implementation details:**\n- Optimizer: Adam\n- Epoch of each fold: maximum 200\n- Initial learning rate: 1e-3\n- Data augmentation (in pytorch): \n torchvision.transform.RandomAffine(degree=15, translate=(0.25,0.25),scale=(0.7,1.1),shear=8)\n- Model training time: One set of training for 5-fold cross validation takes about 4 hours on my PC (RTX 2080+i9-9900KF)\n\n##What didn't work for me\n**1. More folds cross-validation (up to 20 folds)**\n\n**2. Further image augmentations:**\n colorjitter (brightness, contrast... ) will make model worse, also, you might want to \n avoid vertically flip since it will cause wrong classification for similar digits such as 6 and 9.\n\n**3. Critical subclasses classification(0 and 1, 6 and 9)**\n\n**4. Classify after \"Native\" classifier**\n In order to deal with \"newbie effect\" mentioned in this [post](https://www.kaggle.com/c/Kannada-MNIST/discussion/116064), before classifying the digits, I also tried to predict whether the numbers are written by native user or not. Assumes that training set are done by native user, label 1; Dig-MNIST set are done by non-native user, label 0, then train two different models separably for these two kinds of image, but this method doesn't improve in my case.\n\n**5. Several optimizers and learning rate reduce methods:**\n RMSProp, Adamax, AdamW, SGD, SGDR(SGD+CosineAnnealingWarmRestarts)...\n\n**6. Several architecture of neural network**\n Resnet18, Resnet34, Resnet50, SE_Resnet(SE block +residual network) ...\n\n**7. Transfer learning**\n\n**8. Snapshot ensembling**\n\n##What I want to try but got no time:\n**1. Border cut-off effect augmentation**\n\n**2. Different ensemble methods**\n\n**3. Tune hyperparameters in more systematically, or in more smarter ways**\n\n##Final Submission\nMy final models were trained over ~75K images (train set 60k + test set 5K + Dig-MNIST data 10k) and got 0.9924 on the private LB. In order to avoid overfitting on the Dig-MNIST dataset, I also submitted another models which were trained without Dig-MNIST dataset and got 0.9916 on the private LB.\n\nThanks for reading!", "votes": 16 }, { "id": 704302, "postDate": "2019-12-27T09:49:19.110Z", "content": "That's a great approach!. May I ask you 2 questions? First, How many epochs that you train the second model and third model and more...? Second, Every model you retrain again with the new data then you perform pseudo labeling on test images, could you tell me how is the label of test images changing? I think when training model, the model will memorize the label. So that could be a problem when performing pseudo label.
", "rawMarkdown": "That's a great approach!. May I ask you 2 questions? First, How many epochs that you train the second model and third model and more...? Second, Every model you retrain again with the new data then you perform pseudo labeling on test images, could you tell me how is the label of test images changing? I think when training model, the model will memorize the label. So that could be a problem when performing pseudo label.", "votes": 1, "replies": [ { "id": 712483, "postDate": "2020-01-07T09:36:51.673Z", "content": "Thank for your reply!\n1. I set maximum 200 epochs for each models, but with early stopping, most of them ended in < 70 epochs.\n2. After 5 stages of repeatedly training, there's only few changes on pseudo label compared to the first stage(about 20 labels are different), but the number of Dig-MNIST data (number N in my figure) increase about 200, much more than previous one. So I think how good can your model deal with DIG-MNIST dataset is more important than test set in this case.
", "rawMarkdown": "Thank for your reply!\n1. I set maximum 200 epochs for each models, but with early stopping, most of them ended in < 70 epochs.\n2. After 5 stages of repeatedly training, there's only few changes on pseudo label compared to the first stage(about 20 labels are different), but the number of Dig-MNIST data (number N in my figure) increase about 200, much more than previous one. So I think how good can your model deal with DIG-MNIST dataset is more important than test set in this case.", "votes": 2 } ] }, { "id": 698011, "postDate": "2019-12-18T17:02:21.303Z", "content": "Good work to get some benefit from the dig-mnist data for training. I used a function to clean the borders of the dig-mnist data, not as augmentation just as a preprocessing step. The code just set the outside n number of lines in the reshaped 28x28 image to 0. I found n=2 worked quite well. I didn't use the data for training. The preprocessing step led to an improved holdout validation score. Something like this might have helped your approach a little bit. Not that you needed much help 🙂
\n\nCongratulations.
", "rawMarkdown": "Good work to get some benefit from the dig-mnist data for training. I used a function to clean the borders of the dig-mnist data, not as augmentation just as a preprocessing step. The code just set the outside n number of lines in the reshaped 28x28 image to 0. I found n=2 worked quite well. I didn't use the data for training. The preprocessing step led to an improved holdout validation score. Something like this might have helped your approach a little bit. Not that you needed much help 🙂 \n\nCongratulations.", "votes": 1, "replies": [ { "id": 698267, "postDate": "2019-12-19T01:56:58.980Z", "content": "Thank you! border cleaning is also an option, I didn’t do that cause I think translate and zoom in augmentation sometimes do that for me, interesting to hear that it does help improve the accuracy!\nyour works and shared information in this competition are very useful to me, congrats to you too!
", "rawMarkdown": "Thank you! border cleaning is also an option, I didn’t do that cause I think translate and zoom in augmentation sometimes do that for me, interesting to hear that it does help improve the accuracy!\nyour works and shared information in this competition are very useful to me, congrats to you too!", "votes": 1 }, { "id": 701471, "postDate": "2019-12-23T14:19:54.643Z", "content": "Thanks
", "rawMarkdown": "Thanks" } ] }, { "id": 712857, "postDate": "2020-01-07T16:36:17.610Z", "content": "Very clear explanation. Can I have the entire code to learn?
", "rawMarkdown": "Very clear explanation. Can I have the entire code to learn?" }, { "id": 700619, "postDate": "2019-12-22T10:09:22.443Z", "content": "Congrats and thanks for sharing.
", "rawMarkdown": "Congrats and thanks for sharing.", "replies": [ { "id": 701457, "postDate": "2019-12-23T14:07:35.353Z", "content": "Thanks you!
", "rawMarkdown": "Thanks you!" } ] }, { "id": 697757, "postDate": "2019-12-18T11:05:21.963Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 697805, "postDate": "2019-12-18T12:47:35.887Z", "content": "Thank you for the comment!
", "rawMarkdown": "Thank you for the comment!" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T11:35:35Z", "competition_id": 16017, "competition_slug": "Kannada-MNIST", "team_id": 3891052, "solution_url": "/competitions/Kannada-MNIST/writeups/ccchang-5th-place-solution-pseudo-labeling-dig-mni" }