In this thread I looked at some things we can do if we face accuracy in the test set that is noticeably lower than in our model validation. Here are some further thoughts about the gap that is being witnessed between validation accuracy and the test set accuracy on the public leaderboard.
\n\nIt is a possibility that the shape of the characters in the Kannada script means that there is just more possibility for misclassification. Maybe some of the digit shapes are just more similar to each other with this set of numerals than with Roman numerals. However, that would not explain the high validation accuracy that has been achieved in this competition. Therefore this does not seem like a good explanation of the gap between validation and test set accuracy.
\n\nIn reality, this competition is very similar to the original MNIST in terms of the nature and dimensions of the dataset and the problem to be solved. Models taken from the original MNIST competition with little modification (none to the architecture) achieve similar validation scores but lower scores on the public leaderboard. This means that it is likely the problem does not lie with the models. It could be that the public/private split of the Kannada test set has put many of the difficult to classify examples in the public test set and the private test set will score higher. Alternatively, it could be that, at least relative to the original MNIST competition, more of the difficult to classify examples are in the test set. If it is suspected that the properties of the test set are different to the training set then one approach that can be tried is adversarial validation. This can show how how accurately a model can detect if a digit is in the train or test set. It will be interesting to see any kernels that try that.
\n\nIn this competition it could also make sense to explore how the additional Dig-MNIST dataset can help with validation. The competition notes state that it can serve as a further out-of-sample test set which the dataset authors have stated is harder to classify accurately.
", "messages": [ { "id": "642228", "postDate": "10/05/2019 18:16:39", "content": "In this thread I looked at some things we can do if we face accuracy in the test set that is noticeably lower than in our model validation. Here are some further thoughts about the gap that is being witnessed between validation accuracy and the test set accuracy on the public leaderboard.
\n\nIt is a possibility that the shape of the characters in the Kannada script means that there is just more possibility for misclassification. Maybe some of the digit shapes are just more similar to each other with this set of numerals than with Roman numerals. However, that would not explain the high validation accuracy that has been achieved in this competition. Therefore this does not seem like a good explanation of the gap between validation and test set accuracy.
\n\nIn reality, this competition is very similar to the original MNIST in terms of the nature and dimensions of the dataset and the problem to be solved. Models taken from the original MNIST competition with little modification (none to the architecture) achieve similar validation scores but lower scores on the public leaderboard. This means that it is likely the problem does not lie with the models. It could be that the public/private split of the Kannada test set has put many of the difficult to classify examples in the public test set and the private test set will score higher. Alternatively, it could be that, at least relative to the original MNIST competition, more of the difficult to classify examples are in the test set. If it is suspected that the properties of the test set are different to the training set then one approach that can be tried is adversarial validation. This can show how how accurately a model can detect if a digit is in the train or test set. It will be interesting to see any kernels that try that.
\n\nIn this competition it could also make sense to explore how the additional Dig-MNIST dataset can help with validation. The competition notes state that it can serve as a further out-of-sample test set which the dataset authors have stated is harder to classify accurately.
", "rawMarkdown": "In this [thread](https://www.kaggle.com/c/Kannada-MNIST/discussion/111460) I looked at some things we can do if we face accuracy in the test set that is noticeably lower than in our model validation. Here are some further thoughts about the gap that is being witnessed between validation accuracy and the test set accuracy on the public leaderboard.\n\nIt is a possibility that the shape of the characters in the Kannada script means that there is just more possibility for misclassification. Maybe some of the digit shapes are just more similar to each other with this set of numerals than with Roman numerals. However, that would not explain the high validation accuracy that has been achieved in this competition. Therefore this does not seem like a good explanation of the gap between validation and test set accuracy.\n\nIn reality, this competition is very similar to the original MNIST in terms of the nature and dimensions of the dataset and the problem to be solved. Models taken from the original MNIST competition with little modification (none to the architecture) achieve similar validation scores but lower scores on the public leaderboard. This means that it is likely the problem does not lie with the models. It could be that the public/private split of the Kannada test set has put many of the difficult to classify examples in the public test set and the private test set will score higher. Alternatively, it could be that, at least relative to the original MNIST competition, more of the difficult to classify examples are in the test set. If it is suspected that the properties of the test set are different to the training set then one approach that can be tried is adversarial validation. This can show how how accurately a model can detect if a digit is in the train or test set. It will be interesting to see any kernels that try that.\n\nIn this competition it could also make sense to explore how the additional Dig-MNIST dataset can help with validation. The competition notes state that it can serve as a further out-of-sample test set which the dataset authors have stated is harder to classify accurately.", "votes": null }, { "id": "642632", "postDate": "10/06/2019 12:15:44", "content": "Thanks for posting @datahobbit it's very helpful
", "rawMarkdown": "Thanks for posting @datahobbit it's very helpful", "votes": null }, { "id": "642786", "postDate": "10/06/2019 16:46:30", "content": "You are welcome and thanks for the feedback.
", "rawMarkdown": "You are welcome and thanks for the feedback.", "votes": null }, { "id": "644244", "postDate": "10/08/2019 14:26:10", "content": "Interesting details.\nthx\nA
", "rawMarkdown": "Interesting details.\nthx\nA", "votes": null }, { "id": "645746", "postDate": "10/10/2019 12:51:01", "content": "Thanks. The updated version of this kernel implements both cross-validation and validation on the Dig-MNIST data.
", "rawMarkdown": "Thanks. The updated version of this [kernel](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script) implements both cross-validation and validation on the Dig-MNIST data.", "votes": null }, { "id": "657639", "postDate": "10/25/2019 10:17:41", "content": "Great job~ I am a beginner of data science, and I have a question about this. It's all right to 1) add the whole Dig-MNIST dataset to train set, 2) add the whole dataset to validation set, or 3) randomly split and separately add them to train and val set?
\n\nThanks
", "rawMarkdown": "Great job~ I am a beginner of data science, and I have a question about this. It's all right to 1) add the whole Dig-MNIST dataset to train set, 2) add the whole dataset to validation set, or 3) randomly split and separately add them to train and val set?\n\nThanks", "votes": null }, { "id": "657719", "postDate": "10/25/2019 11:46:04", "content": "tl;dr use the Dig-MNIST for validation but not for training
\n\nThese are the official Kaggle comments about the Dig-MNIST dataset:
\n\n> In addition to the main dataset, the author also disseminated an additional real world handwritten dataset (with 10k images), termed as the 'Dig-MNIST dataset' that can serve as an out-of-domain test dataset.
\n\n> This 'dig-MNIST' dataset serves as a more difficult test-set (An accuracy of 76.1% was reported in the paper cited above) and achieving ~98+% accuracy on this test dataset would be rather commendable.
\n\nA couple of people have reported adding the Dig-MNIST to their training data and seeing their scores go down. From the Kaggle comments above it seems that this is because the signal that can be extracted from that dataset is lower, and/or, there is more noise due to the fact it was
\n\n> It was created with the help of volunteers that were non-native users of the language, authored on a smaller sheet and scanned with different scanner settings compared to the main dataset.
\n\nIt seems to me that using the Dig-MNIST dataset for training leads to lower accuracy and doesn't exactly match the way that Kaggle intended it to be used. In my opinion it is worth exploring how it can be used in validation. My kernel explores a couple of different validation options including using Dig-MNIST.
", "rawMarkdown": "tl;dr use the Dig-MNIST for validation but not for training\n\nThese are the official Kaggle comments about the Dig-MNIST dataset:\n\n> In addition to the main dataset, the author also disseminated an additional real world handwritten dataset (with 10k images), termed as the 'Dig-MNIST dataset' that can serve as an out-of-domain test dataset. \n\n> This 'dig-MNIST' dataset serves as a more difficult test-set (An accuracy of 76.1% was reported in the paper cited above) and achieving ~98+% accuracy on this test dataset would be rather commendable.\n\nA couple of people have [reported](https://www.kaggle.com/c/Kannada-MNIST/discussion/112933#latest-654050) adding the Dig-MNIST to their training data and seeing their scores go down. From the Kaggle comments above it seems that this is because the signal that can be extracted from that dataset is lower, and/or, there is more noise due to the fact it was \n\n> It was created with the help of volunteers that were non-native users of the language, authored on a smaller sheet and scanned with different scanner settings compared to the main dataset.\n\nIt seems to me that using the Dig-MNIST dataset for training leads to lower accuracy and doesn't exactly match the way that Kaggle intended it to be used. In my opinion it is worth exploring how it can be used in validation. My [kernel](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script) explores a couple of different validation options including using Dig-MNIST.", "votes": null }, { "id": "657932", "postDate": "10/25/2019 14:24:03", "content": "Thank you so much, Mike. I agree with your thought and I will research your code. Hope you get a Gold.
", "rawMarkdown": "Thank you so much, Mike. I agree with your thought and I will research your code. Hope you get a Gold.", "votes": null }, { "id": "663586", "postDate": "11/02/2019 11:26:33", "content": "Thanks Xiaoyu - I hope the code was helpful. I don't think this competition will be awarding medals.
", "rawMarkdown": "Thanks Xiaoyu - I hope the code was helpful. I don't think this competition will be awarding medals.", "votes": null }, { "id": "680969", "postDate": "11/25/2019 13:20:22", "content": "\n\n\nAlternatively, it could be that, at least relative to the original MNIST competition, more of the difficult to classify examples are in the test set.
\n
Well, sometimes those \"more difficult\" samples are barely even possible to classify for human. At least this seems to be true for the original MNIST dataset i.e. there are samples for which you cannot be sure whether it's e.g. 3 or 9. I dont' think we can do anything if suchs samples exist in the test set, unfortunately. 😉
", "rawMarkdown": "> Alternatively, it could be that, at least relative to the original MNIST competition, more of the difficult to classify examples are in the test set.\n\nWell, sometimes those \"more difficult\" samples are barely even possible to classify for human. At least this seems to be true for the original MNIST dataset i.e. there are samples for which you cannot be sure whether it's e.g. 3 or 9. I dont' think we can do anything if suchs samples exist in the test set, unfortunately. 😉", "votes": null }, { "id": "681131", "postDate": "11/25/2019 17:51:41", "content": "Yes agreed - some samples will be barely possible for humans to classify so there will be a ceiling to what accuracy can be achieved and it will be less than 1. The accuracy on the public leaderboard is quite a lot lower than in validation, so, I was thinking maybe train test split is done in a non-random way to put more difficult samples in the test set.
", "rawMarkdown": "Yes agreed - some samples will be barely possible for humans to classify so there will be a ceiling to what accuracy can be achieved and it will be less than 1. The accuracy on the public leaderboard is quite a lot lower than in validation, so, I was thinking maybe train test split is done in a non-random way to put more difficult samples in the test set.", "votes": null } ], "comments": [ { "id": 642632, "author_name": "chirag9073", "author_url": "", "post_date": "10/06/2019 12:15:44", "content": "Thanks for posting @datahobbit it's very helpful
", "votes": null, "replies": [ { "id": 642786, "author_name": "datahobbit", "author_url": "", "post_date": "10/06/2019 16:46:30", "content": "You are welcome and thanks for the feedback.
", "votes": null, "replies": [] } ] }, { "id": 644244, "author_name": "zinovadr", "author_url": "", "post_date": "10/08/2019 14:26:10", "content": "Interesting details.\nthx\nA
", "votes": null, "replies": [ { "id": 645746, "author_name": "datahobbit", "author_url": "", "post_date": "10/10/2019 12:51:01", "content": "Thanks. The updated version of this kernel implements both cross-validation and validation on the Dig-MNIST data.
", "votes": null, "replies": [] } ] }, { "id": 657639, "author_name": "zhaixiaoyu", "author_url": "", "post_date": "10/25/2019 10:17:41", "content": "Great job~ I am a beginner of data science, and I have a question about this. It's all right to 1) add the whole Dig-MNIST dataset to train set, 2) add the whole dataset to validation set, or 3) randomly split and separately add them to train and val set?
\n\nThanks
", "votes": null, "replies": [ { "id": 657719, "author_name": "datahobbit", "author_url": "", "post_date": "10/25/2019 11:46:04", "content": "tl;dr use the Dig-MNIST for validation but not for training
\n\nThese are the official Kaggle comments about the Dig-MNIST dataset:
\n\n> In addition to the main dataset, the author also disseminated an additional real world handwritten dataset (with 10k images), termed as the 'Dig-MNIST dataset' that can serve as an out-of-domain test dataset.
\n\n> This 'dig-MNIST' dataset serves as a more difficult test-set (An accuracy of 76.1% was reported in the paper cited above) and achieving ~98+% accuracy on this test dataset would be rather commendable.
\n\nA couple of people have reported adding the Dig-MNIST to their training data and seeing their scores go down. From the Kaggle comments above it seems that this is because the signal that can be extracted from that dataset is lower, and/or, there is more noise due to the fact it was
\n\n> It was created with the help of volunteers that were non-native users of the language, authored on a smaller sheet and scanned with different scanner settings compared to the main dataset.
\n\nIt seems to me that using the Dig-MNIST dataset for training leads to lower accuracy and doesn't exactly match the way that Kaggle intended it to be used. In my opinion it is worth exploring how it can be used in validation. My kernel explores a couple of different validation options including using Dig-MNIST.
", "votes": null, "replies": [] }, { "id": 657932, "author_name": "zhaixiaoyu", "author_url": "", "post_date": "10/25/2019 14:24:03", "content": "Thank you so much, Mike. I agree with your thought and I will research your code. Hope you get a Gold.
", "votes": null, "replies": [] }, { "id": 663586, "author_name": "datahobbit", "author_url": "", "post_date": "11/02/2019 11:26:33", "content": "Thanks Xiaoyu - I hope the code was helpful. I don't think this competition will be awarding medals.
", "votes": null, "replies": [] } ] }, { "id": 680969, "author_name": "mtszkw", "author_url": "", "post_date": "11/25/2019 13:20:22", "content": "\n\n\nAlternatively, it could be that, at least relative to the original MNIST competition, more of the difficult to classify examples are in the test set.
\n
Well, sometimes those \"more difficult\" samples are barely even possible to classify for human. At least this seems to be true for the original MNIST dataset i.e. there are samples for which you cannot be sure whether it's e.g. 3 or 9. I dont' think we can do anything if suchs samples exist in the test set, unfortunately. 😉
", "votes": null, "replies": [ { "id": 681131, "author_name": "datahobbit", "author_url": "", "post_date": "11/25/2019 17:51:41", "content": "Yes agreed - some samples will be barely possible for humans to classify so there will be a ceiling to what accuracy can be achieved and it will be less than 1. The accuracy on the public leaderboard is quite a lot lower than in validation, so, I was thinking maybe train test split is done in a non-random way to put more difficult samples in the test set.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "642228": "In this [thread](https://www.kaggle.com/c/Kannada-MNIST/discussion/111460) I looked at some things we can do if we face accuracy in the test set that is noticeably lower than in our model validation. Here are some further thoughts about the gap that is being witnessed between validation accuracy and the test set accuracy on the public leaderboard.\n\nIt is a possibility that the shape of the characters in the Kannada script means that there is just more possibility for misclassification. Maybe some of the digit shapes are just more similar to each other with this set of numerals than with Roman numerals. However, that would not explain the high validation accuracy that has been achieved in this competition. Therefore this does not seem like a good explanation of the gap between validation and test set accuracy.\n\nIn reality, this competition is very similar to the original MNIST in terms of the nature and dimensions of the dataset and the problem to be solved. Models taken from the original MNIST competition with little modification (none to the architecture) achieve similar validation scores but lower scores on the public leaderboard. This means that it is likely the problem does not lie with the models. It could be that the public/private split of the Kannada test set has put many of the difficult to classify examples in the public test set and the private test set will score higher. Alternatively, it could be that, at least relative to the original MNIST competition, more of the difficult to classify examples are in the test set. If it is suspected that the properties of the test set are different to the training set then one approach that can be tried is adversarial validation. This can show how how accurately a model can detect if a digit is in the train or test set. It will be interesting to see any kernels that try that.\n\nIn this competition it could also make sense to explore how the additional Dig-MNIST dataset can help with validation. The competition notes state that it can serve as a further out-of-sample test set which the dataset authors have stated is harder to classify accurately.", "642632": "Thanks for posting @datahobbit it's very helpful", "642786": "You are welcome and thanks for the feedback.", "644244": "Interesting details.\nthx\nA", "645746": "Thanks. The updated version of this [kernel](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script) implements both cross-validation and validation on the Dig-MNIST data.", "657639": "Great job~ I am a beginner of data science, and I have a question about this. It's all right to 1) add the whole Dig-MNIST dataset to train set, 2) add the whole dataset to validation set, or 3) randomly split and separately add them to train and val set?\n\nThanks", "657719": "tl;dr use the Dig-MNIST for validation but not for training\n\nThese are the official Kaggle comments about the Dig-MNIST dataset:\n\n> In addition to the main dataset, the author also disseminated an additional real world handwritten dataset (with 10k images), termed as the 'Dig-MNIST dataset' that can serve as an out-of-domain test dataset. \n\n> This 'dig-MNIST' dataset serves as a more difficult test-set (An accuracy of 76.1% was reported in the paper cited above) and achieving ~98+% accuracy on this test dataset would be rather commendable.\n\nA couple of people have [reported](https://www.kaggle.com/c/Kannada-MNIST/discussion/112933#latest-654050) adding the Dig-MNIST to their training data and seeing their scores go down. From the Kaggle comments above it seems that this is because the signal that can be extracted from that dataset is lower, and/or, there is more noise due to the fact it was \n\n> It was created with the help of volunteers that were non-native users of the language, authored on a smaller sheet and scanned with different scanner settings compared to the main dataset.\n\nIt seems to me that using the Dig-MNIST dataset for training leads to lower accuracy and doesn't exactly match the way that Kaggle intended it to be used. In my opinion it is worth exploring how it can be used in validation. My [kernel](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script) explores a couple of different validation options including using Dig-MNIST.", "657932": "Thank you so much, Mike. I agree with your thought and I will research your code. Hope you get a Gold.", "663586": "Thanks Xiaoyu - I hope the code was helpful. I don't think this competition will be awarding medals.", "680969": "> Alternatively, it could be that, at least relative to the original MNIST competition, more of the difficult to classify examples are in the test set.\n\nWell, sometimes those \"more difficult\" samples are barely even possible to classify for human. At least this seems to be true for the original MNIST dataset i.e. there are samples for which you cannot be sure whether it's e.g. 3 or 9. I dont' think we can do anything if suchs samples exist in the test set, unfortunately. 😉", "681131": "Yes agreed - some samples will be barely possible for humans to classify so there will be a ceiling to what accuracy can be achieved and it will be less than 1. The accuracy on the public leaderboard is quite a lot lower than in validation, so, I was thinking maybe train test split is done in a non-random way to put more difficult samples in the test set." }, "source": "meta" }