![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3429800%2F7b3bc804acdcdcb9999b2bee75e05a8a%2FScreen%20Shot%202019-11-15%20at%2010.05.06%20AM.png?generation=1573784180378194&alt=media\")
In this Kannada MNIST competition using the MNIST data set of numerals in the Kannada script, validation accuracy is above 0.99 for many models and at least one models that has scored highest on the leaderboard had validation accuracy above 0.999. However, on the public test set (public leaderboard), at the time of writing, only 1 of 250 contestants has acheived accuracy > 0.99.
\n\nWe might be tempted to think that validation accuracy > 0.98 is still good. In some settings it might be. If we encountered this gap in a real world situation we would have to think about a few things. If we saw validation accuracy of 0.999 then the error would be 0.001 but if we then saw accuracy on the test set of 0.99 then the error would be 0.01. The error is 10 times larger. This could suggest our model is not generalising well.
\n\nIn real life situations if we faced this then there are 2 questions we could consider initially:\n- Is the validation scheme robust? If we are not using cross-validation should we consider using it?\n- Would using some regularisation help?
\n\nThis kernel implements cross-validation and high regularisation. Full disclosure - it is my kernel! It scores high enough to be in the top 50, at time of writing, but it doesn't really close the gap.
\n\nMy personal suspicion is that this new MNIST dataset has somehow been made harder than the original one. There doesn't seem like much point creating a dataset that can more-or-less be solved by models written for another existing dataset. I could be wrong though. It will be interesting to see what happens when the private leaderboard is revealed in a couple of months.
", "messages": [ { "id": "642223", "postDate": "10/05/2019 18:04:37", "content": "In this Kannada MNIST competition using the MNIST data set of numerals in the Kannada script, validation accuracy is above 0.99 for many models and at least one models that has scored highest on the leaderboard had validation accuracy above 0.999. However, on the public test set (public leaderboard), at the time of writing, only 1 of 250 contestants has acheived accuracy > 0.99.
\n\nWe might be tempted to think that validation accuracy > 0.98 is still good. In some settings it might be. If we encountered this gap in a real world situation we would have to think about a few things. If we saw validation accuracy of 0.999 then the error would be 0.001 but if we then saw accuracy on the test set of 0.99 then the error would be 0.01. The error is 10 times larger. This could suggest our model is not generalising well.
\n\nIn real life situations if we faced this then there are 2 questions we could consider initially:\n- Is the validation scheme robust? If we are not using cross-validation should we consider using it?\n- Would using some regularisation help?
\n\nThis kernel implements cross-validation and high regularisation. Full disclosure - it is my kernel! It scores high enough to be in the top 50, at time of writing, but it doesn't really close the gap.
\n\nMy personal suspicion is that this new MNIST dataset has somehow been made harder than the original one. There doesn't seem like much point creating a dataset that can more-or-less be solved by models written for another existing dataset. I could be wrong though. It will be interesting to see what happens when the private leaderboard is revealed in a couple of months.
", "rawMarkdown": "In this Kannada MNIST competition using the MNIST data set of numerals in the Kannada script, validation accuracy is above 0.99 for many models and at least one models that has scored highest on the leaderboard had validation accuracy above 0.999. However, on the public test set (public leaderboard), at the time of writing, only 1 of 250 contestants has acheived accuracy > 0.99.\n\nWe might be tempted to think that validation accuracy > 0.98 is still good. In some settings it might be. If we encountered this gap in a real world situation we would have to think about a few things. If we saw validation accuracy of 0.999 then the error would be 0.001 but if we then saw accuracy on the test set of 0.99 then the error would be 0.01. The error is 10 times larger. This could suggest our model is not generalising well.\n\nIn real life situations if we faced this then there are 2 questions we could consider initially:\n- Is the validation scheme robust? If we are not using cross-validation should we consider using it?\n- Would using some regularisation help?\n\nThis [kernel](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script?scriptVersionId=21522178) implements cross-validation and high regularisation. Full disclosure - it is my kernel! It scores high enough to be in the top 50, at time of writing, but it doesn't really close the gap.\n\nMy personal suspicion is that this new MNIST dataset has somehow been made harder than the original one. There doesn't seem like much point creating a dataset that can more-or-less be solved by models written for another existing dataset. I could be wrong though. It will be interesting to see what happens when the private leaderboard is revealed in a couple of months.", "votes": null }, { "id": "642301", "postDate": "10/05/2019 20:30:34", "content": "I think above 95% accuracy means that model is already generalizing the situation but does not catch exceptional cases( namely outlier). so it would matter to catch its outliers perfectly.
", "rawMarkdown": "I think above 95% accuracy means that model is already generalizing the situation but does not catch exceptional cases( namely outlier). so it would matter to catch its outliers perfectly.", "votes": null }, { "id": "642305", "postDate": "10/05/2019 20:33:49", "content": "I personally collected some output datas which perform above 98% from the other competitors and compared them with where they do not coincide, and I found that the points where they do not match, predict different candidates respectively. <- I think this is important.
", "rawMarkdown": "I personally collected some output datas which perform above 98% from the other competitors and compared them with where they do not coincide, and I found that the points where they do not match, predict different candidates respectively. <- I think this is important.", "votes": null }, { "id": "642503", "postDate": "10/06/2019 07:10:07", "content": "Yes I agree, there could be something interesting to learn in that situation.
", "rawMarkdown": "Yes I agree, there could be something interesting to learn in that situation.", "votes": null }, { "id": "642706", "postDate": "10/06/2019 14:08:22", "content": "95% accuracy is just for a single digit. In any real-life application the model would have to capture many digits to be useful. For example, in the UK, a mobile telephone number contains 11 digits. In this case, if we assume that the probability of correctly predicting each digit is independent, with an accuracy of 95% per digit, the probability of correctly getting the whole telephone number is 56.88%. In practice this level of accuracy for this task would not be acceptable because the number would be wrong approximately 3 times out of 7. If the accuracy for a single digit is 99% the probability of getting the whole telephone number correct is 89.5338%. Finally, if the accuracy for a single digit is 99.7% then the probability of getting the whole number correct would be 96.7491%.
\n\nThe accuracy ceiling in the other MNIST competition is around 0.997 (unless the model was trained using the whole dataset which is publicly available). We can see that even with accuracy like that for a simple practical application the model would incorrectly capture a hand written telephone number more than 3 times out of 100.
", "rawMarkdown": "95% accuracy is just for a single digit. In any real-life application the model would have to capture many digits to be useful. For example, in the UK, a mobile telephone number contains 11 digits. In this case, if we assume that the probability of correctly predicting each digit is independent, with an accuracy of 95% per digit, the probability of correctly getting the whole telephone number is 56.88%. In practice this level of accuracy for this task would not be acceptable because the number would be wrong approximately 3 times out of 7. If the accuracy for a single digit is 99% the probability of getting the whole telephone number correct is 89.5338%. Finally, if the accuracy for a single digit is 99.7% then the probability of getting the whole number correct would be 96.7491%.\n\nThe accuracy ceiling in the other MNIST competition is around 0.997 (unless the model was trained using the whole dataset which is publicly available). We can see that even with accuracy like that for a simple practical application the model would incorrectly capture a hand written telephone number more than 3 times out of 100.", "votes": null }, { "id": "642801", "postDate": "10/06/2019 17:10:09", "content": "I agree with your opinion when each predictions are processed independently and identically. That would have a probability of prediction rates^(the number of digits). Moreover, There are also some contexts being used to predict digits, letters, and words... in real-life as humans do, then it should follow conditional probability and could narrow down the candidates to pick :) Surely, achieving high accuracy on every single digit will immensely improve overall prediction rates anyhow! :)
", "rawMarkdown": "I agree with your opinion when each predictions are processed independently and identically. That would have a probability of prediction rates^(the number of digits). Moreover, There are also some contexts being used to predict digits, letters, and words... in real-life as humans do, then it should follow conditional probability and could narrow down the candidates to pick :) Surely, achieving high accuracy on every single digit will immensely improve overall prediction rates anyhow! :)", "votes": null }, { "id": "643141", "postDate": "10/07/2019 06:57:56", "content": "\n\n\nSurely, achieving high accuracy on every single digit will immensely improve overall prediction rates anyhow!
\n
Agreed
", "rawMarkdown": "> Surely, achieving high accuracy on every single digit will immensely improve overall prediction rates anyhow! \n\nAgreed", "votes": null }, { "id": "644235", "postDate": "10/08/2019 14:19:50", "content": "HI, Always skeptic about the training/testing data. But if there are some of the objects 98% looking the same and the results are different... In the end the goal is to make something that is working i life rather to hit the score :)\nAnyway very good point. voted up.\nregards, \nA
", "rawMarkdown": "HI, Always skeptic about the training/testing data. But if there are some of the objects 98% looking the same and the results are different... In the end the goal is to make something that is working i life rather to hit the score :)\nAnyway very good point. voted up.\nregards, \nA", "votes": null }, { "id": "644372", "postDate": "10/08/2019 17:52:35", "content": "Sooner or later these type of questions come up in pretty much every competition. What might happen in the shakeup? How similar are train and test? How is the test set split between public and private? I think people remember the big shakeups and often the shakeup is not as big as expected. I agree it is good even when trying to hit a score to think about if something will work in real life.
", "rawMarkdown": "Sooner or later these type of questions come up in pretty much every competition. What might happen in the shakeup? How similar are train and test? How is the test set split between public and private? I think people remember the big shakeups and often the shakeup is not as big as expected. I agree it is good even when trying to hit a score to think about if something will work in real life.", "votes": null }, { "id": "650196", "postDate": "10/16/2019 07:15:32", "content": "I have 0.9983 on the validation set and I have 0.98940 on the test set. I guess some hard examples are in the test-set!
", "rawMarkdown": "I have 0.9983 on the validation set and I have 0.98940 on the test set. I guess some hard examples are in the test-set!", "votes": null }, { "id": "663585", "postDate": "11/02/2019 11:25:16", "content": "0.9983 is a pretty impressive validation score. Do you mind sharing a bit more about your validation setup? Did you do a train test split? What was the split e.g. 90-10?
", "rawMarkdown": "0.9983 is a pretty impressive validation score. Do you mind sharing a bit more about your validation setup? Did you do a train test split? What was the split e.g. 90-10?", "votes": null }, { "id": "673463", "postDate": "11/15/2019 02:18:01", "content": "Great post. \nI got a question on the scores. I'm using CNN (Con2D x 6 + BatchNormalization) to train my model.\nI have tried to run the same Version of kernel with exactly the same parameters for 3 times, but each time I got a different result. Any advise on how to stabilize the results ?
\n\n
Thanks. There are a few reasons the results are not stable:\n1. Neural Networks start with random initialisation and are indeterministic so they don't produce the same result every time.\n2. The test used for the public leaderboard is small so scores are more likely to fluctuate noticeably between runs.\n3. The submission process reruns all your code every time which combines with point 1 to mean you won't get the same result every time.\n4. Training time is limited. With more training time approaches like bagging could help a bit. Because the plb test set is small there will still be some variance I suspect until the number of bags is very high.
\n\nI have also seen unstable results and others have reported this as well. In most Kaggle competitions it is important not to focus too much on the public leaderboard but instead focus on validation scores you are seeing. That may also be true here.
", "rawMarkdown": "Thanks. There are a few reasons the results are not stable:\n1. Neural Networks start with random initialisation and are indeterministic so they don't produce the same result every time.\n2. The test used for the public leaderboard is small so scores are more likely to fluctuate noticeably between runs.\n3. The submission process reruns all your code every time which combines with point 1 to mean you won't get the same result every time.\n4. Training time is limited. With more training time approaches like bagging could help a bit. Because the plb test set is small there will still be some variance I suspect until the number of bags is very high. \n\nI have also seen unstable results and others have reported this as well. In most Kaggle competitions it is important not to focus too much on the public leaderboard but instead focus on validation scores you are seeing. That may also be true here.", "votes": null }, { "id": "677188", "postDate": "11/19/2019 22:44:45", "content": "98-2(to really maxout on training data), I have 0.999 at validation now and 0.9906 as real Score (11th) place.
", "rawMarkdown": "98-2(to really maxout on training data), I have 0.999 at validation now and 0.9906 as real Score (11th) place.", "votes": null } ], "comments": [ { "id": 642301, "author_name": "hanjoonchoe", "author_url": "", "post_date": "10/05/2019 20:30:34", "content": "I think above 95% accuracy means that model is already generalizing the situation but does not catch exceptional cases( namely outlier). so it would matter to catch its outliers perfectly.
", "votes": null, "replies": [ { "id": 642305, "author_name": "hanjoonchoe", "author_url": "", "post_date": "10/05/2019 20:33:49", "content": "I personally collected some output datas which perform above 98% from the other competitors and compared them with where they do not coincide, and I found that the points where they do not match, predict different candidates respectively. <- I think this is important.
", "votes": null, "replies": [] }, { "id": 642503, "author_name": "datahobbit", "author_url": "", "post_date": "10/06/2019 07:10:07", "content": "Yes I agree, there could be something interesting to learn in that situation.
", "votes": null, "replies": [] }, { "id": 642706, "author_name": "datahobbit", "author_url": "", "post_date": "10/06/2019 14:08:22", "content": "95% accuracy is just for a single digit. In any real-life application the model would have to capture many digits to be useful. For example, in the UK, a mobile telephone number contains 11 digits. In this case, if we assume that the probability of correctly predicting each digit is independent, with an accuracy of 95% per digit, the probability of correctly getting the whole telephone number is 56.88%. In practice this level of accuracy for this task would not be acceptable because the number would be wrong approximately 3 times out of 7. If the accuracy for a single digit is 99% the probability of getting the whole telephone number correct is 89.5338%. Finally, if the accuracy for a single digit is 99.7% then the probability of getting the whole number correct would be 96.7491%.
\n\nThe accuracy ceiling in the other MNIST competition is around 0.997 (unless the model was trained using the whole dataset which is publicly available). We can see that even with accuracy like that for a simple practical application the model would incorrectly capture a hand written telephone number more than 3 times out of 100.
", "votes": null, "replies": [] }, { "id": 642801, "author_name": "hanjoonchoe", "author_url": "", "post_date": "10/06/2019 17:10:09", "content": "I agree with your opinion when each predictions are processed independently and identically. That would have a probability of prediction rates^(the number of digits). Moreover, There are also some contexts being used to predict digits, letters, and words... in real-life as humans do, then it should follow conditional probability and could narrow down the candidates to pick :) Surely, achieving high accuracy on every single digit will immensely improve overall prediction rates anyhow! :)
", "votes": null, "replies": [] }, { "id": 643141, "author_name": "datahobbit", "author_url": "", "post_date": "10/07/2019 06:57:56", "content": "\n\n\nSurely, achieving high accuracy on every single digit will immensely improve overall prediction rates anyhow!
\n
Agreed
", "votes": null, "replies": [] } ] }, { "id": 644235, "author_name": "zinovadr", "author_url": "", "post_date": "10/08/2019 14:19:50", "content": "HI, Always skeptic about the training/testing data. But if there are some of the objects 98% looking the same and the results are different... In the end the goal is to make something that is working i life rather to hit the score :)\nAnyway very good point. voted up.\nregards, \nA
", "votes": null, "replies": [ { "id": 644372, "author_name": "datahobbit", "author_url": "", "post_date": "10/08/2019 17:52:35", "content": "Sooner or later these type of questions come up in pretty much every competition. What might happen in the shakeup? How similar are train and test? How is the test set split between public and private? I think people remember the big shakeups and often the shakeup is not as big as expected. I agree it is good even when trying to hit a score to think about if something will work in real life.
", "votes": null, "replies": [] } ] }, { "id": 650196, "author_name": "rincewind007", "author_url": "", "post_date": "10/16/2019 07:15:32", "content": "I have 0.9983 on the validation set and I have 0.98940 on the test set. I guess some hard examples are in the test-set!
", "votes": null, "replies": [ { "id": 663585, "author_name": "datahobbit", "author_url": "", "post_date": "11/02/2019 11:25:16", "content": "0.9983 is a pretty impressive validation score. Do you mind sharing a bit more about your validation setup? Did you do a train test split? What was the split e.g. 90-10?
", "votes": null, "replies": [] }, { "id": 677188, "author_name": "rincewind007", "author_url": "", "post_date": "11/19/2019 22:44:45", "content": "98-2(to really maxout on training data), I have 0.999 at validation now and 0.9906 as real Score (11th) place.
", "votes": null, "replies": [] } ] }, { "id": 673463, "author_name": "xiu0714", "author_url": "", "post_date": "11/15/2019 02:18:01", "content": "Great post. \nI got a question on the scores. I'm using CNN (Con2D x 6 + BatchNormalization) to train my model.\nI have tried to run the same Version of kernel with exactly the same parameters for 3 times, but each time I got a different result. Any advise on how to stabilize the results ?
\n\n
Thanks. There are a few reasons the results are not stable:\n1. Neural Networks start with random initialisation and are indeterministic so they don't produce the same result every time.\n2. The test used for the public leaderboard is small so scores are more likely to fluctuate noticeably between runs.\n3. The submission process reruns all your code every time which combines with point 1 to mean you won't get the same result every time.\n4. Training time is limited. With more training time approaches like bagging could help a bit. Because the plb test set is small there will still be some variance I suspect until the number of bags is very high.
\n\nI have also seen unstable results and others have reported this as well. In most Kaggle competitions it is important not to focus too much on the public leaderboard but instead focus on validation scores you are seeing. That may also be true here.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "642223": "In this Kannada MNIST competition using the MNIST data set of numerals in the Kannada script, validation accuracy is above 0.99 for many models and at least one models that has scored highest on the leaderboard had validation accuracy above 0.999. However, on the public test set (public leaderboard), at the time of writing, only 1 of 250 contestants has acheived accuracy > 0.99.\n\nWe might be tempted to think that validation accuracy > 0.98 is still good. In some settings it might be. If we encountered this gap in a real world situation we would have to think about a few things. If we saw validation accuracy of 0.999 then the error would be 0.001 but if we then saw accuracy on the test set of 0.99 then the error would be 0.01. The error is 10 times larger. This could suggest our model is not generalising well.\n\nIn real life situations if we faced this then there are 2 questions we could consider initially:\n- Is the validation scheme robust? If we are not using cross-validation should we consider using it?\n- Would using some regularisation help?\n\nThis [kernel](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script?scriptVersionId=21522178) implements cross-validation and high regularisation. Full disclosure - it is my kernel! It scores high enough to be in the top 50, at time of writing, but it doesn't really close the gap.\n\nMy personal suspicion is that this new MNIST dataset has somehow been made harder than the original one. There doesn't seem like much point creating a dataset that can more-or-less be solved by models written for another existing dataset. I could be wrong though. It will be interesting to see what happens when the private leaderboard is revealed in a couple of months.", "642301": "I think above 95% accuracy means that model is already generalizing the situation but does not catch exceptional cases( namely outlier). so it would matter to catch its outliers perfectly.", "642305": "I personally collected some output datas which perform above 98% from the other competitors and compared them with where they do not coincide, and I found that the points where they do not match, predict different candidates respectively. <- I think this is important.", "642503": "Yes I agree, there could be something interesting to learn in that situation.", "642706": "95% accuracy is just for a single digit. In any real-life application the model would have to capture many digits to be useful. For example, in the UK, a mobile telephone number contains 11 digits. In this case, if we assume that the probability of correctly predicting each digit is independent, with an accuracy of 95% per digit, the probability of correctly getting the whole telephone number is 56.88%. In practice this level of accuracy for this task would not be acceptable because the number would be wrong approximately 3 times out of 7. If the accuracy for a single digit is 99% the probability of getting the whole telephone number correct is 89.5338%. Finally, if the accuracy for a single digit is 99.7% then the probability of getting the whole number correct would be 96.7491%.\n\nThe accuracy ceiling in the other MNIST competition is around 0.997 (unless the model was trained using the whole dataset which is publicly available). We can see that even with accuracy like that for a simple practical application the model would incorrectly capture a hand written telephone number more than 3 times out of 100.", "642801": "I agree with your opinion when each predictions are processed independently and identically. That would have a probability of prediction rates^(the number of digits). Moreover, There are also some contexts being used to predict digits, letters, and words... in real-life as humans do, then it should follow conditional probability and could narrow down the candidates to pick :) Surely, achieving high accuracy on every single digit will immensely improve overall prediction rates anyhow! :)", "643141": "> Surely, achieving high accuracy on every single digit will immensely improve overall prediction rates anyhow! \n\nAgreed", "644235": "HI, Always skeptic about the training/testing data. But if there are some of the objects 98% looking the same and the results are different... In the end the goal is to make something that is working i life rather to hit the score :)\nAnyway very good point. voted up.\nregards, \nA", "644372": "Sooner or later these type of questions come up in pretty much every competition. What might happen in the shakeup? How similar are train and test? How is the test set split between public and private? I think people remember the big shakeups and often the shakeup is not as big as expected. I agree it is good even when trying to hit a score to think about if something will work in real life.", "650196": "I have 0.9983 on the validation set and I have 0.98940 on the test set. I guess some hard examples are in the test-set!", "663585": "0.9983 is a pretty impressive validation score. Do you mind sharing a bit more about your validation setup? Did you do a train test split? What was the split e.g. 90-10?", "673463": "Great post. \nI got a question on the scores. I'm using CNN (Con2D x 6 + BatchNormalization) to train my model.\nI have tried to run the same Version of kernel with exactly the same parameters for 3 times, but each time I got a different result. Any advise on how to stabilize the results ?\n\n", "673595": "Thanks. There are a few reasons the results are not stable:\n1. Neural Networks start with random initialisation and are indeterministic so they don't produce the same result every time.\n2. The test used for the public leaderboard is small so scores are more likely to fluctuate noticeably between runs.\n3. The submission process reruns all your code every time which combines with point 1 to mean you won't get the same result every time.\n4. Training time is limited. With more training time approaches like bagging could help a bit. Because the plb test set is small there will still be some variance I suspect until the number of bags is very high. \n\nI have also seen unstable results and others have reported this as well. In most Kaggle competitions it is important not to focus too much on the public leaderboard but instead focus on validation scores you are seeing. That may also be true here.", "677188": "98-2(to really maxout on training data), I have 0.999 at validation now and 0.9906 as real Score (11th) place." }, "source": "meta" }