The validation set gives 0,602 accuracy, submitted prediction test gives 0,936. That sounds strange to me so I'm asking fellow kaggelers here to confir this and why is it.
", "messages": [ { "id": "641135", "postDate": "10/04/2019 13:03:02", "content": "The validation set gives 0,602 accuracy, submitted prediction test gives 0,936. That sounds strange to me so I'm asking fellow kaggelers here to confir this and why is it.
", "rawMarkdown": "The validation set gives 0,602 accuracy, submitted prediction test gives 0,936. That sounds strange to me so I'm asking fellow kaggelers here to confir this and why is it.", "votes": null }, { "id": "641383", "postDate": "10/04/2019 16:30:43", "content": "What model did you use? The answer may be different.
\n\nDid you shuffle the data? (train and validation)
", "rawMarkdown": "What model did you use? The answer may be different. \n\nDid you shuffle the data? (train and validation)", "votes": null }, { "id": "641394", "postDate": "10/04/2019 16:43:50", "content": "No, I did'nt shuffle the data, the model used is h2o.deeplearning hidden=c(160,5),epochs = 20 using R
", "rawMarkdown": "No, I did'nt shuffle the data, the model used is h2o.deeplearning hidden=c(160,5),epochs = 20 using R", "votes": null }, { "id": "641401", "postDate": "10/04/2019 16:53:16", "content": "I do not know the logic of h2o.
\n\nSo if i try different assumptions, I think the number of epochs is very low.
\n\nWhat percentage of accuracy did you get from the training dataset?
", "rawMarkdown": "I do not know the logic of h2o.\n\nSo if i try different assumptions, I think the number of epochs is very low.\n\nWhat percentage of accuracy did you get from the training dataset?", "votes": null }, { "id": "641405", "postDate": "10/04/2019 16:56:58", "content": "How did you split validation set and training set? It seems your validation set is really different from public test set.
", "rawMarkdown": "How did you split validation set and training set? It seems your validation set is really different from public test set.", "votes": null }, { "id": "641558", "postDate": "10/04/2019 19:43:16", "content": "Anna, I did'nt split anything, the validation set and the test set have been supplied directly in the input directory of the notebook as well as training set. I guess the validation set was deliberately built outside the train / test sets domain, I wonder why.
", "rawMarkdown": "Anna, I did'nt split anything, the validation set and the test set have been supplied directly in the input directory of the notebook as well as training set. I guess the validation set was deliberately built outside the train / test sets domain, I wonder why.", "votes": null }, { "id": "641584", "postDate": "10/04/2019 20:04:48", "content": "Did you use the Dig-MNIST.csv file for validation? The notes to the competition say that those digits were scanned in a different way and the accuracy reported on that data is much lower than on the train.csv validation and test.csv.
", "rawMarkdown": "Did you use the Dig-MNIST.csv file for validation? The notes to the competition say that those digits were scanned in a different way and the accuracy reported on that data is much lower than on the train.csv validation and test.csv.", "votes": null }, { "id": "641611", "postDate": "10/04/2019 20:53:37", "content": "The h2o.confusionMatrix on train set 5 folds CV says accuracy = 0,9977 in front of 0,63 of validation set run on that model. It seems the validation set not belonging to the domain of kannada mnist train / test set.
", "rawMarkdown": "The h2o.confusionMatrix on train set 5 folds CV says accuracy = 0,9977 in front of 0,63 of validation set run on that model. It seems the validation set not belonging to the domain of kannada mnist train / test set.", "votes": null }, { "id": "641793", "postDate": "10/05/2019 05:35:23", "content": "Yes, I did use Dig-MNIST.csv file for validation so now it's all clear. Thanks for your comment.
", "rawMarkdown": "Yes, I did use Dig-MNIST.csv file for validation so now it's all clear. Thanks for your comment.", "votes": null }, { "id": "642233", "postDate": "10/05/2019 18:23:05", "content": "Glad I could help. I believe that validation will be more of a challenge in this competition than the original MNIST comp. I just posted a couple of threads here and here that look at why high validation scores when using the training data are not replicated on the public test set (public leaderboard). Obviously we have the reverse problem validating with the Dig-MNIST.csv file. In this situation using both seems like a sensible way forward.
", "rawMarkdown": "Glad I could help. I believe that validation will be more of a challenge in this competition than the original MNIST comp. I just posted a couple of threads [here](https://www.kaggle.com/c/Kannada-MNIST/discussion/111460) and [here](https://www.kaggle.com/c/Kannada-MNIST/discussion/111462) that look at why high validation scores when using the training data are not replicated on the public test set (public leaderboard). Obviously we have the reverse problem validating with the Dig-MNIST.csv file. In this situation using both seems like a sensible way forward.", "votes": null }, { "id": "643160", "postDate": "10/07/2019 07:42:24", "content": "Now after realizing the reason why Dig-MNIST used as validation set gives much lower accuracy than that from train set CV confusion matrix , I tried CNN with Keras in R instead of using h2o deeplearning. I was surprised in finding that validation accuracy changes every time I run the model build / evaluate despite the fact I set a seed just before training the model. Any hint on this matter in order to get the same result at any new run?
", "rawMarkdown": "Now after realizing the reason why Dig-MNIST used as validation set gives much lower accuracy than that from train set CV confusion matrix , I tried CNN with Keras in R instead of using h2o deeplearning. I was surprised in finding that validation accuracy changes every time I run the model build / evaluate despite the fact I set a seed just before training the model. Any hint on this matter in order to get the same result at any new run?", "votes": null }, { "id": "643171", "postDate": "10/07/2019 07:54:37", "content": "Neural Networks are generally thought of as being indeterministic so that they do give slightly different results each time. You can try to fix the seed to prevent this from happening this article discusses some of the approaches to this. However when the GPU is involved and you have a complex model then there are many potential sources of randomness and it may not be worth the effort trying to eliminate them all.
\n\nOne approach mentioned in the article is that to get reliable results the experiment needs to be repeated a few times and summary statistics calculated. With the GPU quota there is a limit on how many times the experiment can be repeated. For a small example of repeating validation twice on the holdout test, run just on 2 epochs, see version 15 of my notebook. Version 16 has just started running now and should repeat the holdout validation 4 times on a full number of epochs. If the notebook is helpful upvotes are always appreciated.
", "rawMarkdown": "Neural Networks are generally thought of as being indeterministic so that they do give slightly different results each time. You can try to fix the seed to prevent this from happening [this article](https://machinelearningmastery.com/reproducible-results-neural-networks-keras/) discusses some of the approaches to this. However when the GPU is involved and you have a complex model then there are many potential sources of randomness and it may not be worth the effort trying to eliminate them all. \n\nOne approach mentioned in the article is that to get reliable results the experiment needs to be repeated a few times and summary statistics calculated. With the GPU quota there is a limit on how many times the experiment can be repeated. For a small example of repeating validation twice on the holdout test, run just on 2 epochs, see [version 15](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script?scriptVersionId=21626285) of my [notebook](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script). Version 16 has just started running now and should repeat the holdout validation 4 times on a full number of epochs. If the notebook is helpful upvotes are always appreciated.", "votes": null }, { "id": "643330", "postDate": "10/07/2019 12:12:59", "content": "Thanks a lot Michael for your hint and kernel. I appreciated.
", "rawMarkdown": "Thanks a lot Michael for your hint and kernel. I appreciated.", "votes": null }, { "id": "644238", "postDate": "10/08/2019 14:21:50", "content": "hm interesting. My score is giving me 0.9500 and the LB is 0.9880. Still maybe should be a way to remove the confusion training data.\nvoted up - valid point\nregards,\nA
", "rawMarkdown": "hm interesting. My score is giving me 0.9500 and the LB is 0.9880. Still maybe should be a way to remove the confusion training data.\nvoted up - valid point\nregards,\nA", "votes": null } ], "comments": [ { "id": 641383, "author_name": "subinium", "author_url": "", "post_date": "10/04/2019 16:30:43", "content": "What model did you use? The answer may be different.
\n\nDid you shuffle the data? (train and validation)
", "votes": null, "replies": [] }, { "id": 641394, "author_name": "vinloren", "author_url": "", "post_date": "10/04/2019 16:43:50", "content": "No, I did'nt shuffle the data, the model used is h2o.deeplearning hidden=c(160,5),epochs = 20 using R
", "votes": null, "replies": [ { "id": 641401, "author_name": "subinium", "author_url": "", "post_date": "10/04/2019 16:53:16", "content": "I do not know the logic of h2o.
\n\nSo if i try different assumptions, I think the number of epochs is very low.
\n\nWhat percentage of accuracy did you get from the training dataset?
", "votes": null, "replies": [] }, { "id": 641611, "author_name": "vinloren", "author_url": "", "post_date": "10/04/2019 20:53:37", "content": "The h2o.confusionMatrix on train set 5 folds CV says accuracy = 0,9977 in front of 0,63 of validation set run on that model. It seems the validation set not belonging to the domain of kannada mnist train / test set.
", "votes": null, "replies": [] } ] }, { "id": 641405, "author_name": "demonplus", "author_url": "", "post_date": "10/04/2019 16:56:58", "content": "How did you split validation set and training set? It seems your validation set is really different from public test set.
", "votes": null, "replies": [ { "id": 641558, "author_name": "vinloren", "author_url": "", "post_date": "10/04/2019 19:43:16", "content": "Anna, I did'nt split anything, the validation set and the test set have been supplied directly in the input directory of the notebook as well as training set. I guess the validation set was deliberately built outside the train / test sets domain, I wonder why.
", "votes": null, "replies": [] } ] }, { "id": 641584, "author_name": "datahobbit", "author_url": "", "post_date": "10/04/2019 20:04:48", "content": "Did you use the Dig-MNIST.csv file for validation? The notes to the competition say that those digits were scanned in a different way and the accuracy reported on that data is much lower than on the train.csv validation and test.csv.
", "votes": null, "replies": [ { "id": 641793, "author_name": "vinloren", "author_url": "", "post_date": "10/05/2019 05:35:23", "content": "Yes, I did use Dig-MNIST.csv file for validation so now it's all clear. Thanks for your comment.
", "votes": null, "replies": [] }, { "id": 642233, "author_name": "datahobbit", "author_url": "", "post_date": "10/05/2019 18:23:05", "content": "Glad I could help. I believe that validation will be more of a challenge in this competition than the original MNIST comp. I just posted a couple of threads here and here that look at why high validation scores when using the training data are not replicated on the public test set (public leaderboard). Obviously we have the reverse problem validating with the Dig-MNIST.csv file. In this situation using both seems like a sensible way forward.
", "votes": null, "replies": [] } ] }, { "id": 643160, "author_name": "vinloren", "author_url": "", "post_date": "10/07/2019 07:42:24", "content": "Now after realizing the reason why Dig-MNIST used as validation set gives much lower accuracy than that from train set CV confusion matrix , I tried CNN with Keras in R instead of using h2o deeplearning. I was surprised in finding that validation accuracy changes every time I run the model build / evaluate despite the fact I set a seed just before training the model. Any hint on this matter in order to get the same result at any new run?
", "votes": null, "replies": [ { "id": 643171, "author_name": "datahobbit", "author_url": "", "post_date": "10/07/2019 07:54:37", "content": "Neural Networks are generally thought of as being indeterministic so that they do give slightly different results each time. You can try to fix the seed to prevent this from happening this article discusses some of the approaches to this. However when the GPU is involved and you have a complex model then there are many potential sources of randomness and it may not be worth the effort trying to eliminate them all.
\n\nOne approach mentioned in the article is that to get reliable results the experiment needs to be repeated a few times and summary statistics calculated. With the GPU quota there is a limit on how many times the experiment can be repeated. For a small example of repeating validation twice on the holdout test, run just on 2 epochs, see version 15 of my notebook. Version 16 has just started running now and should repeat the holdout validation 4 times on a full number of epochs. If the notebook is helpful upvotes are always appreciated.
", "votes": null, "replies": [] }, { "id": 643330, "author_name": "vinloren", "author_url": "", "post_date": "10/07/2019 12:12:59", "content": "Thanks a lot Michael for your hint and kernel. I appreciated.
", "votes": null, "replies": [] } ] }, { "id": 644238, "author_name": "zinovadr", "author_url": "", "post_date": "10/08/2019 14:21:50", "content": "hm interesting. My score is giving me 0.9500 and the LB is 0.9880. Still maybe should be a way to remove the confusion training data.\nvoted up - valid point\nregards,\nA
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "641135": "The validation set gives 0,602 accuracy, submitted prediction test gives 0,936. That sounds strange to me so I'm asking fellow kaggelers here to confir this and why is it.", "641383": "What model did you use? The answer may be different. \n\nDid you shuffle the data? (train and validation)", "641394": "No, I did'nt shuffle the data, the model used is h2o.deeplearning hidden=c(160,5),epochs = 20 using R", "641401": "I do not know the logic of h2o.\n\nSo if i try different assumptions, I think the number of epochs is very low.\n\nWhat percentage of accuracy did you get from the training dataset?", "641405": "How did you split validation set and training set? It seems your validation set is really different from public test set.", "641558": "Anna, I did'nt split anything, the validation set and the test set have been supplied directly in the input directory of the notebook as well as training set. I guess the validation set was deliberately built outside the train / test sets domain, I wonder why.", "641584": "Did you use the Dig-MNIST.csv file for validation? The notes to the competition say that those digits were scanned in a different way and the accuracy reported on that data is much lower than on the train.csv validation and test.csv.", "641611": "The h2o.confusionMatrix on train set 5 folds CV says accuracy = 0,9977 in front of 0,63 of validation set run on that model. It seems the validation set not belonging to the domain of kannada mnist train / test set.", "641793": "Yes, I did use Dig-MNIST.csv file for validation so now it's all clear. Thanks for your comment.", "642233": "Glad I could help. I believe that validation will be more of a challenge in this competition than the original MNIST comp. I just posted a couple of threads [here](https://www.kaggle.com/c/Kannada-MNIST/discussion/111460) and [here](https://www.kaggle.com/c/Kannada-MNIST/discussion/111462) that look at why high validation scores when using the training data are not replicated on the public test set (public leaderboard). Obviously we have the reverse problem validating with the Dig-MNIST.csv file. In this situation using both seems like a sensible way forward.", "643160": "Now after realizing the reason why Dig-MNIST used as validation set gives much lower accuracy than that from train set CV confusion matrix , I tried CNN with Keras in R instead of using h2o deeplearning. I was surprised in finding that validation accuracy changes every time I run the model build / evaluate despite the fact I set a seed just before training the model. Any hint on this matter in order to get the same result at any new run?", "643171": "Neural Networks are generally thought of as being indeterministic so that they do give slightly different results each time. You can try to fix the seed to prevent this from happening [this article](https://machinelearningmastery.com/reproducible-results-neural-networks-keras/) discusses some of the approaches to this. However when the GPU is involved and you have a complex model then there are many potential sources of randomness and it may not be worth the effort trying to eliminate them all. \n\nOne approach mentioned in the article is that to get reliable results the experiment needs to be repeated a few times and summary statistics calculated. With the GPU quota there is a limit on how many times the experiment can be repeated. For a small example of repeating validation twice on the holdout test, run just on 2 epochs, see [version 15](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script?scriptVersionId=21626285) of my [notebook](https://www.kaggle.com/datahobbit/cnn-for-digit-recognition-in-the-kannada-script). Version 16 has just started running now and should repeat the holdout validation 4 times on a full number of epochs. If the notebook is helpful upvotes are always appreciated.", "643330": "Thanks a lot Michael for your hint and kernel. I appreciated.", "644238": "hm interesting. My score is giving me 0.9500 and the LB is 0.9880. Still maybe should be a way to remove the confusion training data.\nvoted up - valid point\nregards,\nA" }, "source": "meta" }