Hi,
\n\nI'm able to get a loss as low as 0.24 in training and 0.38 on the validation set, but once I submit to Kaggle, my score is way off (sometimes as high as 4, but mostly I never got below 0.69). Any ideas what could be going wrong?
\n\nFYI, here is my code for writing the predictions to file (using Keras & TF):
\n\npredictions = model.predict(test, verbose=0)\npreds = pd.DataFrame({\n \"label\": map(lambda x: x[0], predictions)\n})\npreds.index += 1\npreds.to_csv('submissions/submission.csv', index_label=\"id\")\nThanks!
", "messages": [ { "id": "155553", "postDate": "01/11/2017 21:42:10", "content": "Hi,
\n\nI'm able to get a loss as low as 0.24 in training and 0.38 on the validation set, but once I submit to Kaggle, my score is way off (sometimes as high as 4, but mostly I never got below 0.69). Any ideas what could be going wrong?
\n\nFYI, here is my code for writing the predictions to file (using Keras & TF):
\n\npredictions = model.predict(test, verbose=0)\npreds = pd.DataFrame({\n \"label\": map(lambda x: x[0], predictions)\n})\npreds.index += 1\npreds.to_csv('submissions/submission.csv', index_label=\"id\")\nThanks!
", "rawMarkdown": "Hi,\r\n\r\nI'm able to get a loss as low as 0.24 in training and 0.38 on the validation set, but once I submit to Kaggle, my score is way off (sometimes as high as 4, but mostly I never got below 0.69). Any ideas what could be going wrong?\r\n\r\nFYI, here is my code for writing the predictions to file (using Keras & TF):\r\n\r\n predictions = model.predict(test, verbose=0)\r\n preds = pd.DataFrame({\r\n \"label\": map(lambda x: x[0], predictions)\r\n })\r\n preds.index += 1\r\n preds.to_csv('submissions/submission.csv', index_label=\"id\")\r\n\r\nThanks!", "votes": null }, { "id": "155667", "postDate": "01/12/2017 10:59:59", "content": "Hi Anas,\nRemember the score is calculated as log loss. This means you will be heavily penalized if you use high certainty predictions such as 1 and 0.
\n\nJeremy Howard explains this far better than I can. Watch this from 30 minutes for an explanation.
\n\nI found i got a much better score when i clipped my prediction\ne.g (python) . np.clip(predictions[:,1],0.0125,0.9875)
\n\nRgds\nIan
", "rawMarkdown": "Hi Anas,\r\nRemember the score is calculated as log loss. This means you will be heavily penalized if you use high certainty predictions such as 1 and 0.\r\n\r\nJeremy Howard explains this far better than I can. Watch [this][1] from 30 minutes for an explanation.\r\n\r\n [1]: https://www.youtube.com/watch?v=e3aM6XTekJc\r\n\r\nI found i got a much better score when i clipped my prediction\r\ne.g (python) . np.clip(predictions[:,1],0.0125,0.9875)\r\n\r\nRgds\r\nIan", "votes": null }, { "id": "155703", "postDate": "01/12/2017 15:16:53", "content": "Hi Ian, Thank you for your reply. I understand the log loss penalizes absolute predictions, and I've already tried replacing 1.0 by 0.95 and 0.0 by 0.05, but that didn't help (much). I'm also use data augmentation through Keras' ImageDataGenerator, which helps in training and validation, but still not on testing.
", "rawMarkdown": "Hi Ian, Thank you for your reply. I understand the log loss penalizes absolute predictions, and I've already tried replacing 1.0 by 0.95 and 0.0 by 0.05, but that didn't help (much). I'm also use data augmentation through Keras' ImageDataGenerator, which helps in training and validation, but still not on testing.", "votes": null }, { "id": "157241", "postDate": "01/19/2017 21:00:24", "content": "I've also tried running this code sample from Keras almost as is (on the full dataset), with the only additions the above code for generating predictions and Ian's line for clipping low/high probabilities.
\nAnd yet I still get a log loss score of 1.4+ on Kaggle.. That makes me think there's an issue with generating the predictions. Any help/ideas would be awesome! Thanks :)
@Anas B The Keras' ImageDataGenerator don't conserve the original order of the id's and I think that this might be your problem when creating the submission DataFrame .
\n\nI've created a tutorial and the last notebook covers how to use Keras' ImageDataGenerator to create a DataFrame in the right order.
", "rawMarkdown": "Anas B The Keras' ImageDataGenerator don't conserve the original order of the id's and I think that this might be your problem when creating the submission DataFrame .\r\n\r\nI've created [a tutorial][1] and the last notebook covers how to use Keras' ImageDataGenerator to create a DataFrame in the right order.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/forums/t/27950/tutorial-how-to-get-0-06-loss-in-70-lines-of-code?forumMessageId=157349#post157349", "votes": null }, { "id": "157384", "postDate": "01/20/2017 16:46:42", "content": "@David de la Iglesia Castro That helped, thank you so much!! The issue was indeed with the predictions output, and using your technique for writing the csv based on the id in the filename fixed my problem. Thank you!!
", "rawMarkdown": "David de la Iglesia Castro That helped, thank you so much!! The issue was indeed with the predictions output, and using your technique for writing the csv based on the id in the filename fixed my problem. Thank you!!", "votes": null }, { "id": "160280", "postDate": "02/06/2017 21:51:21", "content": "Thanks! with your tip I boosted my score up to 0.049 (from 0.14 before)
", "rawMarkdown": "Thanks! with your tip I boosted my score up to 0.049 (from 0.14 before)", "votes": null }, { "id": "160283", "postDate": "02/06/2017 22:04:02", "content": "Nice, good job! Did you use any special techniques, like data augmentation or normalization? Or just a simple convnet trained from scratch?
", "rawMarkdown": "Nice, good job! Did you use any special techniques, like data augmentation or normalization? Or just a simple convnet trained from scratch?", "votes": null }, { "id": "160375", "postDate": "02/07/2017 10:47:41", "content": "I used the pre-trained Inception-v3 available in TensorFlow to extract features and then a linear SVM classifier... I acknowledge it's a bit cheating...\nWith a personal convnet with no data augmentation I was quite far.
", "rawMarkdown": "I used the pre-trained Inception-v3 available in TensorFlow to extract features and then a linear SVM classifier... I acknowledge it's a bit cheating...\nWith a personal convnet with no data augmentation I was quite far.", "votes": null }, { "id": "161912", "postDate": "02/16/2017 12:08:29", "content": "@David de la Iglesia Castro I am not using Data Augmentation. My accuracy on validation data is 89% and log loss is 0.23 using CNN, however submitting the result of predictions on Kaggle gives 10+ log loss. What mistake i could have been doing possibly?
", "rawMarkdown": "David de la Iglesia Castro I am not using Data Augmentation. My accuracy on validation data is 89% and log loss is 0.23 using CNN, however submitting the result of predictions on Kaggle gives 10+ log loss. What mistake i could have been doing possibly?", "votes": null }, { "id": "220434", "postDate": "09/12/2017 05:04:09", "content": "I followed the tip of jeremy howard.\nI have clipped my prediction accuracies between 0.01 and 0.99 since my validation loss was as high as 99.97.\nBut at the leaderboard there is no improvement in the scores.\nCan you please suggest any other trick to try?
", "rawMarkdown": "I followed the tip of jeremy howard.\nI have clipped my prediction accuracies between 0.01 and 0.99 since my validation loss was as high as 99.97.\nBut at the leaderboard there is no improvement in the scores.\nCan you please suggest any other trick to try?", "votes": null }, { "id": "1050474", "postDate": "10/15/2020 12:56:29", "content": "For anyone who might still come to this threat. If you download the testdata from kaggle it is not ordered correctly due to the naming. You can fix this as follows:
\nfor i in os.listdir(test_dir):
\n os.rename(test_dir + i, test_dir + i.split('.')[0].zfill(6) + '.jpg')
\n test_images = [test_dir+i for i in os.listdir(test_dir)]
\n test_images.sort()
For anyone who might still come to this threat. If you download the testdata from kaggle it is not ordered correctly due to the naming. You can fix this as follows:
\nfor i in os.listdir(test_dir):
\n os.rename(test_dir + i, test_dir + i.split('.')[0].zfill(6) + '.jpg')
\n test_images = [test_dir+i for i in os.listdir(test_dir)]
\n test_images.sort()
Hi Anas,\nRemember the score is calculated as log loss. This means you will be heavily penalized if you use high certainty predictions such as 1 and 0.
\n\nJeremy Howard explains this far better than I can. Watch this from 30 minutes for an explanation.
\n\nI found i got a much better score when i clipped my prediction\ne.g (python) . np.clip(predictions[:,1],0.0125,0.9875)
\n\nRgds\nIan
", "votes": null, "replies": [ { "id": 160280, "author_name": "valenpe", "author_url": "", "post_date": "02/06/2017 21:51:21", "content": "Thanks! with your tip I boosted my score up to 0.049 (from 0.14 before)
", "votes": null, "replies": [] }, { "id": 160283, "author_name": "anasbou", "author_url": "", "post_date": "02/06/2017 22:04:02", "content": "Nice, good job! Did you use any special techniques, like data augmentation or normalization? Or just a simple convnet trained from scratch?
", "votes": null, "replies": [] }, { "id": 160375, "author_name": "valenpe", "author_url": "", "post_date": "02/07/2017 10:47:41", "content": "I used the pre-trained Inception-v3 available in TensorFlow to extract features and then a linear SVM classifier... I acknowledge it's a bit cheating...\nWith a personal convnet with no data augmentation I was quite far.
", "votes": null, "replies": [] } ] }, { "id": 155703, "author_name": "anasbou", "author_url": "", "post_date": "01/12/2017 15:16:53", "content": "Hi Ian, Thank you for your reply. I understand the log loss penalizes absolute predictions, and I've already tried replacing 1.0 by 0.95 and 0.0 by 0.05, but that didn't help (much). I'm also use data augmentation through Keras' ImageDataGenerator, which helps in training and validation, but still not on testing.
", "votes": null, "replies": [] }, { "id": 157241, "author_name": "anasbou", "author_url": "", "post_date": "01/19/2017 21:00:24", "content": "I've also tried running this code sample from Keras almost as is (on the full dataset), with the only additions the above code for generating predictions and Ian's line for clipping low/high probabilities.
\nAnd yet I still get a log loss score of 1.4+ on Kaggle.. That makes me think there's an issue with generating the predictions. Any help/ideas would be awesome! Thanks :)
@Anas B The Keras' ImageDataGenerator don't conserve the original order of the id's and I think that this might be your problem when creating the submission DataFrame .
\n\nI've created a tutorial and the last notebook covers how to use Keras' ImageDataGenerator to create a DataFrame in the right order.
", "votes": null, "replies": [] }, { "id": 157384, "author_name": "anasbou", "author_url": "", "post_date": "01/20/2017 16:46:42", "content": "@David de la Iglesia Castro That helped, thank you so much!! The issue was indeed with the predictions output, and using your technique for writing the csv based on the id in the filename fixed my problem. Thank you!!
", "votes": null, "replies": [] }, { "id": 161912, "author_name": "shaheen", "author_url": "", "post_date": "02/16/2017 12:08:29", "content": "@David de la Iglesia Castro I am not using Data Augmentation. My accuracy on validation data is 89% and log loss is 0.23 using CNN, however submitting the result of predictions on Kaggle gives 10+ log loss. What mistake i could have been doing possibly?
", "votes": null, "replies": [] }, { "id": 220434, "author_name": "abhayizm", "author_url": "", "post_date": "09/12/2017 05:04:09", "content": "I followed the tip of jeremy howard.\nI have clipped my prediction accuracies between 0.01 and 0.99 since my validation loss was as high as 99.97.\nBut at the leaderboard there is no improvement in the scores.\nCan you please suggest any other trick to try?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "155553": "Hi,\r\n\r\nI'm able to get a loss as low as 0.24 in training and 0.38 on the validation set, but once I submit to Kaggle, my score is way off (sometimes as high as 4, but mostly I never got below 0.69). Any ideas what could be going wrong?\r\n\r\nFYI, here is my code for writing the predictions to file (using Keras & TF):\r\n\r\n predictions = model.predict(test, verbose=0)\r\n preds = pd.DataFrame({\r\n \"label\": map(lambda x: x[0], predictions)\r\n })\r\n preds.index += 1\r\n preds.to_csv('submissions/submission.csv', index_label=\"id\")\r\n\r\nThanks!", "155667": "Hi Anas,\r\nRemember the score is calculated as log loss. This means you will be heavily penalized if you use high certainty predictions such as 1 and 0.\r\n\r\nJeremy Howard explains this far better than I can. Watch [this][1] from 30 minutes for an explanation.\r\n\r\n [1]: https://www.youtube.com/watch?v=e3aM6XTekJc\r\n\r\nI found i got a much better score when i clipped my prediction\r\ne.g (python) . np.clip(predictions[:,1],0.0125,0.9875)\r\n\r\nRgds\r\nIan", "155703": "Hi Ian, Thank you for your reply. I understand the log loss penalizes absolute predictions, and I've already tried replacing 1.0 by 0.95 and 0.0 by 0.05, but that didn't help (much). I'm also use data augmentation through Keras' ImageDataGenerator, which helps in training and validation, but still not on testing.", "157241": "I've also tried running [this code sample][1] from Keras almost as is (on the full dataset), with the only additions the above code for generating predictions and Ian's line for clipping low/high probabilities. \r\nAnd yet I still get a log loss score of 1.4+ on Kaggle.. That makes me think there's an issue with generating the predictions. Any help/ideas would be awesome! Thanks :)\r\n\r\n\r\n [1]: https://gist.github.com/fchollet/0830affa1f7f19fd47b06d4cf89ed44d", "157353": "Anas B The Keras' ImageDataGenerator don't conserve the original order of the id's and I think that this might be your problem when creating the submission DataFrame .\r\n\r\nI've created [a tutorial][1] and the last notebook covers how to use Keras' ImageDataGenerator to create a DataFrame in the right order.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/forums/t/27950/tutorial-how-to-get-0-06-loss-in-70-lines-of-code?forumMessageId=157349#post157349", "157384": "David de la Iglesia Castro That helped, thank you so much!! The issue was indeed with the predictions output, and using your technique for writing the csv based on the id in the filename fixed my problem. Thank you!!", "160280": "Thanks! with your tip I boosted my score up to 0.049 (from 0.14 before)", "160283": "Nice, good job! Did you use any special techniques, like data augmentation or normalization? Or just a simple convnet trained from scratch?", "160375": "I used the pre-trained Inception-v3 available in TensorFlow to extract features and then a linear SVM classifier... I acknowledge it's a bit cheating...\nWith a personal convnet with no data augmentation I was quite far.", "161912": "David de la Iglesia Castro I am not using Data Augmentation. My accuracy on validation data is 89% and log loss is 0.23 using CNN, however submitting the result of predictions on Kaggle gives 10+ log loss. What mistake i could have been doing possibly?", "220434": "I followed the tip of jeremy howard.\nI have clipped my prediction accuracies between 0.01 and 0.99 since my validation loss was as high as 99.97.\nBut at the leaderboard there is no improvement in the scores.\nCan you please suggest any other trick to try?", "1050474": "For anyone who might still come to this threat. If you download the testdata from kaggle it is not ordered correctly due to the naming. You can fix this as follows:\n\nfor i in os.listdir(test_dir):\n os.rename(test_dir + i, test_dir + i.split('.')[0].zfill(6) + '.jpg')\n test_images = [test_dir+i for i in os.listdir(test_dir)]\n test_images.sort()" }, "source": "meta" }