Hi,\nI meet a problem that all my submissions score below 0.5.
\n\nWhen I forked the code in script "Low freq 0.80+ refactored" and ran it on Kaggle platform I can get a score a little higher than 0.8, but when I download it and run exactly this same code on my GPU machine the final submission scores only 0.37.
\n\nAgain, I downloaded Tim's code in script "Naive NNet 0.71+ / 0.90+" and made a few modifications and run it on my GPU machine. The validation score is above 0.9. But when I submit the csv result, it scores only 0.43. The training and validation use separate series of data.
\n\nI'm almost crazy about this. Can anyone tell me why? Thanks a lot!
", "messages": [ { "id": "88872", "postDate": "08/08/2015 01:02:57", "content": "Hi,\nI meet a problem that all my submissions score below 0.5.
\n\nWhen I forked the code in script "Low freq 0.80+ refactored" and ran it on Kaggle platform I can get a score a little higher than 0.8, but when I download it and run exactly this same code on my GPU machine the final submission scores only 0.37.
\n\nAgain, I downloaded Tim's code in script "Naive NNet 0.71+ / 0.90+" and made a few modifications and run it on my GPU machine. The validation score is above 0.9. But when I submit the csv result, it scores only 0.43. The training and validation use separate series of data.
\n\nI'm almost crazy about this. Can anyone tell me why? Thanks a lot!
", "rawMarkdown": "Hi,\r\nI meet a problem that all my submissions score below 0.5. \r\n\r\nWhen I forked the code in script \"Low freq 0.80+ refactored\" and ran it on Kaggle platform I can get a score a little higher than 0.8, but when I download it and run exactly this same code on my GPU machine the final submission scores only 0.37.\r\n\r\nAgain, I downloaded Tim's code in script \"Naive NNet 0.71+ / 0.90+\" and made a few modifications and run it on my GPU machine. The validation score is above 0.9. But when I submit the csv result, it scores only 0.43. The training and validation use separate series of data.\r\n\r\nI'm almost crazy about this. Can anyone tell me why? Thanks a lot!", "votes": null }, { "id": "88875", "postDate": "08/08/2015 01:18:56", "content": "It looks like there may be a couple of issues here. First make sure your algorithm is returning float probabilities and not integers. Second some algorithms return the probabilities for both classes you only need the probability for the positive class. This error will reverse your score(a .7 will become a .3). But if you are getting .9 validation and .43 leaderboard there is another issue as this is not a simple reversal(also I think kaggle automatically reverses your score if it is below .25).
", "rawMarkdown": "It looks like there may be a couple of issues here. First make sure your algorithm is returning float probabilities and not integers. Second some algorithms return the probabilities for both classes you only need the probability for the positive class. This error will reverse your score(a .7 will become a .3). But if you are getting .9 validation and .43 leaderboard there is another issue as this is not a simple reversal(also I think kaggle automatically reverses your score if it is below .25).", "votes": null }, { "id": "88884", "postDate": "08/08/2015 03:13:19", "content": "Just download dataset again! It changed some time ago...
", "rawMarkdown": "Just download dataset again! It changed some time ago...", "votes": null }, { "id": "88904", "postDate": "08/08/2015 15:25:35", "content": "@Gilberto Titericz Junior, thank you very much! The training data have changed. I downloaded data again and everything seems alright.
\n\n@Devin, thank you very much too!
", "rawMarkdown": "Gilberto Titericz Junior, thank you very much! The training data have changed. I downloaded data again and everything seems alright. \r\n\r\n@Devin, thank you very much too!", "votes": null }, { "id": "88913", "postDate": "08/08/2015 17:13:50", "content": "Wait, Can someone please go into more detail on the training data changing ? What do you mean by that ? I have been working on data that I downloaded over a week ago. I am running into problems that I was about to post about but, if it's a matter of bad training data I will want to do that first,
\n\nChris
", "rawMarkdown": "Wait, Can someone please go into more detail on the training data changing ? What do you mean by that ? I have been working on data that I downloaded over a week ago. I am running into problems that I was about to post about but, if it's a matter of bad training data I will want to do that first, \r\n\r\nChris", "votes": null }, { "id": "88915", "postDate": "08/08/2015 17:46:56", "content": "@Chris
\n\nThe data was changed about 18 days ago see this thread for the official announcement
", "rawMarkdown": "Chris \r\n\r\nThe data was changed about 18 days ago see this [thread][1] for the official announcement\r\n\r\n\r\n [1]: https://www.kaggle.com/c/grasp-and-lift-eeg-detection/forums/t/15413/announcement-error-in-dataset", "votes": null }, { "id": "88926", "postDate": "08/08/2015 21:39:22", "content": "Thank you for the reply. I found the thread shortly after posting the comment. I have been off working on my own. I now know to check in on the site every so often. First project is a learning project in many ways. ;)
", "rawMarkdown": "Thank you for the reply. I found the thread shortly after posting the comment. I have been off working on my own. I now know to check in on the site every so often. First project is a learning project in many ways. ;)", "votes": null }, { "id": "89285", "postDate": "08/13/2015 09:07:04", "content": "Seems you are working with the wrong dataset. Try to redownload it.
", "rawMarkdown": "Seems you are working with the wrong dataset. Try to redownload it.", "votes": null } ], "comments": [ { "id": 88875, "author_name": "devinanzelmo", "author_url": "", "post_date": "08/08/2015 01:18:56", "content": "It looks like there may be a couple of issues here. First make sure your algorithm is returning float probabilities and not integers. Second some algorithms return the probabilities for both classes you only need the probability for the positive class. This error will reverse your score(a .7 will become a .3). But if you are getting .9 validation and .43 leaderboard there is another issue as this is not a simple reversal(also I think kaggle automatically reverses your score if it is below .25).
", "votes": null, "replies": [] }, { "id": 88884, "author_name": "titericz", "author_url": "", "post_date": "08/08/2015 03:13:19", "content": "Just download dataset again! It changed some time ago...
", "votes": null, "replies": [] }, { "id": 88904, "author_name": "jinfengfeng", "author_url": "", "post_date": "08/08/2015 15:25:35", "content": "@Gilberto Titericz Junior, thank you very much! The training data have changed. I downloaded data again and everything seems alright.
\n\n@Devin, thank you very much too!
", "votes": null, "replies": [] }, { "id": 88913, "author_name": "chrisfs", "author_url": "", "post_date": "08/08/2015 17:13:50", "content": "Wait, Can someone please go into more detail on the training data changing ? What do you mean by that ? I have been working on data that I downloaded over a week ago. I am running into problems that I was about to post about but, if it's a matter of bad training data I will want to do that first,
\n\nChris
", "votes": null, "replies": [] }, { "id": 88915, "author_name": "devinanzelmo", "author_url": "", "post_date": "08/08/2015 17:46:56", "content": "@Chris
\n\nThe data was changed about 18 days ago see this thread for the official announcement
", "votes": null, "replies": [] }, { "id": 88926, "author_name": "chrisfs", "author_url": "", "post_date": "08/08/2015 21:39:22", "content": "Thank you for the reply. I found the thread shortly after posting the comment. I have been off working on my own. I now know to check in on the site every so often. First project is a learning project in many ways. ;)
", "votes": null, "replies": [] }, { "id": 89285, "author_name": "vk4arm", "author_url": "", "post_date": "08/13/2015 09:07:04", "content": "Seems you are working with the wrong dataset. Try to redownload it.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "88872": "Hi,\r\nI meet a problem that all my submissions score below 0.5. \r\n\r\nWhen I forked the code in script \"Low freq 0.80+ refactored\" and ran it on Kaggle platform I can get a score a little higher than 0.8, but when I download it and run exactly this same code on my GPU machine the final submission scores only 0.37.\r\n\r\nAgain, I downloaded Tim's code in script \"Naive NNet 0.71+ / 0.90+\" and made a few modifications and run it on my GPU machine. The validation score is above 0.9. But when I submit the csv result, it scores only 0.43. The training and validation use separate series of data.\r\n\r\nI'm almost crazy about this. Can anyone tell me why? Thanks a lot!", "88875": "It looks like there may be a couple of issues here. First make sure your algorithm is returning float probabilities and not integers. Second some algorithms return the probabilities for both classes you only need the probability for the positive class. This error will reverse your score(a .7 will become a .3). But if you are getting .9 validation and .43 leaderboard there is another issue as this is not a simple reversal(also I think kaggle automatically reverses your score if it is below .25).", "88884": "Just download dataset again! It changed some time ago...", "88904": "Gilberto Titericz Junior, thank you very much! The training data have changed. I downloaded data again and everything seems alright. \r\n\r\n@Devin, thank you very much too!", "88913": "Wait, Can someone please go into more detail on the training data changing ? What do you mean by that ? I have been working on data that I downloaded over a week ago. I am running into problems that I was about to post about but, if it's a matter of bad training data I will want to do that first, \r\n\r\nChris", "88915": "Chris \r\n\r\nThe data was changed about 18 days ago see this [thread][1] for the official announcement\r\n\r\n\r\n [1]: https://www.kaggle.com/c/grasp-and-lift-eeg-detection/forums/t/15413/announcement-error-in-dataset", "88926": "Thank you for the reply. I found the thread shortly after posting the comment. I have been off working on my own. I now know to check in on the site every so often. First project is a learning project in many ways. ;)", "89285": "Seems you are working with the wrong dataset. Try to redownload it." }, "source": "meta" }