I am having a lot of trouble with the submission. The models I have trained have an area under roc curve score of 0.9 in training set and very slightly less in the CV and test sets. However, when I submit, I get evaluation scores of around 0.5, which is the same as a random submission.
\nI have gone through my code in a fine toothed comb over the past couple of days, and cannot find any bugs that will resolve this. The IDs are being carried through properly. I have compared the raw data, the feature added data, and the design matrix, and they all line up according to the IDs.
\nI have even tried submitting in the exact same order as the provided evaluation set. Still no luck.
\nHas anyone else experienced this? If so, how is it resolved?
\np.s. I am using the Python Anaconda environment.
", "messages": [ { "id": "80169", "postDate": "05/27/2015 18:44:45", "content": "I am having a lot of trouble with the submission. The models I have trained have an area under roc curve score of 0.9 in training set and very slightly less in the CV and test sets. However, when I submit, I get evaluation scores of around 0.5, which is the same as a random submission.
\nI have gone through my code in a fine toothed comb over the past couple of days, and cannot find any bugs that will resolve this. The IDs are being carried through properly. I have compared the raw data, the feature added data, and the design matrix, and they all line up according to the IDs.
\nI have even tried submitting in the exact same order as the provided evaluation set. Still no luck.
\nHas anyone else experienced this? If so, how is it resolved?
\np.s. I am using the Python Anaconda environment.
", "rawMarkdown": "", "votes": null }, { "id": "80170", "postDate": "05/27/2015 18:50:27", "content": "Maybe paste a few lines of your submission here for us to have a look?
", "rawMarkdown": "", "votes": null }, { "id": "80180", "postDate": "05/27/2015 19:13:54", "content": "[quote=cfh;80170]
\nMaybe paste a few lines of your submission here for us to have a look?
\n[/quote]
\nIsn't that against the rules?
", "rawMarkdown": "", "votes": null }, { "id": "80186", "postDate": "05/27/2015 20:12:19", "content": "Does your prediction consist entirely of 0s and 1s? I accidentally did that on my first submission (submitted the most likely classifier instead of a related probability), and my score was below 0.6.
\nMy submissions now look like 0.1234, 0.9876, 0.5432 or whatever rather than 0,1,1. Of course you won't get a perfect score that way, but it mitigates some big false negative/positive penalties.
\nJust a possibility.
", "rawMarkdown": "", "votes": null }, { "id": "80187", "postDate": "05/27/2015 20:14:30", "content": "Nope, I am definitely submitting the probabilities.I check each time. My numbers are all between 0 and 1, non inclusive.
", "rawMarkdown": "", "votes": null }, { "id": "80188", "postDate": "05/27/2015 20:18:28", "content": "[quote=ChipMonkey;80186]
\nDoes your prediction consist entirely of 0s and 1s? I accidentally did that on my first submission (submitted the most likely classifier instead of a related probability), and my score was below 0.6.
\nMy submissions now look like 0.1234, 0.9876, 0.5432 or whatever rather than 0,1,1. Of course you won't get a perfect score that way, but it mitigates some big false negative/positive penalties.
\nJust a possibility.
\n[/quote]
\n\nNope, I am definitely submitting probabilities. Here is the score for one of the bidders. 0.033623940355661144. They are all similar in the number of decimal places.
", "rawMarkdown": "", "votes": null }, { "id": "80191", "postDate": "05/27/2015 20:32:41", "content": "I do think it's easy to overtrain here. My training scores are actually lower than my public scores, which is striking me as odd, but they're still about 0.05 different (0.0848 train scores 0.905 on the leaderboard). I tend to do a histogram of my training predictions to compare against my test predictions to make sure the order of magnitude looks right (most of your guesses should be near 0, given the skew of the outcome variable). Otherwise I'm not sure what else to look at.
\nI'm using R, but that shouldn't matter really. I don't know if people would frown on you sharing code that scores poorly this late in the game, but you never know. If you could strip anything special out and post a simple version that recreates the problem you might get some help.
\nSorry, that's probably not a lot of help. Good luck.
", "rawMarkdown": "", "votes": null }, { "id": "80199", "postDate": "05/27/2015 21:58:55", "content": "[quote=saq7;80180]
\n[quote=cfh;80170]
\nMaybe paste a few lines of your submission here for us to have a look?
\n[/quote]
\nIsn't that against the rules?
\n[/quote]
\nI'm not aware of that, which rule would that be? I thought only posting code is forbidden? Surely nobody could glean anything useful from a few lines of your CSV file.
", "rawMarkdown": "", "votes": null }, { "id": "80200", "postDate": "05/27/2015 22:09:52", "content": "> I thought only posting code is forbidden?
\n>> Sharing code or data outside of teams is not permitted. This includes making code or data made available to all participants, such as on the forums.
\n> Surely nobody could glean anything useful from a few lines of your CSV file.
\nI do not think it matters if the data is useful or not. Posting part of your solution's CSV data is probably the closest you are going to get to making data (whatever the official interpretation of that may be) available to all on the forums.
", "rawMarkdown": "", "votes": null }, { "id": "80202", "postDate": "05/27/2015 22:37:43", "content": "You're sure you have not accidentally picked the sample file for submission?
", "rawMarkdown": "", "votes": null }, { "id": "80203", "postDate": "05/27/2015 22:39:02", "content": "[quote=skyl;80202]
\nYou're sure you have not accidentally picked the sample file for submission?
\n[/quote]
\nYeah I am sure. The submission file has the correct number of rows - 4700 (plus header).
", "rawMarkdown": "", "votes": null }, { "id": "80204", "postDate": "05/27/2015 22:40:17", "content": "[quote=saq7;80203]
\n[quote=skyl;80202]
\nYou're sure you have not accidentally picked the sample file for submission?
\n[/quote]
\nYeah I am sure. The submission file has the correct number of rows - 4700 (plus header).
\n[/quote]
\nJust realized that the sample also has 4700 rows. However I am using the file called test.csv
", "rawMarkdown": "", "votes": null }, { "id": "80223", "postDate": "05/28/2015 07:59:31", "content": "[quote=Triskelion;80200]
\nI do not think it matters if the data is useful or not. Posting part of your solution's CSV data is probably the closest you are going to get to making data (whatever the official interpretation of that may be) available to all on the forums.
\n[/quote]
\nIn my opinion one shouldn't be overly legalistic in these matters and interpret the spirit of the rules rather than stick to their letter religiously, but it's clear that your view is different here.
\n\nTo the OP: you probably checked for this, but there are several bidders both in the training and in the test data which do not appear in the bids data. Make sure that this doesn't create any off-by-one errors when you create your submission.
", "rawMarkdown": "", "votes": null }, { "id": "80228", "postDate": "05/28/2015 09:19:14", "content": "Is it possible that you've mixed up your bidder_ids, so each predicted probability doesn't actually match up with the bidder_id that it's intended to?
", "rawMarkdown": "", "votes": null }, { "id": "80259", "postDate": "05/28/2015 15:56:31", "content": "[quote=Snails;80228]
\nIs it possible that you've mixed up your bidder_ids, so each predicted probability doesn't actually match up with the bidder_id that it's intended to?
\n[/quote]
\nI have checked many times and this is definitely not the case.
", "rawMarkdown": "", "votes": null }, { "id": "80260", "postDate": "05/28/2015 15:57:32", "content": "[quote=cfh;80223]
\n[quote=Triskelion;80200]
\nI do not think it matters if the data is useful or not. Posting part of your solution's CSV data is probably the closest you are going to get to making data (whatever the official interpretation of that may be) available to all on the forums.
\n[/quote]
\nIn my opinion one shouldn't be overly legalistic in these matters and interpret the spirit of the rules rather than stick to their letter religiously, but it's clear that your view is different here.
\nTo the OP: you probably checked for this, but there are several bidders both in the training and in the test data which do not appear in the bids data. Make sure that this doesn't create any off-by-one errors when you create your submission.
\n[/quote]
\nYeah, I have checked for this too. :(
", "rawMarkdown": "", "votes": null }, { "id": "80268", "postDate": "05/28/2015 17:03:33", "content": "Are you sure you are sending the probability of being a robot? meaning that you are sending the prediction of the 1 class.
", "rawMarkdown": "", "votes": null }, { "id": "80270", "postDate": "05/28/2015 17:06:44", "content": "[quote=ma2afify;80268]
\nAre you sure you are sending the probability of being a robot? meaning that you are sending the prediction of the 1 class.
\n[/quote]
\nYes, I have checked this as well, as per the documentation of the sklearn function. (as well as the source code).
\nI have also tried submitting probabilities of the other class, and the score remains about the same.
", "rawMarkdown": "", "votes": null }, { "id": "80275", "postDate": "05/28/2015 17:19:23", "content": ""Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth." If your submission file is fine in its formatting, but you still get scores at around 0.5 when submitting, denoting random submitted probabilities, there's only one possible simple explanation: a misalignment between bidder_id and prediction. Did you shuffle your test data?
", "rawMarkdown": "", "votes": null }, { "id": "80277", "postDate": "05/28/2015 17:22:47", "content": "[quote=saq7;80259]
\n[quote=Snails;80228]
\nIs it possible that you've mixed up your bidder_ids, so each predicted probability doesn't actually match up with the bidder_id that it's intended to?
\n[/quote]
\nI have checked many times and this is definitely not the case.
\n[/quote]
\n\nThere are only two columns in the submission file and because of this there are only a few ways to get it wrong. You can either get the wrong probability placed in each bidder_id or you have just have a wrong probability in general which is a modeling issue.
\n\nYou can check your model or that your probabilities are in the right place. If you have done bug checking you find out that sometimes even though you checked a code many times it ends up wrong for a subtle reason. There are bugs suddenly found on code that has had years and hundreds of eyes looking at it.
", "rawMarkdown": "", "votes": null }, { "id": "80278", "postDate": "05/28/2015 17:24:20", "content": "[quote=Luca Massaron;80275]
\n"Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth." If your submission file is fine in its formatting, but you still get scores at around 0.5 when submitting, denoting random submitted probabilities, there's only one possible simple explanation: a misalignment between bidder_id and prediction. Did you shuffle your test data?
\n[/quote]
\nLove that quote, and no I didn't shuffle the test data. I haven't been able to find any point where the bidder_id and prediction might get mis-matched.
\nThe only possibility now might be that my models are shit. Though even if they are, they should still be better than random. I have engineered thoughtful features, created the design matrix properly, run the algorithms correctly, used cross validation to tune the parameters so that the training data is not being overfit (and cv score tracks the training score)...
", "rawMarkdown": "", "votes": null }, { "id": "80282", "postDate": "05/28/2015 17:41:15", "content": ""Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth."
\nGood quote, but I still question your conclusion, Luca, because there's a lot left to be eliminated.
\nIt's possible, for example, that your model, saq7, simply isn't better than random. This could be due to severe over-fitting (I had this problem using IP address too specifically), or just a bug in the model, training, or prediction. Are each of your input variables in the same order, from train to test? Are you definitely training on the "outcome" variable? I use array indexes and they differ between test.csv and train.csv due to the outcome variable (which is not in test), and that gets me from time to time. It could be a million things.
\nI see you (saq7) haven't finished any other Kaggle competitions -- perhaps you should run some submissions through the training competitions just to be sure you're doing everything else properly in general? Luca is correct -- if you've eliminated all the obvious things we're throwing at you it must be more fundamental. This competition (since it's closed to code sharing) is probably not the best one on which to work it out. :-\\
\nUpdate: Ah, I see you've already considered this... the comments hit page 2. Oh well.
", "rawMarkdown": "", "votes": null }, { "id": "80293", "postDate": "05/28/2015 18:10:01", "content": "Ok, I stripped the training data of all the features I had created. This stripped down data was giving a score of 0.2 LESS than the full data (still significantly above random) when all the features were included. The submission score did not change though, so the problem does not lie with the feature engineering.
", "rawMarkdown": "", "votes": null }, { "id": "80295", "postDate": "05/28/2015 18:26:48", "content": "I also tried submitting the probabilities of the 0 class (human). I expected the score to be 1 - prev_score. But it remained more or less the same (same till the 3 decimal place). However, when I compare to the csv files for the 0 class submission and the 1 class submission, the scores for each bidder_id add up to ~ 1.0. But still the submission score on Kaggle remains the same between the two files.
", "rawMarkdown": "", "votes": null }, { "id": "80302", "postDate": "05/28/2015 20:26:10", "content": "Weird! Do you maybe have negative probabilities? Or perhaps some values above 1? Perhaps some of your values driving your results to be wrong (a few wrong predictions you are confident about can add up with a small dataset). But inverting your probabilities really should have an effect, unless I do not understand the evaluation metric. Or I guess that would also be the case if a lot of your probabilities are close to 50%. It is too bad you can't post your code or file, would be much easier to debug.
", "rawMarkdown": "", "votes": null }, { "id": "80303", "postDate": "05/28/2015 20:31:15", "content": "[quote=Jared Turkewitz;80302]
\nWeird! Do you maybe have negative probabilities? Or perhaps some values above 1? Perhaps some of your values driving your results to be wrong (a few wrong predictions you are confident about can add up with a small dataset). But inverting your probabilities really should have an effect, unless I do not understand the evaluation metric. Or I guess that would also be the case if a lot of your probabilities are close to 50%. It is too bad you can't post your code or file, would be much easier to debug.
\n[/quote]
\nAll the values are between 0 and 1, so no negative values or values greater than 1. And no they aren't all close to 50%, the spread is close to that of the training data. I wonder if this might be a problem with Kaggle's submission parser...
", "rawMarkdown": "", "votes": null }, { "id": "80313", "postDate": "05/28/2015 21:42:23", "content": "[quote=saq7;80303]
\n[quote=Jared Turkewitz;80302]
\nWeird! Do you maybe have negative probabilities? Or perhaps some values above 1? Perhaps some of your values driving your results to be wrong (a few wrong predictions you are confident about can add up with a small dataset). But inverting your probabilities really should have an effect, unless I do not understand the evaluation metric. Or I guess that would also be the case if a lot of your probabilities are close to 50%. It is too bad you can't post your code or file, would be much easier to debug.
\n[/quote]
\nAll the values are between 0 and 1, so no negative values or values greater than 1. And no they aren't all close to 50%, the spread is close to that of the training data. I wonder if this might be a problem with Kaggle's submission parser...
\n[/quote]
\nThe parser works great given your submission follows the assumptions that a submission must follow (proper probabilities, CSV). The ids don't have to even be given in any specific order.
", "rawMarkdown": "", "votes": null }, { "id": "80315", "postDate": "05/28/2015 22:14:35", "content": "Are you sure you aren't leaking from your train to holdout set? I mean really really sure? It happens to everyone. I can see three of your scores. 0.412 and 0.588 are your first 2. I'm guessing these are the same submission with the order inverted. Your third score is 0.599. These are not likely to be random given the evaluation metric.
", "rawMarkdown": "", "votes": null }, { "id": "80323", "postDate": "05/29/2015 00:04:57", "content": "[quote=Jason Sumpter;80315]
\nAre you sure you aren't leaking from your train to holdout set? I mean really really sure? It happens to everyone. I can see three of your scores. 0.412 and 0.588 are your first 2. I'm guessing these are the same submission with the order inverted. Your third score is 0.599. These are not likely to be random given the evaluation metric.
\n[/quote]
\nHmm... let me look into that. Thanks for comment.
", "rawMarkdown": "", "votes": null }, { "id": "80427", "postDate": "05/30/2015 01:18:56", "content": "[quote=ChipMonkey;80186]
\nDoes your prediction consist entirely of 0s and 1s? I accidentally did that on my first submission (submitted the most likely classifier instead of a related probability), and my score was below 0.6.
\nMy submissions now look like 0.1234, 0.9876, 0.5432 or whatever rather than 0,1,1. Of course you won't get a perfect score that way, but it mitigates some big false negative/positive penalties.
\nJust a possibility.
\n[/quote]
\n\nnot sure if that was in the instructions but that helped my score. my prediction still stinks (only like 69% ATM) but that's def' what they're looking for. i must have missed that in the instructions. ...guess i'm just saying "thank you" for pointing that out! :)
", "rawMarkdown": "", "votes": null }, { "id": "80428", "postDate": "05/30/2015 01:40:37", "content": "I'd be willing to bet that, since the training set is relatively small (with only 103 "robots" and over 1800 "humans"), that you're just overfitting the training set.
\nAre you using a separate validation set from the training data? That might help you determine how well your model works for new data.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 80170, "author_name": "cfhlearns", "author_url": "", "post_date": "05/27/2015 18:50:27", "content": "Maybe paste a few lines of your submission here for us to have a look?
", "votes": null, "replies": [] }, { "id": 80180, "author_name": "saquibk", "author_url": "", "post_date": "05/27/2015 19:13:54", "content": "[quote=cfh;80170]
\nMaybe paste a few lines of your submission here for us to have a look?
\n[/quote]
\nIsn't that against the rules?
", "votes": null, "replies": [] }, { "id": 80186, "author_name": "chipmonkey", "author_url": "", "post_date": "05/27/2015 20:12:19", "content": "Does your prediction consist entirely of 0s and 1s? I accidentally did that on my first submission (submitted the most likely classifier instead of a related probability), and my score was below 0.6.
\nMy submissions now look like 0.1234, 0.9876, 0.5432 or whatever rather than 0,1,1. Of course you won't get a perfect score that way, but it mitigates some big false negative/positive penalties.
\nJust a possibility.
", "votes": null, "replies": [] }, { "id": 80187, "author_name": "saquibk", "author_url": "", "post_date": "05/27/2015 20:14:30", "content": "Nope, I am definitely submitting the probabilities.I check each time. My numbers are all between 0 and 1, non inclusive.
", "votes": null, "replies": [] }, { "id": 80188, "author_name": "saquibk", "author_url": "", "post_date": "05/27/2015 20:18:28", "content": "[quote=ChipMonkey;80186]
\nDoes your prediction consist entirely of 0s and 1s? I accidentally did that on my first submission (submitted the most likely classifier instead of a related probability), and my score was below 0.6.
\nMy submissions now look like 0.1234, 0.9876, 0.5432 or whatever rather than 0,1,1. Of course you won't get a perfect score that way, but it mitigates some big false negative/positive penalties.
\nJust a possibility.
\n[/quote]
\n\nNope, I am definitely submitting probabilities. Here is the score for one of the bidders. 0.033623940355661144. They are all similar in the number of decimal places.
", "votes": null, "replies": [] }, { "id": 80191, "author_name": "chipmonkey", "author_url": "", "post_date": "05/27/2015 20:32:41", "content": "I do think it's easy to overtrain here. My training scores are actually lower than my public scores, which is striking me as odd, but they're still about 0.05 different (0.0848 train scores 0.905 on the leaderboard). I tend to do a histogram of my training predictions to compare against my test predictions to make sure the order of magnitude looks right (most of your guesses should be near 0, given the skew of the outcome variable). Otherwise I'm not sure what else to look at.
\nI'm using R, but that shouldn't matter really. I don't know if people would frown on you sharing code that scores poorly this late in the game, but you never know. If you could strip anything special out and post a simple version that recreates the problem you might get some help.
\nSorry, that's probably not a lot of help. Good luck.
", "votes": null, "replies": [] }, { "id": 80199, "author_name": "cfhlearns", "author_url": "", "post_date": "05/27/2015 21:58:55", "content": "[quote=saq7;80180]
\n[quote=cfh;80170]
\nMaybe paste a few lines of your submission here for us to have a look?
\n[/quote]
\nIsn't that against the rules?
\n[/quote]
\nI'm not aware of that, which rule would that be? I thought only posting code is forbidden? Surely nobody could glean anything useful from a few lines of your CSV file.
", "votes": null, "replies": [] }, { "id": 80200, "author_name": "triskelion", "author_url": "", "post_date": "05/27/2015 22:09:52", "content": "> I thought only posting code is forbidden?
\n>> Sharing code or data outside of teams is not permitted. This includes making code or data made available to all participants, such as on the forums.
\n> Surely nobody could glean anything useful from a few lines of your CSV file.
\nI do not think it matters if the data is useful or not. Posting part of your solution's CSV data is probably the closest you are going to get to making data (whatever the official interpretation of that may be) available to all on the forums.
", "votes": null, "replies": [] }, { "id": 80202, "author_name": "kiangchia", "author_url": "", "post_date": "05/27/2015 22:37:43", "content": "You're sure you have not accidentally picked the sample file for submission?
", "votes": null, "replies": [] }, { "id": 80203, "author_name": "saquibk", "author_url": "", "post_date": "05/27/2015 22:39:02", "content": "[quote=skyl;80202]
\nYou're sure you have not accidentally picked the sample file for submission?
\n[/quote]
\nYeah I am sure. The submission file has the correct number of rows - 4700 (plus header).
", "votes": null, "replies": [] }, { "id": 80204, "author_name": "saquibk", "author_url": "", "post_date": "05/27/2015 22:40:17", "content": "[quote=saq7;80203]
\n[quote=skyl;80202]
\nYou're sure you have not accidentally picked the sample file for submission?
\n[/quote]
\nYeah I am sure. The submission file has the correct number of rows - 4700 (plus header).
\n[/quote]
\nJust realized that the sample also has 4700 rows. However I am using the file called test.csv
", "votes": null, "replies": [] }, { "id": 80223, "author_name": "cfhlearns", "author_url": "", "post_date": "05/28/2015 07:59:31", "content": "[quote=Triskelion;80200]
\nI do not think it matters if the data is useful or not. Posting part of your solution's CSV data is probably the closest you are going to get to making data (whatever the official interpretation of that may be) available to all on the forums.
\n[/quote]
\nIn my opinion one shouldn't be overly legalistic in these matters and interpret the spirit of the rules rather than stick to their letter religiously, but it's clear that your view is different here.
\n\nTo the OP: you probably checked for this, but there are several bidders both in the training and in the test data which do not appear in the bids data. Make sure that this doesn't create any off-by-one errors when you create your submission.
", "votes": null, "replies": [] }, { "id": 80228, "author_name": "tyronecragg", "author_url": "", "post_date": "05/28/2015 09:19:14", "content": "Is it possible that you've mixed up your bidder_ids, so each predicted probability doesn't actually match up with the bidder_id that it's intended to?
", "votes": null, "replies": [] }, { "id": 80259, "author_name": "saquibk", "author_url": "", "post_date": "05/28/2015 15:56:31", "content": "[quote=Snails;80228]
\nIs it possible that you've mixed up your bidder_ids, so each predicted probability doesn't actually match up with the bidder_id that it's intended to?
\n[/quote]
\nI have checked many times and this is definitely not the case.
", "votes": null, "replies": [] }, { "id": 80260, "author_name": "saquibk", "author_url": "", "post_date": "05/28/2015 15:57:32", "content": "[quote=cfh;80223]
\n[quote=Triskelion;80200]
\nI do not think it matters if the data is useful or not. Posting part of your solution's CSV data is probably the closest you are going to get to making data (whatever the official interpretation of that may be) available to all on the forums.
\n[/quote]
\nIn my opinion one shouldn't be overly legalistic in these matters and interpret the spirit of the rules rather than stick to their letter religiously, but it's clear that your view is different here.
\nTo the OP: you probably checked for this, but there are several bidders both in the training and in the test data which do not appear in the bids data. Make sure that this doesn't create any off-by-one errors when you create your submission.
\n[/quote]
\nYeah, I have checked for this too. :(
", "votes": null, "replies": [] }, { "id": 80268, "author_name": "ma2afify", "author_url": "", "post_date": "05/28/2015 17:03:33", "content": "Are you sure you are sending the probability of being a robot? meaning that you are sending the prediction of the 1 class.
", "votes": null, "replies": [] }, { "id": 80270, "author_name": "saquibk", "author_url": "", "post_date": "05/28/2015 17:06:44", "content": "[quote=ma2afify;80268]
\nAre you sure you are sending the probability of being a robot? meaning that you are sending the prediction of the 1 class.
\n[/quote]
\nYes, I have checked this as well, as per the documentation of the sklearn function. (as well as the source code).
\nI have also tried submitting probabilities of the other class, and the score remains about the same.
", "votes": null, "replies": [] }, { "id": 80275, "author_name": "lucamassaron", "author_url": "", "post_date": "05/28/2015 17:19:23", "content": ""Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth." If your submission file is fine in its formatting, but you still get scores at around 0.5 when submitting, denoting random submitted probabilities, there's only one possible simple explanation: a misalignment between bidder_id and prediction. Did you shuffle your test data?
", "votes": null, "replies": [] }, { "id": 80277, "author_name": "jbsilva", "author_url": "", "post_date": "05/28/2015 17:22:47", "content": "[quote=saq7;80259]
\n[quote=Snails;80228]
\nIs it possible that you've mixed up your bidder_ids, so each predicted probability doesn't actually match up with the bidder_id that it's intended to?
\n[/quote]
\nI have checked many times and this is definitely not the case.
\n[/quote]
\n\nThere are only two columns in the submission file and because of this there are only a few ways to get it wrong. You can either get the wrong probability placed in each bidder_id or you have just have a wrong probability in general which is a modeling issue.
\n\nYou can check your model or that your probabilities are in the right place. If you have done bug checking you find out that sometimes even though you checked a code many times it ends up wrong for a subtle reason. There are bugs suddenly found on code that has had years and hundreds of eyes looking at it.
", "votes": null, "replies": [] }, { "id": 80278, "author_name": "saquibk", "author_url": "", "post_date": "05/28/2015 17:24:20", "content": "[quote=Luca Massaron;80275]
\n"Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth." If your submission file is fine in its formatting, but you still get scores at around 0.5 when submitting, denoting random submitted probabilities, there's only one possible simple explanation: a misalignment between bidder_id and prediction. Did you shuffle your test data?
\n[/quote]
\nLove that quote, and no I didn't shuffle the test data. I haven't been able to find any point where the bidder_id and prediction might get mis-matched.
\nThe only possibility now might be that my models are shit. Though even if they are, they should still be better than random. I have engineered thoughtful features, created the design matrix properly, run the algorithms correctly, used cross validation to tune the parameters so that the training data is not being overfit (and cv score tracks the training score)...
", "votes": null, "replies": [] }, { "id": 80282, "author_name": "chipmonkey", "author_url": "", "post_date": "05/28/2015 17:41:15", "content": ""Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth."
\nGood quote, but I still question your conclusion, Luca, because there's a lot left to be eliminated.
\nIt's possible, for example, that your model, saq7, simply isn't better than random. This could be due to severe over-fitting (I had this problem using IP address too specifically), or just a bug in the model, training, or prediction. Are each of your input variables in the same order, from train to test? Are you definitely training on the "outcome" variable? I use array indexes and they differ between test.csv and train.csv due to the outcome variable (which is not in test), and that gets me from time to time. It could be a million things.
\nI see you (saq7) haven't finished any other Kaggle competitions -- perhaps you should run some submissions through the training competitions just to be sure you're doing everything else properly in general? Luca is correct -- if you've eliminated all the obvious things we're throwing at you it must be more fundamental. This competition (since it's closed to code sharing) is probably not the best one on which to work it out. :-\\
\nUpdate: Ah, I see you've already considered this... the comments hit page 2. Oh well.
", "votes": null, "replies": [] }, { "id": 80293, "author_name": "saquibk", "author_url": "", "post_date": "05/28/2015 18:10:01", "content": "Ok, I stripped the training data of all the features I had created. This stripped down data was giving a score of 0.2 LESS than the full data (still significantly above random) when all the features were included. The submission score did not change though, so the problem does not lie with the feature engineering.
", "votes": null, "replies": [] }, { "id": 80295, "author_name": "saquibk", "author_url": "", "post_date": "05/28/2015 18:26:48", "content": "I also tried submitting the probabilities of the 0 class (human). I expected the score to be 1 - prev_score. But it remained more or less the same (same till the 3 decimal place). However, when I compare to the csv files for the 0 class submission and the 1 class submission, the scores for each bidder_id add up to ~ 1.0. But still the submission score on Kaggle remains the same between the two files.
", "votes": null, "replies": [] }, { "id": 80302, "author_name": "jturkewitz", "author_url": "", "post_date": "05/28/2015 20:26:10", "content": "Weird! Do you maybe have negative probabilities? Or perhaps some values above 1? Perhaps some of your values driving your results to be wrong (a few wrong predictions you are confident about can add up with a small dataset). But inverting your probabilities really should have an effect, unless I do not understand the evaluation metric. Or I guess that would also be the case if a lot of your probabilities are close to 50%. It is too bad you can't post your code or file, would be much easier to debug.
", "votes": null, "replies": [] }, { "id": 80303, "author_name": "saquibk", "author_url": "", "post_date": "05/28/2015 20:31:15", "content": "[quote=Jared Turkewitz;80302]
\nWeird! Do you maybe have negative probabilities? Or perhaps some values above 1? Perhaps some of your values driving your results to be wrong (a few wrong predictions you are confident about can add up with a small dataset). But inverting your probabilities really should have an effect, unless I do not understand the evaluation metric. Or I guess that would also be the case if a lot of your probabilities are close to 50%. It is too bad you can't post your code or file, would be much easier to debug.
\n[/quote]
\nAll the values are between 0 and 1, so no negative values or values greater than 1. And no they aren't all close to 50%, the spread is close to that of the training data. I wonder if this might be a problem with Kaggle's submission parser...
", "votes": null, "replies": [] }, { "id": 80313, "author_name": "jbsilva", "author_url": "", "post_date": "05/28/2015 21:42:23", "content": "[quote=saq7;80303]
\n[quote=Jared Turkewitz;80302]
\nWeird! Do you maybe have negative probabilities? Or perhaps some values above 1? Perhaps some of your values driving your results to be wrong (a few wrong predictions you are confident about can add up with a small dataset). But inverting your probabilities really should have an effect, unless I do not understand the evaluation metric. Or I guess that would also be the case if a lot of your probabilities are close to 50%. It is too bad you can't post your code or file, would be much easier to debug.
\n[/quote]
\nAll the values are between 0 and 1, so no negative values or values greater than 1. And no they aren't all close to 50%, the spread is close to that of the training data. I wonder if this might be a problem with Kaggle's submission parser...
\n[/quote]
\nThe parser works great given your submission follows the assumptions that a submission must follow (proper probabilities, CSV). The ids don't have to even be given in any specific order.
", "votes": null, "replies": [] }, { "id": 80315, "author_name": "jasonsumpter", "author_url": "", "post_date": "05/28/2015 22:14:35", "content": "Are you sure you aren't leaking from your train to holdout set? I mean really really sure? It happens to everyone. I can see three of your scores. 0.412 and 0.588 are your first 2. I'm guessing these are the same submission with the order inverted. Your third score is 0.599. These are not likely to be random given the evaluation metric.
", "votes": null, "replies": [] }, { "id": 80323, "author_name": "saquibk", "author_url": "", "post_date": "05/29/2015 00:04:57", "content": "[quote=Jason Sumpter;80315]
\nAre you sure you aren't leaking from your train to holdout set? I mean really really sure? It happens to everyone. I can see three of your scores. 0.412 and 0.588 are your first 2. I'm guessing these are the same submission with the order inverted. Your third score is 0.599. These are not likely to be random given the evaluation metric.
\n[/quote]
\nHmm... let me look into that. Thanks for comment.
", "votes": null, "replies": [] }, { "id": 80427, "author_name": "messels", "author_url": "", "post_date": "05/30/2015 01:18:56", "content": "[quote=ChipMonkey;80186]
\nDoes your prediction consist entirely of 0s and 1s? I accidentally did that on my first submission (submitted the most likely classifier instead of a related probability), and my score was below 0.6.
\nMy submissions now look like 0.1234, 0.9876, 0.5432 or whatever rather than 0,1,1. Of course you won't get a perfect score that way, but it mitigates some big false negative/positive penalties.
\nJust a possibility.
\n[/quote]
\n\nnot sure if that was in the instructions but that helped my score. my prediction still stinks (only like 69% ATM) but that's def' what they're looking for. i must have missed that in the instructions. ...guess i'm just saying "thank you" for pointing that out! :)
", "votes": null, "replies": [] }, { "id": 80428, "author_name": "marcogiancarli", "author_url": "", "post_date": "05/30/2015 01:40:37", "content": "I'd be willing to bet that, since the training set is relatively small (with only 103 "robots" and over 1800 "humans"), that you're just overfitting the training set.
\nAre you using a separate validation set from the training data? That might help you determine how well your model works for new data.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "80169": "", "80170": "", "80180": "", "80186": "", "80187": "", "80188": "", "80191": "", "80199": "", "80200": "", "80202": "", "80203": "", "80204": "", "80223": "", "80228": "", "80259": "", "80260": "", "80268": "", "80270": "", "80275": "", "80277": "", "80278": "", "80282": "", "80293": "", "80295": "", "80302": "", "80303": "", "80313": "", "80315": "", "80323": "", "80427": "", "80428": "" }, "source": "meta" }