{ "id": 14178, "title": "How are you all gauging performance before submitting?", "url": "/competitions/facebook-recruiting-iv-human-or-bot/discussion/14178", "author_name": "", "post_date": "2015-05-12T20:24:02.390Z", "votes": null, "comment_count": 19, "views": 2829, "content": "

I'm calculating AUC via cross-validation but the scores are quite different when submitting to the leaderboard. e.g. AUC locally of .95 results in .85 in leaderboard.

", "messages": [ { "id": "78122", "postDate": "05/12/2015 20:24:02", "content": "

I'm calculating AUC via cross-validation but the scores are quite different when submitting to the leaderboard. e.g. AUC locally of .95 results in .85 in leaderboard.

", "rawMarkdown": "", "votes": null }, { "id": "78133", "postDate": "05/12/2015 22:12:02", "content": "

Overfitting ?

", "rawMarkdown": "", "votes": null }, { "id": "78134", "postDate": "05/12/2015 22:58:48", "content": "

I doing a similar process except I'm also checking to make sure the learning curve levels off and the roc plot does produce a similar value the the roc auc score calculated.  I see a drop too, but more like 5%.   

\n

I have not, but will start, holding out my own test set of about 300 values to estimate what it will be on the test set.   I'll respond will how closley these tests match the actual scores kaggle gave them once I get them done.

", "rawMarkdown": "", "votes": null }, { "id": "78154", "postDate": "05/13/2015 03:32:51", "content": "

any body facing a problem with Xgboost cv function? 

\n

I see that my own CV implemented using looping is more reflective of leaderboard performance, the auto function over estimates performance even when employing same number of folds as the other routine.

\n

On the other hand the auto function is much faster, don't know how it iterates so fast.

", "rawMarkdown": "", "votes": null }, { "id": "78189", "postDate": "05/13/2015 11:50:06", "content": "

You can try leaving a 30% for in time validation and then run your model with cross validation. This way you get some reflection of your score before hand.

", "rawMarkdown": "", "votes": null }, { "id": "78195", "postDate": "05/13/2015 12:45:48", "content": "

I do a k-fold cross validation and average the AUC scores from each fold.

\n

The results are reasonably close to the private leaderboard. 

", "rawMarkdown": "", "votes": null }, { "id": "79515", "postDate": "05/20/2015 20:25:54", "content": "

i did k-fold cross validation as well. My leadboard is 0.08 lower than local cv score for a model, which is just totally weird.

", "rawMarkdown": "", "votes": null }, { "id": "79532", "postDate": "05/20/2015 22:09:54", "content": "

I do a 10-fold cross-validation, also with results very close to the ones obtained in the public leaderboard (around 0.05 difference).

", "rawMarkdown": "", "votes": null }, { "id": "79549", "postDate": "05/21/2015 00:25:01", "content": "

I do a 10 times, 10-fold cross-validation, also with results very close to the ones obtained in the public leaderboard (around 0.025 difference).

", "rawMarkdown": "", "votes": null }, { "id": "79560", "postDate": "05/21/2015 02:21:11", "content": "

If you are using R, I strongly recommend you try caret to train your model, and choose "ROC" as the train metric. The results of caret are more stable compared to my random 70/30 split. And the most important thing is that the result is quite close to my leader board score(around 0.005 difference)

", "rawMarkdown": "", "votes": null }, { "id": "79565", "postDate": "05/21/2015 03:29:55", "content": "

[quote=Thársis T. P. Souza;79549]

\n

I do a 10 times, 10-fold cross-validation, also with results very close to the ones obtained in the public leaderboard (around 0.025 difference).

\n

[/quote]

\n

10 times 10 fold , that would mean a 100 runs? Are you tuning or simply estimating expected performance?

\n

For tuning, it would effectively mean 100 iterations for each try!

\n\n

I am trying out 20 fold, my leaderboard drops between 2.5 to 3% which is disappointing. Maybe I should verify using your approach.

", "rawMarkdown": "", "votes": null }, { "id": "79614", "postDate": "05/21/2015 14:01:43", "content": "

yeah, 100 iterations and it only takes a few minutes.anyway, I believe feature engineering is the key in this competition not the model/tuning itself.

", "rawMarkdown": "", "votes": null }, { "id": "79665", "postDate": "05/21/2015 20:03:15", "content": "

Model evaluation has been an issue for me ... Privately , i have built models that have gained a score of more than 0.90... but on Leader-board, I am only around 0.88... and the problem is: discrepancy between private scores and leader-board scores varies with model. for some, it is 5% and it is 1.5% for others. I am using stratified k fold and i have also tried shuffle split. I wonder if my score might move up once the competition gets over as PB score is on a certain 30% of test data. :)

\n

Or a more serious explanation could be that i am over-fitting training data with certain models more than others. I do agree that feature engineering is key here and not the model/tuning itself.

", "rawMarkdown": "", "votes": null }, { "id": "79667", "postDate": "05/21/2015 20:13:54", "content": "

[quote=KR;79665]

\n

Or a more serious explanation could be that i am over-fitting training data with certain models more than others. I do agree that feature engineering is key here and not the model/tuning itself.

\n

[/quote]

\n

In my experience, simpler models (logistic regression) tend to yield CV values closer to the actual leaderboard score after submission. I have had 0.92 AUC using boosting trees, but that was an illusion.

\n

I completely agree that feature extraction is much more critical than the actual classifier used. But isn't it almost always true? :-)

", "rawMarkdown": "", "votes": null }, { "id": "79683", "postDate": "05/21/2015 22:58:27", "content": "

[quote=José María Mateos;79667]

\n

[quote=KR;79665]

\n

Or a more serious explanation could be that i am over-fitting training data with certain models more than others. I do agree that feature engineering is key here and not the model/tuning itself.

\n

[/quote]

\n

In my experience, simpler models (logistic regression) tend to yield CV values closer to the actual leaderboard score after submission. I have had 0.92 AUC using boosting trees, but that was an illusion.

\n

I completely agree that feature extraction is much more critical than the actual classifier used. But isn't it almost always true? :-)

\n

[/quote]

\n\n

I do not know why i wrote that. :)

", "rawMarkdown": "", "votes": null }, { "id": "79768", "postDate": "05/22/2015 16:20:20", "content": "

I just joined the competition and made a first submission.

\n

I used a 10 fold stratified cross validation and got like 0.90 auc, but when I submitted I got 0.64 !!!

\n

I am sure there is a data leak in one of my features, but I was curious if one of you guys had the same problem.

\n

Thank you !

", "rawMarkdown": "", "votes": null }, { "id": "79769", "postDate": "05/22/2015 16:20:29", "content": "

I just joined the competition and made a first submission.

\n

I used a 10 fold stratified cross validation and got like 0.90 auc, but when I submitted I got 0.64 !!!

\n

I am sure there is a data leak in one of my features, but I was curious if one of you guys had the same problem.

\n

Thank you !

", "rawMarkdown": "", "votes": null }, { "id": "79792", "postDate": "05/22/2015 19:40:27", "content": "

[quote=José María Mateos;79667]

\n

[quote=KR;79665]

\n

Or a more serious explanation could be that i am over-fitting training data with certain models more than others. I do agree that feature engineering is key here and not the model/tuning itself.

\n

[/quote]

\n

In my experience, simpler models (logistic regression) tend to yield CV values closer to the actual leaderboard score after submission. I have had 0.92 AUC using boosting trees, but that was an illusion.

\n

I completely agree that feature extraction is much more critical than the actual classifier used. But isn't it almost always true? :-)

\n

[/quote]

\n\n

Model tuning and blending has increased my score from .890 to .919 on the same feature set, so I wouldnt say tuning isnt important, I might have been hill climbing, the private leaderboard will reveal that.

\n

I would wager that the top leaders would have almost similar features extracted by now - tuning/blending can give significant increases in the score.

", "rawMarkdown": "", "votes": null }, { "id": "79793", "postDate": "05/22/2015 19:48:38", "content": "

I believe the entire leadership board will change a lot on private board from the public board. Model blending can really help you out here but not necessarily will give you best result on rest 70%. The key is - You should nominate your best model regardless of Public score for evaluation.

", "rawMarkdown": "", "votes": null }, { "id": "79807", "postDate": "05/22/2015 23:15:26", "content": "

By blending you understand mixing in some way results from several models?

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 78133, "author_name": "carloshuertas", "author_url": "", "post_date": "05/12/2015 22:12:02", "content": "

Overfitting ?

", "votes": null, "replies": [] }, { "id": 78134, "author_name": "bryantravissmith", "author_url": "", "post_date": "05/12/2015 22:58:48", "content": "

I doing a similar process except I'm also checking to make sure the learning curve levels off and the roc plot does produce a similar value the the roc auc score calculated.  I see a drop too, but more like 5%.   

\n

I have not, but will start, holding out my own test set of about 300 values to estimate what it will be on the test set.   I'll respond will how closley these tests match the actual scores kaggle gave them once I get them done.

", "votes": null, "replies": [] }, { "id": 78154, "author_name": "karansarao", "author_url": "", "post_date": "05/13/2015 03:32:51", "content": "

any body facing a problem with Xgboost cv function? 

\n

I see that my own CV implemented using looping is more reflective of leaderboard performance, the auto function over estimates performance even when employing same number of folds as the other routine.

\n

On the other hand the auto function is much faster, don't know how it iterates so fast.

", "votes": null, "replies": [] }, { "id": 78189, "author_name": "tavish", "author_url": "", "post_date": "05/13/2015 11:50:06", "content": "

You can try leaving a 30% for in time validation and then run your model with cross validation. This way you get some reflection of your score before hand.

", "votes": null, "replies": [] }, { "id": 78195, "author_name": "cihanb", "author_url": "", "post_date": "05/13/2015 12:45:48", "content": "

I do a k-fold cross validation and average the AUC scores from each fold.

\n

The results are reasonably close to the private leaderboard. 

", "votes": null, "replies": [] }, { "id": 79515, "author_name": "links520", "author_url": "", "post_date": "05/20/2015 20:25:54", "content": "

i did k-fold cross validation as well. My leadboard is 0.08 lower than local cv score for a model, which is just totally weird.

", "votes": null, "replies": [] }, { "id": 79532, "author_name": "rinzewind", "author_url": "", "post_date": "05/20/2015 22:09:54", "content": "

I do a 10-fold cross-validation, also with results very close to the ones obtained in the public leaderboard (around 0.05 difference).

", "votes": null, "replies": [] }, { "id": 79549, "author_name": "", "author_url": "", "post_date": "05/21/2015 00:25:01", "content": "

I do a 10 times, 10-fold cross-validation, also with results very close to the ones obtained in the public leaderboard (around 0.025 difference).

", "votes": null, "replies": [] }, { "id": 79560, "author_name": "lijunjie", "author_url": "", "post_date": "05/21/2015 02:21:11", "content": "

If you are using R, I strongly recommend you try caret to train your model, and choose "ROC" as the train metric. The results of caret are more stable compared to my random 70/30 split. And the most important thing is that the result is quite close to my leader board score(around 0.005 difference)

", "votes": null, "replies": [] }, { "id": 79565, "author_name": "karansarao", "author_url": "", "post_date": "05/21/2015 03:29:55", "content": "

[quote=Thársis T. P. Souza;79549]

\n

I do a 10 times, 10-fold cross-validation, also with results very close to the ones obtained in the public leaderboard (around 0.025 difference).

\n

[/quote]

\n

10 times 10 fold , that would mean a 100 runs? Are you tuning or simply estimating expected performance?

\n

For tuning, it would effectively mean 100 iterations for each try!

\n\n

I am trying out 20 fold, my leaderboard drops between 2.5 to 3% which is disappointing. Maybe I should verify using your approach.

", "votes": null, "replies": [] }, { "id": 79614, "author_name": "", "author_url": "", "post_date": "05/21/2015 14:01:43", "content": "

yeah, 100 iterations and it only takes a few minutes.anyway, I believe feature engineering is the key in this competition not the model/tuning itself.

", "votes": null, "replies": [] }, { "id": 79665, "author_name": "", "author_url": "", "post_date": "05/21/2015 20:03:15", "content": "

Model evaluation has been an issue for me ... Privately , i have built models that have gained a score of more than 0.90... but on Leader-board, I am only around 0.88... and the problem is: discrepancy between private scores and leader-board scores varies with model. for some, it is 5% and it is 1.5% for others. I am using stratified k fold and i have also tried shuffle split. I wonder if my score might move up once the competition gets over as PB score is on a certain 30% of test data. :)

\n

Or a more serious explanation could be that i am over-fitting training data with certain models more than others. I do agree that feature engineering is key here and not the model/tuning itself.

", "votes": null, "replies": [] }, { "id": 79667, "author_name": "rinzewind", "author_url": "", "post_date": "05/21/2015 20:13:54", "content": "

[quote=KR;79665]

\n

Or a more serious explanation could be that i am over-fitting training data with certain models more than others. I do agree that feature engineering is key here and not the model/tuning itself.

\n

[/quote]

\n

In my experience, simpler models (logistic regression) tend to yield CV values closer to the actual leaderboard score after submission. I have had 0.92 AUC using boosting trees, but that was an illusion.

\n

I completely agree that feature extraction is much more critical than the actual classifier used. But isn't it almost always true? :-)

", "votes": null, "replies": [] }, { "id": 79683, "author_name": "", "author_url": "", "post_date": "05/21/2015 22:58:27", "content": "

[quote=José María Mateos;79667]

\n

[quote=KR;79665]

\n

Or a more serious explanation could be that i am over-fitting training data with certain models more than others. I do agree that feature engineering is key here and not the model/tuning itself.

\n

[/quote]

\n

In my experience, simpler models (logistic regression) tend to yield CV values closer to the actual leaderboard score after submission. I have had 0.92 AUC using boosting trees, but that was an illusion.

\n

I completely agree that feature extraction is much more critical than the actual classifier used. But isn't it almost always true? :-)

\n

[/quote]

\n\n

I do not know why i wrote that. :)

", "votes": null, "replies": [] }, { "id": 79768, "author_name": "bamine", "author_url": "", "post_date": "05/22/2015 16:20:20", "content": "

I just joined the competition and made a first submission.

\n

I used a 10 fold stratified cross validation and got like 0.90 auc, but when I submitted I got 0.64 !!!

\n

I am sure there is a data leak in one of my features, but I was curious if one of you guys had the same problem.

\n

Thank you !

", "votes": null, "replies": [] }, { "id": 79769, "author_name": "bamine", "author_url": "", "post_date": "05/22/2015 16:20:29", "content": "

I just joined the competition and made a first submission.

\n

I used a 10 fold stratified cross validation and got like 0.90 auc, but when I submitted I got 0.64 !!!

\n

I am sure there is a data leak in one of my features, but I was curious if one of you guys had the same problem.

\n

Thank you !

", "votes": null, "replies": [] }, { "id": 79792, "author_name": "karansarao", "author_url": "", "post_date": "05/22/2015 19:40:27", "content": "

[quote=José María Mateos;79667]

\n

[quote=KR;79665]

\n

Or a more serious explanation could be that i am over-fitting training data with certain models more than others. I do agree that feature engineering is key here and not the model/tuning itself.

\n

[/quote]

\n

In my experience, simpler models (logistic regression) tend to yield CV values closer to the actual leaderboard score after submission. I have had 0.92 AUC using boosting trees, but that was an illusion.

\n

I completely agree that feature extraction is much more critical than the actual classifier used. But isn't it almost always true? :-)

\n

[/quote]

\n\n

Model tuning and blending has increased my score from .890 to .919 on the same feature set, so I wouldnt say tuning isnt important, I might have been hill climbing, the private leaderboard will reveal that.

\n

I would wager that the top leaders would have almost similar features extracted by now - tuning/blending can give significant increases in the score.

", "votes": null, "replies": [] }, { "id": 79793, "author_name": "tavish", "author_url": "", "post_date": "05/22/2015 19:48:38", "content": "

I believe the entire leadership board will change a lot on private board from the public board. Model blending can really help you out here but not necessarily will give you best result on rest 70%. The key is - You should nominate your best model regardless of Public score for evaluation.

", "votes": null, "replies": [] }, { "id": 79807, "author_name": "mpekalski", "author_url": "", "post_date": "05/22/2015 23:15:26", "content": "

By blending you understand mixing in some way results from several models?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "78122": "", "78133": "", "78134": "", "78154": "", "78189": "", "78195": "", "79515": "", "79532": "", "79549": "", "79560": "", "79565": "", "79614": "", "79665": "", "79667": "", "79683": "", "79768": "", "79769": "", "79792": "", "79793": "", "79807": "" }, "source": "meta" }