{ "id": 14639, "title": "Poor result due to overfitting; how to prevent next time?", "url": "/competitions/facebook-recruiting-iv-human-or-bot/discussion/14639", "author_name": "", "post_date": "2015-06-09T16:14:23.320Z", "votes": null, "comment_count": 9, "views": 2129, "content": "

I created a random-forest-model which performed really bad. I created some usual features like uniqueCountries, uniqueDevices, merchant etc.

\n

I also calculated for i.e. url how much of those url's where robots. After doing so I calculated the max per bidder for such an url. The idea was that if a bidder came from an url and that url was 100% robot then the bidder is also a robot. At the training set I got brilliant results :-), at the testset poor results.

\n

What do I have to do to prevent this in the future? I'd like to learn from this. I tried to lower the mtry-parameter, it didn't help.

", "messages": [ { "id": "81368", "postDate": "06/09/2015 16:14:23", "content": "

I created a random-forest-model which performed really bad. I created some usual features like uniqueCountries, uniqueDevices, merchant etc.

\n

I also calculated for i.e. url how much of those url's where robots. After doing so I calculated the max per bidder for such an url. The idea was that if a bidder came from an url and that url was 100% robot then the bidder is also a robot. At the training set I got brilliant results :-), at the testset poor results.

\n

What do I have to do to prevent this in the future? I'd like to learn from this. I tried to lower the mtry-parameter, it didn't help.

", "rawMarkdown": "", "votes": null }, { "id": "81370", "postDate": "06/09/2015 16:27:51", "content": "

I didn't look at your code (not that fluent in R), but did you make sure you didn't make use of the labels of the holdout set when you created the features you mention in the second paragraph? When creating features that depend (indirectly) on labels one has to be really careful in CV as to prevent leakage

", "rawMarkdown": "", "votes": null }, { "id": "81372", "postDate": "06/09/2015 17:03:53", "content": "

Yes I did.

\n

Per bidder I calculated the url which was used with the highest probability of being robot.

", "rawMarkdown": "", "votes": null }, { "id": "81375", "postDate": "06/09/2015 17:15:24", "content": "

Why you are worried about the overfitting if you have bad approach?

\n

Read this threads:

\n

https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/forums/t/14616/new-model-scores-0-9232-but-i-have-no-quota-left-today

\n

https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/forums/t/14629/share-your-codes

\n

https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/forums/t/14628/share-your-secret-sauce

", "rawMarkdown": "", "votes": null }, { "id": "81388", "postDate": "06/09/2015 18:15:55", "content": "

To decrease the chances at overfit look at ensembling/model averaging methods. These are supposed to reduce the generalization error. So next time create an ExtraTrees model and a GBM model too and try a simple (rank) average. Don't use too many features and when your CV is too good to be true, it likely is.

\n

Though this was a very difficult competition. I think the majority underfitted or overfitted. Those 150 jumps in ranks are not healthy: You can overfit and get a lucky split.

\n

I beat too many Master Kagglers and saw too many ambiguous labeling for me to take this challenge and its results completely seriously.

\n

Also, try to find out the point of diminishing returns. If you have a 100 subs, then:

\n

A) likely you have a sub that would do better when selected, falsely giving you impression that you overfitted.

\n

B) you kept on improving public leaderboard, while the real gains stopped happening 50 subs ago.

", "rawMarkdown": "", "votes": null }, { "id": "81420", "postDate": "06/09/2015 23:04:35", "content": "

I could have come 2nd and potentially even have won this :( I had 2 submissions that outperformed my selected submission. My best was 0.94186 on the private LB but I discarded that approach and stopped iterating on improving that because it was performing worse on the Public leaderboard by almost 0.1. In the end I thought it was too good to be true (was getting a CV AUC of 0.955-0.96) and decided not to follow that approach. I guess the lesson I learned is to put more faith in your cross validations.

\n

Irrespective, I am happy nonetheless since this is one of my first competitions. 

", "rawMarkdown": "", "votes": null }, { "id": "81421", "postDate": "06/09/2015 23:04:54", "content": "

Oops double posted..

", "rawMarkdown": "", "votes": null }, { "id": "81446", "postDate": "06/10/2015 05:15:59", "content": "

My best unchosen submission was 0.935 AUC apparently. Live and learn. I got overconfident as most of the time I have underfit and chose submissions that were too aggressive. I do like learning from these more closed competitions were you get to do more feature engineering and selection.

", "rawMarkdown": "", "votes": null }, { "id": "81453", "postDate": "06/10/2015 06:55:18", "content": "

[quote=Wijnand;81368]

\n

I created a random-forest-model which performed really bad. I created some usual features like uniqueCountries, uniqueDevices, merchant etc.

\n

I also calculated for i.e. url how much of those url's where robots. After doing so I calculated the max per bidder for such an url. The idea was that if a bidder came from an url and that url was 100% robot then the bidder is also a robot. At the training set I got brilliant results :-), at the testset poor results.

\n

What do I have to do to prevent this in the future? I'd like to learn from this. I tried to lower the mtry-parameter, it didn't help.

\n

[/quote]

\n\n

The number of referring URLs far outnumbered the number of bidders, if I remember correctly there were around 1,786,351 URLs. If you calculated the probability of a referring URL being used by a bot, then you are implicitly relying on seeing the URL as a reference in the future. This approach has some obvious pitfalls, although you might safely assume that you would encounter the same country, the same first two IP octets, even the same device in the future (it's a minor gamble with a device pool of 7,351, not very appropriate but it was useful in this particular competition). Presuming to encounter the same URL in the future,  with a set this diverse, is a major gamble. Plus if you leave out the samples in the validation set while calculating these probabilities, this decrease in model performance would be obvious even in a 5-fold stratified CV. (I would recommend a 33% validation partitioning, that worked great for me).

", "rawMarkdown": "", "votes": null }, { "id": "81456", "postDate": "06/10/2015 07:32:02", "content": "

So this competition was all about feature engineering and choosing a right model. I knew that there is a high chance to overfit for this competition. Yet I did overfit. My best score was 0.9376 which I achieved by my 9th submission and did not choose for private LB scoring.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 81370, "author_name": "cdjbee", "author_url": "", "post_date": "06/09/2015 16:27:51", "content": "

I didn't look at your code (not that fluent in R), but did you make sure you didn't make use of the labels of the holdout set when you created the features you mention in the second paragraph? When creating features that depend (indirectly) on labels one has to be really careful in CV as to prevent leakage

", "votes": null, "replies": [] }, { "id": 81372, "author_name": "wijnand", "author_url": "", "post_date": "06/09/2015 17:03:53", "content": "

Yes I did.

\n

Per bidder I calculated the url which was used with the highest probability of being robot.

", "votes": null, "replies": [] }, { "id": 81375, "author_name": "rushter", "author_url": "", "post_date": "06/09/2015 17:15:24", "content": "

Why you are worried about the overfitting if you have bad approach?

\n

Read this threads:

\n

https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/forums/t/14616/new-model-scores-0-9232-but-i-have-no-quota-left-today

\n

https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/forums/t/14629/share-your-codes

\n

https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/forums/t/14628/share-your-secret-sauce

", "votes": null, "replies": [] }, { "id": 81388, "author_name": "triskelion", "author_url": "", "post_date": "06/09/2015 18:15:55", "content": "

To decrease the chances at overfit look at ensembling/model averaging methods. These are supposed to reduce the generalization error. So next time create an ExtraTrees model and a GBM model too and try a simple (rank) average. Don't use too many features and when your CV is too good to be true, it likely is.

\n

Though this was a very difficult competition. I think the majority underfitted or overfitted. Those 150 jumps in ranks are not healthy: You can overfit and get a lucky split.

\n

I beat too many Master Kagglers and saw too many ambiguous labeling for me to take this challenge and its results completely seriously.

\n

Also, try to find out the point of diminishing returns. If you have a 100 subs, then:

\n

A) likely you have a sub that would do better when selected, falsely giving you impression that you overfitted.

\n

B) you kept on improving public leaderboard, while the real gains stopped happening 50 subs ago.

", "votes": null, "replies": [] }, { "id": 81420, "author_name": "ask788", "author_url": "", "post_date": "06/09/2015 23:04:35", "content": "

I could have come 2nd and potentially even have won this :( I had 2 submissions that outperformed my selected submission. My best was 0.94186 on the private LB but I discarded that approach and stopped iterating on improving that because it was performing worse on the Public leaderboard by almost 0.1. In the end I thought it was too good to be true (was getting a CV AUC of 0.955-0.96) and decided not to follow that approach. I guess the lesson I learned is to put more faith in your cross validations.

\n

Irrespective, I am happy nonetheless since this is one of my first competitions. 

", "votes": null, "replies": [] }, { "id": 81421, "author_name": "ask788", "author_url": "", "post_date": "06/09/2015 23:04:54", "content": "

Oops double posted..

", "votes": null, "replies": [] }, { "id": 81446, "author_name": "jbsilva", "author_url": "", "post_date": "06/10/2015 05:15:59", "content": "

My best unchosen submission was 0.935 AUC apparently. Live and learn. I got overconfident as most of the time I have underfit and chose submissions that were too aggressive. I do like learning from these more closed competitions were you get to do more feature engineering and selection.

", "votes": null, "replies": [] }, { "id": 81453, "author_name": "sushanttripathy", "author_url": "", "post_date": "06/10/2015 06:55:18", "content": "

[quote=Wijnand;81368]

\n

I created a random-forest-model which performed really bad. I created some usual features like uniqueCountries, uniqueDevices, merchant etc.

\n

I also calculated for i.e. url how much of those url's where robots. After doing so I calculated the max per bidder for such an url. The idea was that if a bidder came from an url and that url was 100% robot then the bidder is also a robot. At the training set I got brilliant results :-), at the testset poor results.

\n

What do I have to do to prevent this in the future? I'd like to learn from this. I tried to lower the mtry-parameter, it didn't help.

\n

[/quote]

\n\n

The number of referring URLs far outnumbered the number of bidders, if I remember correctly there were around 1,786,351 URLs. If you calculated the probability of a referring URL being used by a bot, then you are implicitly relying on seeing the URL as a reference in the future. This approach has some obvious pitfalls, although you might safely assume that you would encounter the same country, the same first two IP octets, even the same device in the future (it's a minor gamble with a device pool of 7,351, not very appropriate but it was useful in this particular competition). Presuming to encounter the same URL in the future,  with a set this diverse, is a major gamble. Plus if you leave out the samples in the validation set while calculating these probabilities, this decrease in model performance would be obvious even in a 5-fold stratified CV. (I would recommend a 33% validation partitioning, that worked great for me).

", "votes": null, "replies": [] }, { "id": 81456, "author_name": "tsendeemts", "author_url": "", "post_date": "06/10/2015 07:32:02", "content": "

So this competition was all about feature engineering and choosing a right model. I knew that there is a high chance to overfit for this competition. Yet I did overfit. My best score was 0.9376 which I achieved by my 9th submission and did not choose for private LB scoring.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "81368": "", "81370": "", "81372": "", "81375": "", "81388": "", "81420": "", "81421": "", "81446": "", "81453": "", "81456": "" }, "source": "meta" }