{ "id": 12387, "title": "Advice for selecting submissions", "url": "/competitions/avazu-ctr-prediction/discussion/12387", "author_name": "", "post_date": "2015-02-03T14:14:25.120Z", "votes": null, "comment_count": 2, "views": 1064, "content": "

Before the end of the competition we can select up to 2 submissions that will be used to calculate our final leaderboard score. Otherwise, the submissions with best score in the public leaderboard are used by default.

\n

I wonder if I should use the submissions with best public score:

\n

CV score ~ 0.393  ->  LB score ~0.395

\n

or those with best CV score:

\n

CV score ~ 0.38  ->  LB score ~0.41

\n

I calculated the CV score testing in 1 of every 10 rows which had been previously removed from the training.

\n

Any advice will be much appreciated. Thanks!

", "messages": [ { "id": "63484", "postDate": "02/03/2015 14:14:25", "content": "

Before the end of the competition we can select up to 2 submissions that will be used to calculate our final leaderboard score. Otherwise, the submissions with best score in the public leaderboard are used by default.

\n

I wonder if I should use the submissions with best public score:

\n

CV score ~ 0.393  ->  LB score ~0.395

\n

or those with best CV score:

\n

CV score ~ 0.38  ->  LB score ~0.41

\n

I calculated the CV score testing in 1 of every 10 rows which had been previously removed from the training.

\n

Any advice will be much appreciated. Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "63485", "postDate": "02/03/2015 14:48:03", "content": "

Hello,

\n

With your CV you are interpolating and not extrapolating as far as I can see.
It's better to use the last X records (1-2 days) as CV.

\n

So I'd trust the LB.

\n

Also.. there are 4M records for evaluation so public and private LB should not differ too much.

", "rawMarkdown": "", "votes": null }, { "id": "63561", "postDate": "02/04/2015 12:52:30", "content": "

To add to Julian's very good suggestion of using the last X records for CV, you can additionally perform a weighted average of your local CV scores, and the public leaderboard score. 

Suppose you have c datapoints in your CV set, with a CV score of p and d datapoints in the public leaderboard set (.2*size_of_test_set) with a score of q. Then, the weighted average CV score would simply be: (c*p + d*q)/(c + d). 

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 63485, "author_name": "juliandewit", "author_url": "", "post_date": "02/03/2015 14:48:03", "content": "

Hello,

\n

With your CV you are interpolating and not extrapolating as far as I can see.
It's better to use the last X records (1-2 days) as CV.

\n

So I'd trust the LB.

\n

Also.. there are 4M records for evaluation so public and private LB should not differ too much.

", "votes": null, "replies": [] }, { "id": 63561, "author_name": "miroslaw", "author_url": "", "post_date": "02/04/2015 12:52:30", "content": "

To add to Julian's very good suggestion of using the last X records for CV, you can additionally perform a weighted average of your local CV scores, and the public leaderboard score. 

Suppose you have c datapoints in your CV set, with a CV score of p and d datapoints in the public leaderboard set (.2*size_of_test_set) with a score of q. Then, the weighted average CV score would simply be: (c*p + d*q)/(c + d). 

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "63484": "", "63485": "", "63561": "" }, "source": "meta" }