{ "id": 2721, "title": "how did you do validation?", "url": "/competitions/MerckActivity/discussion/2721", "author_name": "", "post_date": "2012-09-20T10:30:13.903Z", "votes": null, "comment_count": 14, "views": 6375, "content": "

My best CV score model always get poor result in public leaderboard.

\r\n

From the Dataset page:it is expected that the distribution of descriptors will not necessarily be the same between the training and test sets.

\r\n

maybe this is the reason why my CV score is hard to  match up with public leaderboard score.

\r\n

I don't think do validation can let things better.How did you do?

", "messages": [ { "id": "14627", "postDate": "09/20/2012 10:30:13", "content": "

My best CV score model always get poor result in public leaderboard.

\r\n

From the Dataset page:it is expected that the distribution of descriptors will not necessarily be the same between the training and test sets.

\r\n

maybe this is the reason why my CV score is hard to  match up with public leaderboard score.

\r\n

I don't think do validation can let things better.How did you do?

", "rawMarkdown": "", "votes": null }, { "id": "14628", "postDate": "09/20/2012 12:58:53", "content": "

same here.

\r\n

unless the timestamp of the training rows are added, everybody is flying blind.

\r\n

problem is with the distribution of descriptors -

", "rawMarkdown": "", "votes": null }, { "id": "14653", "postDate": "09/21/2012 04:00:59", "content": "

Hi all!
\r\nI have the following error on a different set of data:

\r\n

Date Err CV
\r\nAct1 0.5858759
\r\nAct2 0.4944779
\r\nAct3 0.2612551
\r\nAct4 0.4009051
\r\nAct5 0.6876679
\r\nAct6 0.7243370
\r\nAct7 0.6265660
\r\nAct8 0.4257228
\r\nAct9 0.5311976
\r\nAct10 0.3202557
\r\nAct11 0.4699731
\r\nAct12 0.5130320
\r\nAct13 0.1071174
\r\nAct14 0.3755090
\r\nAct_15 0.7570607

", "rawMarkdown": "", "votes": null }, { "id": "14654", "postDate": "09/21/2012 04:15:13", "content": "

interesting Alexander! - I got similar scores but  when I submitted, the leaderboard score was lower!

\r\n

[quote=Alexander Larko;14653]

\r\n

Hi all!
\r\nI have the following error on a different set of data:

\r\n

Date Err CV
\r\nAct1 0.5858759
\r\nAct2 0.4944779
\r\nAct3 0.2612551
\r\nAct4 0.4009051
\r\nAct5 0.6876679
\r\nAct6 0.7243370
\r\nAct7 0.6265660
\r\nAct8 0.4257228
\r\nAct9 0.5311976
\r\nAct10 0.3202557
\r\nAct11 0.4699731
\r\nAct12 0.5130320
\r\nAct13 0.1071174
\r\nAct14 0.3755090
\r\nAct_15 0.7570607

\r\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "14723", "postDate": "09/23/2012 13:13:48", "content": "

It seems that quite a few folks have a similar problem with the public leaderboard score significantly lower than CV scores as this concern was brought up in another thread as well.

", "rawMarkdown": "", "votes": null }, { "id": "14775", "postDate": "09/25/2012 08:52:37", "content": "

My interpretation is that it is expected that the score from within the set (samples from the same time as the training data) would differ from the official scoring that is made on test sets taken from samples created at a later time than the training sets.

\r\n

 

\r\n

Question is how much lower?

\r\n

I have not made a public submission so I can't give any facts here.

\r\n

My internal k-fold crossvalidation is hovering around R2 = 0.65 but I expect this to land a score around 0.40 on the leaderboard. Somewhat better than a naive random forest but not near the leading scores.

", "rawMarkdown": "", "votes": null }, { "id": "14783", "postDate": "09/25/2012 13:17:27", "content": "

Jonathan,

\r\n

Indeed, I too expect the leaderboard scoring to be different from cross validation, and that is what I observe for myself as well. The scores you have quoted seems well within the reasonable range (0.65 -> 0.40).

", "rawMarkdown": "", "votes": null }, { "id": "14786", "postDate": "09/25/2012 14:13:08", "content": "

Jonathan, shinmagi

\r\n

That sounds reasonable, my CV shows 0,722 -> 0,463. The real difference seems to be somewhere in the range of 0,25 - 0,26.

", "rawMarkdown": "", "votes": null }, { "id": "14828", "postDate": "09/26/2012 09:51:07", "content": "

My CV is 0.65 and leaderboard is 0.45, which I think is bit less in difference what you people are saying. Distribution of train and test may be different but we all can try to make the distribution of both same based on the features, then I think CV will\r\n start making more sense.

", "rawMarkdown": "", "votes": null }, { "id": "14841", "postDate": "09/26/2012 17:22:22", "content": "

And here is results for my first submission CV R2 of 0.624 -> 0.39 in the official scoring.

", "rawMarkdown": "", "votes": null }, { "id": "15124", "postDate": "10/03/2012 12:33:46", "content": "

Hi Everyone

\r\n

I have one doubt while doing validation. Currently I m dividing training data randomly into 80:20 and fitting model on 80% and predicting for 20%.

\r\n

Other aspect of the data is that it is time split data, so should I divide training data also as time split to check my model performance. Till now I haven't taken this approach but if somebody is using this approach then please shed some light on it.

\r\n

Also my average validation score is currently coming near to 0.78 using my current approach, I haven't submitted this model yet. I want to confirm which approach is better for doing validation.

\r\n

Thanks in advance

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "15129", "postDate": "10/03/2012 14:47:56", "content": "

I use 5-fold validation for sets 4,5,7,14; 3-fold for 2,3,8,9,10,11,12,13,15 and 2-fold for 1,6.
\r\nSeems to me to be a good tradeoff between reliability and speed.

", "rawMarkdown": "", "votes": null }, { "id": "15130", "postDate": "10/03/2012 15:36:08", "content": "

@pawzaw

\r\n

So your CV scores matches the leaderboard score.

\r\n

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "15179", "postDate": "10/04/2012 19:27:47", "content": "

[quote=Alexander Larko;14653]

\r\n

Hi all!
\r\nI have the following error on a different set of data:

\r\n

Date Err CV
\r\nAct1 0.5858759
\r\nAct2 0.4944779
\r\nAct3 0.2612551
\r\nAct4 0.4009051
\r\nAct5 0.6876679
\r\nAct6 0.7243370
\r\nAct7 0.6265660
\r\nAct8 0.4257228
\r\nAct9 0.5311976
\r\nAct10 0.3202557
\r\nAct11 0.4699731
\r\nAct12 0.5130320
\r\nAct13 0.1071174
\r\nAct14 0.3755090
\r\nAct_15 0.7570607

\r\n

[/quote]

\r\n

Hi, Alexander

\r\n

Don't understand "error on a different set of data", is this CV error of R-sq? or is an estimate of test R-sq?

\r\n

Do you say with 0.4859 of mean CV you get in 0.45-46 range leaderboard?

\r\n

My gap is in the 0.20-0.25 range as seems usual.

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "15189", "postDate": "10/05/2012 03:52:44", "content": "

 

\r\n

Blind\r\n Ape wrote:

\r\n

____________________________________________________________________________\r\n

\r\n

Hi, Alexander

\r\n

Don't understand "error on a different set of data", is this CV error of R-sq? or is an estimate of test R-sq?

\r\n

Do you say with 0.4859 of mean CV you get in 0.45-46 range leaderboard?

\r\n

My gap is in the 0.20-0.25 range as seems usual.

\r\n

__________________________________________________________________________

\r\n

Hi, Blind Ape!

\r\n

 

\r\n

My mean CV\r\n for present-day day is 0.513473.

\r\n

 

\r\n

All\r\n the best,

\r\n

Alexander.

\r\n


\r\n
\r\n

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 14628, "author_name": "rkirana", "author_url": "", "post_date": "09/20/2012 12:58:53", "content": "

same here.

\r\n

unless the timestamp of the training rows are added, everybody is flying blind.

\r\n

problem is with the distribution of descriptors -

", "votes": null, "replies": [] }, { "id": 14653, "author_name": "alexxanderlarko", "author_url": "", "post_date": "09/21/2012 04:00:59", "content": "

Hi all!
\r\nI have the following error on a different set of data:

\r\n

Date Err CV
\r\nAct1 0.5858759
\r\nAct2 0.4944779
\r\nAct3 0.2612551
\r\nAct4 0.4009051
\r\nAct5 0.6876679
\r\nAct6 0.7243370
\r\nAct7 0.6265660
\r\nAct8 0.4257228
\r\nAct9 0.5311976
\r\nAct10 0.3202557
\r\nAct11 0.4699731
\r\nAct12 0.5130320
\r\nAct13 0.1071174
\r\nAct14 0.3755090
\r\nAct_15 0.7570607

", "votes": null, "replies": [] }, { "id": 14654, "author_name": "rkirana", "author_url": "", "post_date": "09/21/2012 04:15:13", "content": "

interesting Alexander! - I got similar scores but  when I submitted, the leaderboard score was lower!

\r\n

[quote=Alexander Larko;14653]

\r\n

Hi all!
\r\nI have the following error on a different set of data:

\r\n

Date Err CV
\r\nAct1 0.5858759
\r\nAct2 0.4944779
\r\nAct3 0.2612551
\r\nAct4 0.4009051
\r\nAct5 0.6876679
\r\nAct6 0.7243370
\r\nAct7 0.6265660
\r\nAct8 0.4257228
\r\nAct9 0.5311976
\r\nAct10 0.3202557
\r\nAct11 0.4699731
\r\nAct12 0.5130320
\r\nAct13 0.1071174
\r\nAct14 0.3755090
\r\nAct_15 0.7570607

\r\n

[/quote]

", "votes": null, "replies": [] }, { "id": 14723, "author_name": "shinmagi", "author_url": "", "post_date": "09/23/2012 13:13:48", "content": "

It seems that quite a few folks have a similar problem with the public leaderboard score significantly lower than CV scores as this concern was brought up in another thread as well.

", "votes": null, "replies": [] }, { "id": 14775, "author_name": "jontis", "author_url": "", "post_date": "09/25/2012 08:52:37", "content": "

My interpretation is that it is expected that the score from within the set (samples from the same time as the training data) would differ from the official scoring that is made on test sets taken from samples created at a later time than the training sets.

\r\n

 

\r\n

Question is how much lower?

\r\n

I have not made a public submission so I can't give any facts here.

\r\n

My internal k-fold crossvalidation is hovering around R2 = 0.65 but I expect this to land a score around 0.40 on the leaderboard. Somewhat better than a naive random forest but not near the leading scores.

", "votes": null, "replies": [] }, { "id": 14783, "author_name": "shinmagi", "author_url": "", "post_date": "09/25/2012 13:17:27", "content": "

Jonathan,

\r\n

Indeed, I too expect the leaderboard scoring to be different from cross validation, and that is what I observe for myself as well. The scores you have quoted seems well within the reasonable range (0.65 -> 0.40).

", "votes": null, "replies": [] }, { "id": 14786, "author_name": "dmitrim", "author_url": "", "post_date": "09/25/2012 14:13:08", "content": "

Jonathan, shinmagi

\r\n

That sounds reasonable, my CV shows 0,722 -> 0,463. The real difference seems to be somewhere in the range of 0,25 - 0,26.

", "votes": null, "replies": [] }, { "id": 14828, "author_name": "thakurrajanand", "author_url": "", "post_date": "09/26/2012 09:51:07", "content": "

My CV is 0.65 and leaderboard is 0.45, which I think is bit less in difference what you people are saying. Distribution of train and test may be different but we all can try to make the distribution of both same based on the features, then I think CV will\r\n start making more sense.

", "votes": null, "replies": [] }, { "id": 14841, "author_name": "jontis", "author_url": "", "post_date": "09/26/2012 17:22:22", "content": "

And here is results for my first submission CV R2 of 0.624 -> 0.39 in the official scoring.

", "votes": null, "replies": [] }, { "id": 15124, "author_name": "thakurrajanand", "author_url": "", "post_date": "10/03/2012 12:33:46", "content": "

Hi Everyone

\r\n

I have one doubt while doing validation. Currently I m dividing training data randomly into 80:20 and fitting model on 80% and predicting for 20%.

\r\n

Other aspect of the data is that it is time split data, so should I divide training data also as time split to check my model performance. Till now I haven't taken this approach but if somebody is using this approach then please shed some light on it.

\r\n

Also my average validation score is currently coming near to 0.78 using my current approach, I haven't submitted this model yet. I want to confirm which approach is better for doing validation.

\r\n

Thanks in advance

\r\n

 

", "votes": null, "replies": [] }, { "id": 15129, "author_name": "pzawal", "author_url": "", "post_date": "10/03/2012 14:47:56", "content": "

I use 5-fold validation for sets 4,5,7,14; 3-fold for 2,3,8,9,10,11,12,13,15 and 2-fold for 1,6.
\r\nSeems to me to be a good tradeoff between reliability and speed.

", "votes": null, "replies": [] }, { "id": 15130, "author_name": "thakurrajanand", "author_url": "", "post_date": "10/03/2012 15:36:08", "content": "

@pawzaw

\r\n

So your CV scores matches the leaderboard score.

\r\n

Thanks

", "votes": null, "replies": [] }, { "id": 15179, "author_name": "blindape", "author_url": "", "post_date": "10/04/2012 19:27:47", "content": "

[quote=Alexander Larko;14653]

\r\n

Hi all!
\r\nI have the following error on a different set of data:

\r\n

Date Err CV
\r\nAct1 0.5858759
\r\nAct2 0.4944779
\r\nAct3 0.2612551
\r\nAct4 0.4009051
\r\nAct5 0.6876679
\r\nAct6 0.7243370
\r\nAct7 0.6265660
\r\nAct8 0.4257228
\r\nAct9 0.5311976
\r\nAct10 0.3202557
\r\nAct11 0.4699731
\r\nAct12 0.5130320
\r\nAct13 0.1071174
\r\nAct14 0.3755090
\r\nAct_15 0.7570607

\r\n

[/quote]

\r\n

Hi, Alexander

\r\n

Don't understand "error on a different set of data", is this CV error of R-sq? or is an estimate of test R-sq?

\r\n

Do you say with 0.4859 of mean CV you get in 0.45-46 range leaderboard?

\r\n

My gap is in the 0.20-0.25 range as seems usual.

\r\n

 

", "votes": null, "replies": [] }, { "id": 15189, "author_name": "alexxanderlarko", "author_url": "", "post_date": "10/05/2012 03:52:44", "content": "

 

\r\n

Blind\r\n Ape wrote:

\r\n

____________________________________________________________________________\r\n

\r\n

Hi, Alexander

\r\n

Don't understand "error on a different set of data", is this CV error of R-sq? or is an estimate of test R-sq?

\r\n

Do you say with 0.4859 of mean CV you get in 0.45-46 range leaderboard?

\r\n

My gap is in the 0.20-0.25 range as seems usual.

\r\n

__________________________________________________________________________

\r\n

Hi, Blind Ape!

\r\n

 

\r\n

My mean CV\r\n for present-day day is 0.513473.

\r\n

 

\r\n

All\r\n the best,

\r\n

Alexander.

\r\n


\r\n
\r\n

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "14627": "", "14628": "", "14653": "", "14654": "", "14723": "", "14775": "", "14783": "", "14786": "", "14828": "", "14841": "", "15124": "", "15129": "", "15130": "", "15179": "", "15189": "" }, "source": "meta" }