{ "id": 1192, "title": "Gaps between validation, public, and private leaderboard scores", "url": "/competitions/AlgorithmicTradingChallenge/discussion/1192", "author_name": "", "post_date": "2011-12-23T23:20:30.233Z", "votes": null, "comment_count": 5, "views": 5032, "content": "

If you read \r\nthe 'Milestone Entries and Reviews' thread, you'll see that for the Nov 30 prize we have these public leaderboard scores:

\r\n

  Xiaoshi Lu, 0.76133, 1st on public leaderboard
\r\n  Alec Stephenson, 0.77847, 5th on public leaderboard

\r\n

and for the Dec 22 prize we have:

\r\n

  Xiaoshi Lu, 0.75567, 1st on public leaderboard
\r\n  alegro, 0.78206, 20th on public leaderboard

\r\n

yet evidently both Alec Stephenson and alegro had better private leaderboard scores, despite large lag on public scores. So it looks like in this competition, the public score has little bearing on the private score, it's not even a rough indicator.

\r\n

I guess rumors of large, seemingly random differences betweenn local validation and public leaderboard scores in this competition are true. Anyone care to comment ? 

", "messages": [ { "id": "7481", "postDate": "12/23/2011 23:20:30", "content": "

If you read \r\nthe 'Milestone Entries and Reviews' thread, you'll see that for the Nov 30 prize we have these public leaderboard scores:

\r\n

  Xiaoshi Lu, 0.76133, 1st on public leaderboard
\r\n  Alec Stephenson, 0.77847, 5th on public leaderboard

\r\n

and for the Dec 22 prize we have:

\r\n

  Xiaoshi Lu, 0.75567, 1st on public leaderboard
\r\n  alegro, 0.78206, 20th on public leaderboard

\r\n

yet evidently both Alec Stephenson and alegro had better private leaderboard scores, despite large lag on public scores. So it looks like in this competition, the public score has little bearing on the private score, it's not even a rough indicator.

\r\n

I guess rumors of large, seemingly random differences betweenn local validation and public leaderboard scores in this competition are true. Anyone care to comment ? 

", "rawMarkdown": "", "votes": null }, { "id": "7483", "postDate": "12/23/2011 23:33:47", "content": "

Such variance is quite normal for this specific program, I will give a detailed analysis after the competition, but I can safely say the final prize is just a lottery, that's also why I try hard to get the milestone prize, which in my mind was the only things\r\n can get through enough efforts.

", "rawMarkdown": "", "votes": null }, { "id": "7486", "postDate": "12/24/2011 00:09:23", "content": "

[quote=B Yang;7481]

\r\n

I guess rumors of large, seemingly random differences betweenn local validation and public leaderboard scores in this competition are true. Anyone care to comment ? 

\r\n

[/quote]

\r\n

The leaderboard and private scores will be different as far as I can tell because..

\r\n

1) The rmse is not normalised per commodity. Thus efforts going into getting the predictions of 99% of your commodities good could be blown out of the water by a single commodity with large values that are unpredictable - and a lucky guess could end up winning

\r\n

 2) The leaderboard is not really a random sample. The particular events are randomly sampled, but then all 50 predictions for that event used.

", "rawMarkdown": "", "votes": null }, { "id": "7494", "postDate": "12/24/2011 06:33:28", "content": "

Bo, yes, in comparison to other Kaggle contests, I would say that in this one the leaderboard seems somewhat less meaningful (and that's a shame). Many have already commented on how the holdout & test & training RMSEs don't line up, and the fact that the\r\n milestone-prize winners weren't near the top of the leaderboard is in my opinion yet another symptom of the weirdness of this data set.

\r\n

...

\r\n

An anectdote: A few nights ago, I was using the last 50k lines of training as a holdout set, and for two different submissions I got a.most exactly the same holdout RMSE score. But when I submitted them, I got testing / leaderboard scores that were far apart\r\n -- by about 0.0400. That inconsistency left me scratching my head. 

", "rawMarkdown": "", "votes": null }, { "id": "7499", "postDate": "12/25/2011 06:41:13", "content": "

Again, I believe there is something wrong with the score system.

\r\n

I may try to do some check once the final result are out if Kaggle can release the test set true data. 

", "rawMarkdown": "", "votes": null }, { "id": "7506", "postDate": "12/26/2011 04:59:40", "content": "

I did some experiments - apply a model developed from ~200k examples to 15k random test samples (selected from 35k examples) - and noted a large variance in results due to a few examples with very large spreads. Given this, I am not surprised by the current\r\n results and do not think the public leaderboard is that relevant. But I do not think there is evidence that the scoring is off. In fact, when selecting a subset of the training data similar to the test data, I get very similar scores.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 7483, "author_name": "xiaoshilu", "author_url": "", "post_date": "12/23/2011 23:33:47", "content": "

Such variance is quite normal for this specific program, I will give a detailed analysis after the competition, but I can safely say the final prize is just a lottery, that's also why I try hard to get the milestone prize, which in my mind was the only things\r\n can get through enough efforts.

", "votes": null, "replies": [] }, { "id": 7486, "author_name": "salimali", "author_url": "", "post_date": "12/24/2011 00:09:23", "content": "

[quote=B Yang;7481]

\r\n

I guess rumors of large, seemingly random differences betweenn local validation and public leaderboard scores in this competition are true. Anyone care to comment ? 

\r\n

[/quote]

\r\n

The leaderboard and private scores will be different as far as I can tell because..

\r\n

1) The rmse is not normalised per commodity. Thus efforts going into getting the predictions of 99% of your commodities good could be blown out of the water by a single commodity with large values that are unpredictable - and a lucky guess could end up winning

\r\n

 2) The leaderboard is not really a random sample. The particular events are randomly sampled, but then all 50 predictions for that event used.

", "votes": null, "replies": [] }, { "id": 7494, "author_name": "chefele", "author_url": "", "post_date": "12/24/2011 06:33:28", "content": "

Bo, yes, in comparison to other Kaggle contests, I would say that in this one the leaderboard seems somewhat less meaningful (and that's a shame). Many have already commented on how the holdout & test & training RMSEs don't line up, and the fact that the\r\n milestone-prize winners weren't near the top of the leaderboard is in my opinion yet another symptom of the weirdness of this data set.

\r\n

...

\r\n

An anectdote: A few nights ago, I was using the last 50k lines of training as a holdout set, and for two different submissions I got a.most exactly the same holdout RMSE score. But when I submitted them, I got testing / leaderboard scores that were far apart\r\n -- by about 0.0400. That inconsistency left me scratching my head. 

", "votes": null, "replies": [] }, { "id": 7499, "author_name": "woshialex", "author_url": "", "post_date": "12/25/2011 06:41:13", "content": "

Again, I believe there is something wrong with the score system.

\r\n

I may try to do some check once the final result are out if Kaggle can release the test set true data. 

", "votes": null, "replies": [] }, { "id": 7506, "author_name": "dejavu", "author_url": "", "post_date": "12/26/2011 04:59:40", "content": "

I did some experiments - apply a model developed from ~200k examples to 15k random test samples (selected from 35k examples) - and noted a large variance in results due to a few examples with very large spreads. Given this, I am not surprised by the current\r\n results and do not think the public leaderboard is that relevant. But I do not think there is evidence that the scoring is off. In fact, when selecting a subset of the training data similar to the test data, I get very similar scores.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "7481": "", "7483": "", "7486": "", "7494": "", "7499": "", "7506": "" }, "source": "meta" }