I agree the concept of doing the public/private (25%/75%) split by sample date would make for a very misleading public leaderboard. I rather doubt they did that. However, since they haven't really responded at all thus far I suppose we'll never know. My\r\n original question was whether they did stratified sampling by activity. By that I meant is the public leaderboard based on 25% of activity #1, 25% of activity #2, etc. If they didn't stratify it, the sample % of the smaller activities could easily range from\r\n 10%-40% I imagine (I'm sure someone will calculate the actual range shortly.) I mainly asked this because I could see an error like this creep in if they re-used code from other contests to make the public/private split.
", "votes": 1 }, { "id": 15463, "postDate": "2012-10-12T03:10:21.003Z", "content": "Every single team who is on the public leaderboard has tuned their models to the public leaderboard unless they didn't look at their position on the leaderboard, but only at the warnings/errors of their submission.
", "votes": 1 }, { "id": 15818, "postDate": "2012-10-17T15:05:08.877Z", "content": "So private leaderboard was very similar to public leaderboard. Not much volatility
" }, { "id": 15777, "postDate": "2012-10-17T04:06:26.573Z", "content": "me wondering as well - it is usually readily available.
" }, { "id": 15757, "postDate": "2012-10-17T00:17:58.157Z", "content": "Hi, what's the standard delay for displaying full leaderboard? Minutes or days? I'm asking because it's 2 am for me and I don't know if waiting makes sense.
\r\nOn topic:
\r\nMy best (on full set) stacked model has 0.45970 on full and 0.45902 on public, unfortunately I didn't select it as it was worse from simple ensembles on public set (0.46443 public, 0.45588 private). Public leaderboard indeed should be ignored in favor of theoretically\r\n sound models...
Also, the averaging of 15 different R^2s washes out some of the issues with this. An N of 15 is not perfect for central limit theorem, but is a start in that direction. There will be a reduction in variance just based on the fact that the evaluation metric\r\n is the mean of the 15 activities. Although, in reality the N is different than 15 since the variance of the public R^2 for each activity is not homogeneous. The variance due to sampling in an over sampled activity such as 1 or 6 will be less than those from\r\n under sampled activities like 4.
\r\nOr in simpler terms, a 0.1 change in R^2 for a single activity only has a 0.00666... effect on the final evaluation metric. Now considering some may change positively or negatively, the overall impact may not be that large. Even considering a 0.0 R^2 would\r\n have a total effect of 0.04-0.6 on the final metric. (Although, an R^2 of 0.0 would be useful in building investment portfolios.)
" }, { "id": 15552, "postDate": "2012-10-14T01:16:51.303Z", "content": "Alright, looks like I was all worried for nothing. If I did the calcs right (which is not assured this time of night) the IQR of Private sample size for Activity 7 is ~386-399. That's not wide enough to be of any real issue. The min out of 1k simulations\r\n was 360, which still isn't that bad. The IQR for the public sample size of Activity 7 is 123-137. That is a bit more volatile, but the money doesn't depend on the public board.
" }, { "id": 15551, "postDate": "2012-10-14T01:01:02.823Z", "content": "Thanks for the info sir. Anyone want to do the calculations on the range of likely sample sizes for the smaller activities in the private data set? I might churn out some estimates here soon. And to note, I didn't actually see an error, I just saw a place\r\n where code re-use would likely cause problems. I know from my job how tricky proper weighting and stratification can be. This gives me more hope for our chances in this competition, but I do worry the top teams are pulling away far enough this might not matter.
" }, { "id": 15549, "postDate": "2012-10-14T00:25:42.877Z", "content": "The public leaderboard is based on a random 25% sample of the test set. It was constructed through a naive sampling of the combined 15 test sets, not a stratified sampling. Shea is correct in spotting the error in the test set split code, an error that will\r\n hopefully not come up again in future competitions with multiple data sets.
" }, { "id": 15523, "postDate": "2012-10-13T19:04:32.573Z", "content": "This is my first competition and I found myself using the leaderboard feedback a lot. Perhaps, a common beginners' mistake :). However, as I understood, the public scores is an average of the random 25% across all 15 activity sets. In this case I would not\r\n expect the private scores to be wildly different since averaging should make it more stable, right? (correct me if I'm wrong).
\r\nOn the other hand, if the private scores are not some random 25%, but say some unshuffled fixed part of the test set, then surely the final results will be very different. But I don't think it makes sense for Merck to use the latter evaluation and make the\r\n competitors' life harder because the best model would not be as good as it could be (as someone already mentioned in this thread).
" }, { "id": 15483, "postDate": "2012-10-12T20:21:59.600Z", "content": "I would expect lesser variation than the predicting biological response. I had participated in that and saw my position go up as I had not overfitted or used leaderboard feedback.
\r\nStill there will be significant variation. I am guessing Shea Parkes will be having the last laugh :)
" }, { "id": 15467, "postDate": "2012-10-12T05:36:28.927Z", "content": "I personally believe that Merck doesn't only want to obtain the best model, but also the best technology of its development. Used desсriptors are quite simple and work moderatуly well. If more comprehensive descriptors are used, they will be able to significantly\r\n improve predictions.
" }, { "id": 15459, "postDate": "2012-10-12T01:27:23.337Z", "content": "[quote=Halla;15455]
\r\nif I were Merck, I'd have the public 25% be based on the 25% where the internal benchmark does quite well, and award prizes on the 75% where the internal model does poorly. That might help them to obtain the best improvement to their internal model.
\r\n[/quote]
\r\nHalla, if Merck only cares about the actual test cases in the test set they have released, then they should just use all the test data on the public leaderboard. However, I strongly suspect that Merck is more interested in a model that generalizes to new\r\n test cases since they (hopefully!) already know the correct answer for all the test data in the competition. In this case Merck is best served by making performance on the 25% a random sample and thus predictive of performance on the other 75% since otherwise\r\n competitors will waste some of their effort and not end up with as good a solution as they could have given the time limit. And Merck then runs the risk of paying for the disclosure of a model that isn't as useful.
\r\nThe crux of the issue is that if the public leaderboard is based on cases that the internal model handles easily, competitors will tune their models for cases that are already handled well! If anything, I would think the REVERSE of what you said would be\r\n something they might want to do. But the reverse has its own problems, since there need to be cases they care about in both the public leaderboard data and the final data, otherwise the final evaluation step isn't useful.
" }, { "id": 15448, "postDate": "2012-10-11T20:17:16.577Z", "content": "Re: 50k+ Rows
\r\nIt is true there is more total data in this contest, but the stratification and equal activity weighting will drastically reduce the effective sample size. It wouldn't surprise me to see similar shuffle as the Biological Response contest mentioned above.\r\n BTW, we went from Public-38th to Private-6th on that contest. Way too many other people used explicit or implicit leaderboard feedback for that contest.
" }, { "id": 15445, "postDate": "2012-10-11T19:21:14.903Z", "content": "I am not familiar with that particular competition, but why such great variations in two different rankings? It almost seems as if there is no correlation between the two.
\r\n[quote=Halla;15438]
\r\nFor comparison: in the "Predicting a Biological Response" competition, here's who was ranked top 10 in the public leaderboard and how they fared on the private leaderboard.
\r\nPublic:
\r\n1. bluehat, 60 entries (Private 3rd place)
\r\n2. H. Solo, 66 entries (Private 29th place)
\r\n3. Outis, 36 entries (Private 25th place)
\r\n4. Efimov and Bers and Cragin and vsu, 172 entries (Private 8th place)
\r\n5. n_m, 102 entries (Private 36th place)
\r\n6. mkedz, 87 entries (Private 30th place)
\r\n7. bluemaster and imran, 91 entries (Private 7th place)
\r\n8. mike, 77 entries (Private 97th place)
\r\n9. uspminer, 39 entries (Private 22nd place)
\r\n10. HappyHour, 51 entries (Private 18th place)
The winners of the private competition, Winter is Coming and Sergey, were ranked 11th on the public leaderboard.
\r\n[/quote]
" }, { "id": 15440, "postDate": "2012-10-11T18:06:58.927Z", "content": "as usual great point by Shea.
\r\nMy understanding is that cor is calculated separately for each activity set. The scores are then averaged. Is'nt that correct?
" }, { "id": 15439, "postDate": "2012-10-11T18:04:28.803Z", "content": "that's why I don't trust the leaderboard scores for this contest. I have seen bizarre things happen in Impermium contest.
" }, { "id": 15429, "postDate": "2012-10-11T14:36:09.473Z", "content": "Since we're closing in on the end, I thought I'd spend a bit trying to judge the likely volatility between public and private leaderboard scores. A ~25% public split was chosen for this contest. Was that sampling stratified by activity when it was implemented?\r\n If not, I think we may be in for a very bumpy ride. I just noticed the sample scoring script "RSquared.R" doesn't stratify by activity either. I'm sure the actual evaluation isn't being done in R, but I do hope it is properly stratified to match the contest\r\n description." }, { "id": 15758, "postDate": "2012-10-17T00:28:32.993Z", "rawMarkdown": "", "isDeleted": true }, { "id": 15520, "postDate": "2012-10-13T16:41:29.270Z", "rawMarkdown": "", "isDeleted": true }, { "id": 15517, "postDate": "2012-10-13T16:14:57.550Z", "rawMarkdown": "", "isDeleted": true }, { "id": 15461, "postDate": "2012-10-12T01:40:43.647Z", "rawMarkdown": "", "isDeleted": true }, { "id": 15455, "postDate": "2012-10-12T00:27:10.377Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 15449, "postDate": "2012-10-11T20:25:07.837Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 15446, "postDate": "2012-10-11T19:41:27.513Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 15438, "postDate": "2012-10-11T17:52:08.683Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 15529, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-13T20:15:55.670000", "content": "I agree the concept of doing the public/private (25%/75%) split by sample date would make for a very misleading public leaderboard. I rather doubt they did that. However, since they haven't really responded at all thus far I suppose we'll never know. My\r\n original question was whether they did stratified sampling by activity. By that I meant is the public leaderboard based on 25% of activity #1, 25% of activity #2, etc. If they didn't stratify it, the sample % of the smaller activities could easily range from\r\n 10%-40% I imagine (I'm sure someone will calculate the actual range shortly.) I mainly asked this because I could see an error like this creep in if they re-used code from other contests to make the public/private split.
", "votes": 1, "replies": [] }, { "id": 15463, "author_name": "gggg", "author_url": "", "post_date": "2012-10-12T03:10:21.003000", "content": "Every single team who is on the public leaderboard has tuned their models to the public leaderboard unless they didn't look at their position on the leaderboard, but only at the warnings/errors of their submission.
", "votes": 1, "replies": [] }, { "id": 15818, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-10-17T15:05:08.877000", "content": "So private leaderboard was very similar to public leaderboard. Not much volatility
", "votes": 0, "replies": [] }, { "id": 15777, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-10-17T04:06:26.573000", "content": "me wondering as well - it is usually readily available.
", "votes": 0, "replies": [] }, { "id": 15757, "author_name": "pzaw", "author_url": "", "post_date": "2012-10-17T00:17:58.157000", "content": "Hi, what's the standard delay for displaying full leaderboard? Minutes or days? I'm asking because it's 2 am for me and I don't know if waiting makes sense.
\r\nOn topic:
\r\nMy best (on full set) stacked model has 0.45970 on full and 0.45902 on public, unfortunately I didn't select it as it was worse from simple ensembles on public set (0.46443 public, 0.45588 private). Public leaderboard indeed should be ignored in favor of theoretically\r\n sound models...
Also, the averaging of 15 different R^2s washes out some of the issues with this. An N of 15 is not perfect for central limit theorem, but is a start in that direction. There will be a reduction in variance just based on the fact that the evaluation metric\r\n is the mean of the 15 activities. Although, in reality the N is different than 15 since the variance of the public R^2 for each activity is not homogeneous. The variance due to sampling in an over sampled activity such as 1 or 6 will be less than those from\r\n under sampled activities like 4.
\r\nOr in simpler terms, a 0.1 change in R^2 for a single activity only has a 0.00666... effect on the final evaluation metric. Now considering some may change positively or negatively, the overall impact may not be that large. Even considering a 0.0 R^2 would\r\n have a total effect of 0.04-0.6 on the final metric. (Although, an R^2 of 0.0 would be useful in building investment portfolios.)
", "votes": 0, "replies": [] }, { "id": 15552, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-14T01:16:51.303000", "content": "Alright, looks like I was all worried for nothing. If I did the calcs right (which is not assured this time of night) the IQR of Private sample size for Activity 7 is ~386-399. That's not wide enough to be of any real issue. The min out of 1k simulations\r\n was 360, which still isn't that bad. The IQR for the public sample size of Activity 7 is 123-137. That is a bit more volatile, but the money doesn't depend on the public board.
", "votes": 0, "replies": [] }, { "id": 15551, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-14T01:01:02.823000", "content": "Thanks for the info sir. Anyone want to do the calculations on the range of likely sample sizes for the smaller activities in the private data set? I might churn out some estimates here soon. And to note, I didn't actually see an error, I just saw a place\r\n where code re-use would likely cause problems. I know from my job how tricky proper weighting and stratification can be. This gives me more hope for our chances in this competition, but I do worry the top teams are pulling away far enough this might not matter.
", "votes": 0, "replies": [] }, { "id": 15549, "author_name": "joycenv", "author_url": "", "post_date": "2012-10-14T00:25:42.877000", "content": "The public leaderboard is based on a random 25% sample of the test set. It was constructed through a naive sampling of the combined 15 test sets, not a stratified sampling. Shea is correct in spotting the error in the test set split code, an error that will\r\n hopefully not come up again in future competitions with multiple data sets.
", "votes": 0, "replies": [] }, { "id": 15523, "author_name": "dmitrim", "author_url": "", "post_date": "2012-10-13T19:04:32.573000", "content": "This is my first competition and I found myself using the leaderboard feedback a lot. Perhaps, a common beginners' mistake :). However, as I understood, the public scores is an average of the random 25% across all 15 activity sets. In this case I would not\r\n expect the private scores to be wildly different since averaging should make it more stable, right? (correct me if I'm wrong).
\r\nOn the other hand, if the private scores are not some random 25%, but say some unshuffled fixed part of the test set, then surely the final results will be very different. But I don't think it makes sense for Merck to use the latter evaluation and make the\r\n competitors' life harder because the best model would not be as good as it could be (as someone already mentioned in this thread).
", "votes": 0, "replies": [] }, { "id": 15483, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-10-12T20:21:59.600000", "content": "I would expect lesser variation than the predicting biological response. I had participated in that and saw my position go up as I had not overfitted or used leaderboard feedback.
\r\nStill there will be significant variation. I am guessing Shea Parkes will be having the last laugh :)
", "votes": 0, "replies": [] }, { "id": 15467, "author_name": "Drrr", "author_url": "", "post_date": "2012-10-12T05:36:28.927000", "content": "I personally believe that Merck doesn't only want to obtain the best model, but also the best technology of its development. Used desсriptors are quite simple and work moderatуly well. If more comprehensive descriptors are used, they will be able to significantly\r\n improve predictions.
", "votes": 0, "replies": [] }, { "id": 15459, "author_name": "gggg", "author_url": "", "post_date": "2012-10-12T01:27:23.337000", "content": "[quote=Halla;15455]
\r\nif I were Merck, I'd have the public 25% be based on the 25% where the internal benchmark does quite well, and award prizes on the 75% where the internal model does poorly. That might help them to obtain the best improvement to their internal model.
\r\n[/quote]
\r\nHalla, if Merck only cares about the actual test cases in the test set they have released, then they should just use all the test data on the public leaderboard. However, I strongly suspect that Merck is more interested in a model that generalizes to new\r\n test cases since they (hopefully!) already know the correct answer for all the test data in the competition. In this case Merck is best served by making performance on the 25% a random sample and thus predictive of performance on the other 75% since otherwise\r\n competitors will waste some of their effort and not end up with as good a solution as they could have given the time limit. And Merck then runs the risk of paying for the disclosure of a model that isn't as useful.
\r\nThe crux of the issue is that if the public leaderboard is based on cases that the internal model handles easily, competitors will tune their models for cases that are already handled well! If anything, I would think the REVERSE of what you said would be\r\n something they might want to do. But the reverse has its own problems, since there need to be cases they care about in both the public leaderboard data and the final data, otherwise the final evaluation step isn't useful.
", "votes": 0, "replies": [] }, { "id": 15448, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-11T20:17:16.577000", "content": "Re: 50k+ Rows
\r\nIt is true there is more total data in this contest, but the stratification and equal activity weighting will drastically reduce the effective sample size. It wouldn't surprise me to see similar shuffle as the Biological Response contest mentioned above.\r\n BTW, we went from Public-38th to Private-6th on that contest. Way too many other people used explicit or implicit leaderboard feedback for that contest.
", "votes": 0, "replies": [] }, { "id": 15445, "author_name": "shinmagi", "author_url": "", "post_date": "2012-10-11T19:21:14.903000", "content": "I am not familiar with that particular competition, but why such great variations in two different rankings? It almost seems as if there is no correlation between the two.
\r\n[quote=Halla;15438]
\r\nFor comparison: in the "Predicting a Biological Response" competition, here's who was ranked top 10 in the public leaderboard and how they fared on the private leaderboard.
\r\nPublic:
\r\n1. bluehat, 60 entries (Private 3rd place)
\r\n2. H. Solo, 66 entries (Private 29th place)
\r\n3. Outis, 36 entries (Private 25th place)
\r\n4. Efimov and Bers and Cragin and vsu, 172 entries (Private 8th place)
\r\n5. n_m, 102 entries (Private 36th place)
\r\n6. mkedz, 87 entries (Private 30th place)
\r\n7. bluemaster and imran, 91 entries (Private 7th place)
\r\n8. mike, 77 entries (Private 97th place)
\r\n9. uspminer, 39 entries (Private 22nd place)
\r\n10. HappyHour, 51 entries (Private 18th place)
The winners of the private competition, Winter is Coming and Sergey, were ranked 11th on the public leaderboard.
\r\n[/quote]
", "votes": 0, "replies": [] }, { "id": 15440, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-10-11T18:06:58.927000", "content": "as usual great point by Shea.
\r\nMy understanding is that cor is calculated separately for each activity set. The scores are then averaged. Is'nt that correct?
", "votes": 0, "replies": [] }, { "id": 15439, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-10-11T18:04:28.803000", "content": "that's why I don't trust the leaderboard scores for this contest. I have seen bizarre things happen in Impermium contest.
", "votes": 0, "replies": [] }, { "id": 15758, "author_name": "", "author_url": "", "post_date": "2012-10-17T00:28:32.993000", "content": "", "votes": 0, "replies": [] }, { "id": 15520, "author_name": "", "author_url": "", "post_date": "2012-10-13T16:41:29.270000", "content": "", "votes": 0, "replies": [] }, { "id": 15517, "author_name": "", "author_url": "", "post_date": "2012-10-13T16:14:57.550000", "content": "", "votes": 0, "replies": [] }, { "id": 15461, "author_name": "", "author_url": "", "post_date": "2012-10-12T01:40:43.647000", "content": "", "votes": 0, "replies": [] }, { "id": 15455, "author_name": "", "author_url": "", "post_date": "2012-10-12T00:27:10.377000", "content": "", "votes": 1, "replies": [] }, { "id": 15449, "author_name": "", "author_url": "", "post_date": "2012-10-11T20:25:07.837000", "content": "", "votes": 1, "replies": [] }, { "id": 15446, "author_name": "", "author_url": "", "post_date": "2012-10-11T19:41:27.513000", "content": "", "votes": 1, "replies": [] }, { "id": 15438, "author_name": "", "author_url": "", "post_date": "2012-10-11T17:52:08.683000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "15529": "", "15463": "", "15818": "", "15777": "", "15757": "", "15555": "", "15552": "", "15551": "", "15549": "", "15523": "", "15483": "", "15467": "", "15459": "", "15448": "", "15445": "", "15440": "", "15439": "", "15429": "", "15758": "", "15520": "", "15517": "", "15461": "", "15455": "", "15449": "", "15446": "", "15438": "" } }