Hello everyone,
\r\nWe are curently doing some work behind the scenes to verify the results in this competition. We expect to have the final results ready by the end of this week, if not sooner. Until then, only the public leaderboard will remain visible. Thanks for your patience.
", "messages": [ { "id": "15763", "postDate": "10/17/2012 02:07:22", "content": "Hello everyone,
\r\nWe are curently doing some work behind the scenes to verify the results in this competition. We expect to have the final results ready by the end of this week, if not sooner. Until then, only the public leaderboard will remain visible. Thanks for your patience.
", "rawMarkdown": "", "votes": null }, { "id": "15778", "postDate": "10/17/2012 04:07:19", "content": "Usually the leaderboard is immediately available at end of competition. Hope no change in rules like new dataset - or changed private dataset etc
", "rawMarkdown": "", "votes": null }, { "id": "15781", "postDate": "10/17/2012 05:16:38", "content": "[quote=jcnhvnhck;15763]
\r\nHello everyone,
\r\nWe are curently doing some work behind the scenes to verify the results in this competition. We expect to have the final results ready by the end of this week, if not sooner. Until then, only the public leaderboard will remain visible. Thanks for your patience.
\r\n[/quote]
\r\nThe private score has come out, could you give me the unoffical leaderboard now? Besides, how do you verify the results, the private score will be changed?
", "rawMarkdown": "", "votes": null }, { "id": "15794", "postDate": "10/17/2012 08:23:53", "content": "Posted some analysis of the implied private leaderboard here https://www.kaggle.com/c/MerckActivity/forums/t/2908/unofficial-leaderboard/15792#post15792
", "rawMarkdown": "", "votes": null }, { "id": "15815", "postDate": "10/17/2012 14:37:34", "content": "Why don't kaggle just make the leaderboard scores available and final standings? This last minute hassle is unwarranted
", "rawMarkdown": "", "votes": null }, { "id": "15819", "postDate": "10/17/2012 15:08:15", "content": "BlackMagic, it's likely that Kaggle wants to make sure none of the winners violated the rules of the competition. Given the number of sock puppets in this competition, it is clear that someone cheated. Let's give the administrators a chance to run whatever\r\n tools and checks are at their disposal before posting the final leaderboard. I see zero upside to Kaggle in being impatient and hasty.
", "rawMarkdown": "", "votes": null }, { "id": "15820", "postDate": "10/17/2012 15:10:21", "content": "yes, that's a good callout.
\r\nGood to weed out sock puppets.
I think the private scores we saw on our submissions might not be correct. Because they are very close to the public leaderboard scores.
\r\nThis is very unlikely - as there was a lot of variance expected (as we see in experienced campaigner Shea Parkes posts)
Just because there can be a lot of variance doesn't mean there will be.
\r\nStill, given how little variance there was, it gives me hope they didn't stratify the R-sqrs and we might shuffle higher. I agree though, checking out puppet accounts is more likely than altering the evaluation metric.
", "rawMarkdown": "", "votes": null }, { "id": "15840", "postDate": "10/17/2012 17:25:41", "content": "yes that's a key question;
\r\nin the final evaluation they are supposed to find Rsquared separately for each molecule set and average all the Rsquares. Maybe they just took an Rsquared over the entire set.
I am surprised that public and private leaderboards scores are so close. I was expecting huge variation coming from the earlier biological response competition
", "rawMarkdown": "", "votes": null }, { "id": "15845", "postDate": "10/17/2012 17:45:55", "content": "Black Magic,
\r\nYes, I have also expected this, such close results are very strange...
", "rawMarkdown": "", "votes": null }, { "id": "15848", "postDate": "10/17/2012 19:04:00", "content": "[quote=Black Magic;15840]
\r\nI am surprised that public and private leaderboards scores are so close. I was expecting huge variation coming from the earlier biological response competition
\r\n[/quote]
\r\nIn the biological response competition you had ~2400 molecules that were totally different. That was clearly seen when private scores were published and almost everyone got a much better logloss on those as if molecules for the public leaderboard where somewhat\r\n harder to predict (on average) than molecules used for the private leaderboard.
\r\nHere you have a bunch of clusters of molecules that are similar because of the time split. It's normal that when you develop a new drug, for example some vitamin D derivative, you first make some small alteration to the molecule e.g. add a methyl group and\r\n mesaure its activity, then you add an acetyl group, then both and so on. After you switch to some other molecule e.g. some nucleotide and synthesize its derivatives and test its binding affinity to some receptor and so on. In the end you end up with a bunch\r\n of clusters of derivatives of some molecule. If you look at it as a (financial) time series it's somewhat similar to volatility clustering.
\r\nNow, having the public/private test set split done by random sampling it's probable that you'll end up with some representatives of each cluster in the public test set. That means that your results one the public leaderboard should be highly predictive of\r\n the private leaderboard. It also means that people that probed the public leaderboard had a serious advantage over people like me that looked only on their CV scores (of course done sequentially and not by mixing all the molecules together which would be just\r\n data snooping/leaking). If the public/private split was based on time the results would be totally different - promoting people that did not do data snooping. This also means that probably many of the models build by users (maybe even from the top ten) won't\r\n necesserily generalize to some different test set. If I were Merck I would definitely release a different test set with new data to see which users actually have good models and which don't.
", "rawMarkdown": "", "votes": null }, { "id": "15849", "postDate": "10/17/2012 19:49:43", "content": "very good point.
\r\nMerck should verify results differently. Current method is just favoring those that did leaderboard probing UNLESS there is a method for them to evaluate using a different metric
Are merck representatives on the forum? Would love to hear their thoughts
", "rawMarkdown": "", "votes": null }, { "id": "15859", "postDate": "10/17/2012 22:49:09", "content": "I just realized one could make a submission with horrible predictions for Activity #7. If the stratification were correct, the leaderboard scores should decline by 6.7%. I didn't consider doing such things during the contest since it went into a morally\r\n ambiguous realm. Post-contest I don't have such qualms. I'll likely do the experiment later tonight.
", "rawMarkdown": "", "votes": null }, { "id": "15868", "postDate": "10/18/2012 01:15:18", "content": "Well, the experiment was a bit more extensive that I thought it would need to be, but I feel satisfied the scores are properly stratified by activity. I assume they are spending this time checking out potential puppet accounts.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 15778, "author_name": "rkirana", "author_url": "", "post_date": "10/17/2012 04:07:19", "content": "Usually the leaderboard is immediately available at end of competition. Hope no change in rules like new dataset - or changed private dataset etc
", "votes": null, "replies": [] }, { "id": 15781, "author_name": "lijiefei", "author_url": "", "post_date": "10/17/2012 05:16:38", "content": "[quote=jcnhvnhck;15763]
\r\nHello everyone,
\r\nWe are curently doing some work behind the scenes to verify the results in this competition. We expect to have the final results ready by the end of this week, if not sooner. Until then, only the public leaderboard will remain visible. Thanks for your patience.
\r\n[/quote]
\r\nThe private score has come out, could you give me the unoffical leaderboard now? Besides, how do you verify the results, the private score will be changed?
", "votes": null, "replies": [] }, { "id": 15794, "author_name": "jasontigg", "author_url": "", "post_date": "10/17/2012 08:23:53", "content": "Posted some analysis of the implied private leaderboard here https://www.kaggle.com/c/MerckActivity/forums/t/2908/unofficial-leaderboard/15792#post15792
", "votes": null, "replies": [] }, { "id": 15815, "author_name": "rkirana", "author_url": "", "post_date": "10/17/2012 14:37:34", "content": "Why don't kaggle just make the leaderboard scores available and final standings? This last minute hassle is unwarranted
", "votes": null, "replies": [] }, { "id": 15819, "author_name": "hallayang", "author_url": "", "post_date": "10/17/2012 15:08:15", "content": "BlackMagic, it's likely that Kaggle wants to make sure none of the winners violated the rules of the competition. Given the number of sock puppets in this competition, it is clear that someone cheated. Let's give the administrators a chance to run whatever\r\n tools and checks are at their disposal before posting the final leaderboard. I see zero upside to Kaggle in being impatient and hasty.
", "votes": null, "replies": [] }, { "id": 15820, "author_name": "rkirana", "author_url": "", "post_date": "10/17/2012 15:10:21", "content": "yes, that's a good callout.
\r\nGood to weed out sock puppets.
I think the private scores we saw on our submissions might not be correct. Because they are very close to the public leaderboard scores.
\r\nThis is very unlikely - as there was a lot of variance expected (as we see in experienced campaigner Shea Parkes posts)
Just because there can be a lot of variance doesn't mean there will be.
\r\nStill, given how little variance there was, it gives me hope they didn't stratify the R-sqrs and we might shuffle higher. I agree though, checking out puppet accounts is more likely than altering the evaluation metric.
", "votes": null, "replies": [] }, { "id": 15840, "author_name": "rkirana", "author_url": "", "post_date": "10/17/2012 17:25:41", "content": "yes that's a key question;
\r\nin the final evaluation they are supposed to find Rsquared separately for each molecule set and average all the Rsquares. Maybe they just took an Rsquared over the entire set.
I am surprised that public and private leaderboards scores are so close. I was expecting huge variation coming from the earlier biological response competition
", "votes": null, "replies": [] }, { "id": 15845, "author_name": "efimov", "author_url": "", "post_date": "10/17/2012 17:45:55", "content": "Black Magic,
\r\nYes, I have also expected this, such close results are very strange...
", "votes": null, "replies": [] }, { "id": 15848, "author_name": "xylen333", "author_url": "", "post_date": "10/17/2012 19:04:00", "content": "[quote=Black Magic;15840]
\r\nI am surprised that public and private leaderboards scores are so close. I was expecting huge variation coming from the earlier biological response competition
\r\n[/quote]
\r\nIn the biological response competition you had ~2400 molecules that were totally different. That was clearly seen when private scores were published and almost everyone got a much better logloss on those as if molecules for the public leaderboard where somewhat\r\n harder to predict (on average) than molecules used for the private leaderboard.
\r\nHere you have a bunch of clusters of molecules that are similar because of the time split. It's normal that when you develop a new drug, for example some vitamin D derivative, you first make some small alteration to the molecule e.g. add a methyl group and\r\n mesaure its activity, then you add an acetyl group, then both and so on. After you switch to some other molecule e.g. some nucleotide and synthesize its derivatives and test its binding affinity to some receptor and so on. In the end you end up with a bunch\r\n of clusters of derivatives of some molecule. If you look at it as a (financial) time series it's somewhat similar to volatility clustering.
\r\nNow, having the public/private test set split done by random sampling it's probable that you'll end up with some representatives of each cluster in the public test set. That means that your results one the public leaderboard should be highly predictive of\r\n the private leaderboard. It also means that people that probed the public leaderboard had a serious advantage over people like me that looked only on their CV scores (of course done sequentially and not by mixing all the molecules together which would be just\r\n data snooping/leaking). If the public/private split was based on time the results would be totally different - promoting people that did not do data snooping. This also means that probably many of the models build by users (maybe even from the top ten) won't\r\n necesserily generalize to some different test set. If I were Merck I would definitely release a different test set with new data to see which users actually have good models and which don't.
", "votes": null, "replies": [] }, { "id": 15849, "author_name": "rkirana", "author_url": "", "post_date": "10/17/2012 19:49:43", "content": "very good point.
\r\nMerck should verify results differently. Current method is just favoring those that did leaderboard probing UNLESS there is a method for them to evaluate using a different metric
Are merck representatives on the forum? Would love to hear their thoughts
", "votes": null, "replies": [] }, { "id": 15859, "author_name": "biasvariance", "author_url": "", "post_date": "10/17/2012 22:49:09", "content": "I just realized one could make a submission with horrible predictions for Activity #7. If the stratification were correct, the leaderboard scores should decline by 6.7%. I didn't consider doing such things during the contest since it went into a morally\r\n ambiguous realm. Post-contest I don't have such qualms. I'll likely do the experiment later tonight.
", "votes": null, "replies": [] }, { "id": 15868, "author_name": "biasvariance", "author_url": "", "post_date": "10/18/2012 01:15:18", "content": "Well, the experiment was a bit more extensive that I thought it would need to be, but I feel satisfied the scores are properly stratified by activity. I assume they are spending this time checking out potential puppet accounts.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "15763": "", "15778": "", "15781": "", "15794": "", "15815": "", "15819": "", "15820": "", "15833": "", "15838": "", "15840": "", "15845": "", "15848": "", "15849": "", "15859": "", "15868": "" }, "source": "meta" }