I invested couple of days in this competition and its just getting weirder and weirder.
\n1) Useless tutorial notebook (didn't get any help whatsoever)
\n2) Really weird evaluation metric (I'm really interested to know why this particular metric is chosen)
\n3) \"Public leaderboard targets are publicly available and are provided as part of the competition dataset\". Weirdest of them all to make public LB useless. How are we supposed to perform validation when there are 100 ppl already with perfect scores?
And finally the clincher… I'll probably die of old age before the submission gets scored. It's really that slow.
\nI'd love to get some perspective here. Anyone? Thoughts?
", "messages": [ { "id": "1573555", "postDate": "11/06/2021 18:11:58", "content": "I invested couple of days in this competition and its just getting weirder and weirder.
\n1) Useless tutorial notebook (didn't get any help whatsoever)
\n2) Really weird evaluation metric (I'm really interested to know why this particular metric is chosen)
\n3) \"Public leaderboard targets are publicly available and are provided as part of the competition dataset\". Weirdest of them all to make public LB useless. How are we supposed to perform validation when there are 100 ppl already with perfect scores?
And finally the clincher… I'll probably die of old age before the submission gets scored. It's really that slow.
\nI'd love to get some perspective here. Anyone? Thoughts?
", "rawMarkdown": "I invested couple of days in this competition and its just getting weirder and weirder.\n\n1) Useless tutorial notebook (didn't get any help whatsoever)\n2) Really weird evaluation metric (I'm really interested to know why this particular metric is chosen)\n3) \"*Public leaderboard targets are publicly available and are provided as part of the competition dataset*\". Weirdest of them all to make public LB useless. How are we supposed to perform validation when there are 100 ppl already with perfect scores?\n\nAnd finally the clincher... I'll probably die of old age before the submission gets scored. It's really that slow.\n\nI'd love to get some perspective here. Anyone? Thoughts?", "votes": null }, { "id": "1574448", "postDate": "11/07/2021 14:59:36", "content": "Did you figure out how the evaluation metric is computed? For the mean in the weighted correlation, is it the cross-sectional weighted mean or the global weighted mean?
\nThe wikipage is not pretty clear about this, and ref links are all dead.
", "rawMarkdown": "Did you figure out how the evaluation metric is computed? For the mean in the weighted correlation, is it the cross-sectional weighted mean or the global weighted mean? \n\nThe wikipage is not pretty clear about this, and ref links are all dead.", "votes": null }, { "id": "1574605", "postDate": "11/07/2021 17:50:46", "content": "https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285834 Exactly my thoughts
", "rawMarkdown": "https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285834 Exactly my thoughts", "votes": null }, { "id": "1574629", "postDate": "11/07/2021 18:14:38", "content": "\n\nHow are we supposed to perform validation when there are 100 ppl already with perfect scores?
\n
The public leaderboard is going to have no utility in this competition. The best you can do is hold out some of the provided training data for testing/validation, try to replicate their scoring metric, and submit your best results.
\n\n\nAnd finally the clincher… I'll probably die of old age before the submission gets scored. It's really that slow.
\n
I am very dissatisfied with how long the scoring takes. I don't see why with a baseline model that took two minutes to train its takes over half an hour to deliver a scoring result.
", "rawMarkdown": "> How are we supposed to perform validation when there are 100 ppl already with perfect scores?\n\nThe public leaderboard is going to have no utility in this competition. The best you can do is hold out some of the provided training data for testing/validation, try to replicate their scoring metric, and submit your best results. \n\n> And finally the clincher… I'll probably die of old age before the submission gets scored. It's really that slow.\n\nI am very dissatisfied with how long the scoring takes. I don't see why with a baseline model that took two minutes to train its takes over half an hour to deliver a scoring result.", "votes": null }, { "id": "1574778", "postDate": "11/07/2021 21:29:18", "content": "The problems you encounter are the pendant of the good aspects of this competition.
\nHaving a Time series API => this competition is better than Optiver competition
\nHaving deanonymised data => this competition is better than Jane Street competition
\nThe weird metrics are part of the challenge IMO.
\nThe only thing that could really be changed are the public LB (using simulated data, putting the warning banner in bright red, deactivating it entirely) and the intro notebook. If the data are public there is no reason to not share the code used for computations. (Target, VWAP, base currency… etc.)
The public leaderboard can have utility, check this notebook about making a meaningful LB substracting the test data from the train data.
", "rawMarkdown": "The public leaderboard can have utility, check this [notebook ](https://www.kaggle.com/julian3833/s-proposal-for-a-meaningful-lb) about making a meaningful LB substracting the test data from the train data.", "votes": null }, { "id": "1575248", "postDate": "11/08/2021 09:16:48", "content": "Yes, we have some missing details here and there.
", "rawMarkdown": "Yes, we have some missing details here and there.", "votes": null }, { "id": "1575367", "postDate": "11/08/2021 10:56:13", "content": "100%. I think though that the public LB makes all the difference.. It doesn't feel like a \"kaggle competition\" without the leaderboard.
\nAnd yes the metric is weird but if they release the labeling code then.. we can live with that? 🤷
", "rawMarkdown": "100%. I think though that the public LB makes all the difference.. It doesn't feel like a \"kaggle competition\" without the leaderboard. \n\nAnd yes the metric is weird but if they release the labeling code then.. we can live with that? 🤷", "votes": null }, { "id": "1581011", "postDate": "11/13/2021 10:16:09", "content": "The individual weight is provided in asset_details.csv
", "rawMarkdown": "The individual weight is provided in asset_details.csv", "votes": null }, { "id": "1581477", "postDate": "11/13/2021 20:35:51", "content": "\n\n3) \"Public leaderboard targets are publicly available and are provided as part of the competition dataset\". Weirdest of them all to make public LB useless. How are we supposed to perform validation when there are 100 ppl already with perfect scores?
\n
I find it a good strategy against people seeking for \"leakage\".
", "rawMarkdown": "> 3) \"Public leaderboard targets are publicly available and are provided as part of the competition dataset\". Weirdest of them all to make public LB useless. How are we supposed to perform validation when there are 100 ppl already with perfect scores?\n\nI find it a good strategy against people seeking for \"leakage\".", "votes": null } ], "comments": [ { "id": 1574448, "author_name": "hantianzheng", "author_url": "", "post_date": "11/07/2021 14:59:36", "content": "Did you figure out how the evaluation metric is computed? For the mean in the weighted correlation, is it the cross-sectional weighted mean or the global weighted mean?
\nThe wikipage is not pretty clear about this, and ref links are all dead.
", "votes": null, "replies": [ { "id": 1581011, "author_name": "ajumzu", "author_url": "", "post_date": "11/13/2021 10:16:09", "content": "The individual weight is provided in asset_details.csv
", "votes": null, "replies": [] } ] }, { "id": 1574605, "author_name": "munumbutt", "author_url": "", "post_date": "11/07/2021 17:50:46", "content": "https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285834 Exactly my thoughts
", "votes": null, "replies": [] }, { "id": 1574629, "author_name": "attackgnome", "author_url": "", "post_date": "11/07/2021 18:14:38", "content": "\n\nHow are we supposed to perform validation when there are 100 ppl already with perfect scores?
\n
The public leaderboard is going to have no utility in this competition. The best you can do is hold out some of the provided training data for testing/validation, try to replicate their scoring metric, and submit your best results.
\n\n\nAnd finally the clincher… I'll probably die of old age before the submission gets scored. It's really that slow.
\n
I am very dissatisfied with how long the scoring takes. I don't see why with a baseline model that took two minutes to train its takes over half an hour to deliver a scoring result.
", "votes": null, "replies": [ { "id": 1575236, "author_name": "ajpass", "author_url": "", "post_date": "11/08/2021 09:08:36", "content": "The public leaderboard can have utility, check this notebook about making a meaningful LB substracting the test data from the train data.
", "votes": null, "replies": [] } ] }, { "id": 1574778, "author_name": "lucasmorin", "author_url": "", "post_date": "11/07/2021 21:29:18", "content": "The problems you encounter are the pendant of the good aspects of this competition.
\nHaving a Time series API => this competition is better than Optiver competition
\nHaving deanonymised data => this competition is better than Jane Street competition
\nThe weird metrics are part of the challenge IMO.
\nThe only thing that could really be changed are the public LB (using simulated data, putting the warning banner in bright red, deactivating it entirely) and the intro notebook. If the data are public there is no reason to not share the code used for computations. (Target, VWAP, base currency… etc.)
Yes, we have some missing details here and there.
", "votes": null, "replies": [] }, { "id": 1575367, "author_name": "", "author_url": "", "post_date": "11/08/2021 10:56:13", "content": "100%. I think though that the public LB makes all the difference.. It doesn't feel like a \"kaggle competition\" without the leaderboard.
\nAnd yes the metric is weird but if they release the labeling code then.. we can live with that? 🤷
", "votes": null, "replies": [] } ] }, { "id": 1581477, "author_name": "yuanzhezhou", "author_url": "", "post_date": "11/13/2021 20:35:51", "content": "\n\n3) \"Public leaderboard targets are publicly available and are provided as part of the competition dataset\". Weirdest of them all to make public LB useless. How are we supposed to perform validation when there are 100 ppl already with perfect scores?
\n
I find it a good strategy against people seeking for \"leakage\".
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1573555": "I invested couple of days in this competition and its just getting weirder and weirder.\n\n1) Useless tutorial notebook (didn't get any help whatsoever)\n2) Really weird evaluation metric (I'm really interested to know why this particular metric is chosen)\n3) \"*Public leaderboard targets are publicly available and are provided as part of the competition dataset*\". Weirdest of them all to make public LB useless. How are we supposed to perform validation when there are 100 ppl already with perfect scores?\n\nAnd finally the clincher... I'll probably die of old age before the submission gets scored. It's really that slow.\n\nI'd love to get some perspective here. Anyone? Thoughts?", "1574448": "Did you figure out how the evaluation metric is computed? For the mean in the weighted correlation, is it the cross-sectional weighted mean or the global weighted mean? \n\nThe wikipage is not pretty clear about this, and ref links are all dead.", "1574605": "https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285834 Exactly my thoughts", "1574629": "> How are we supposed to perform validation when there are 100 ppl already with perfect scores?\n\nThe public leaderboard is going to have no utility in this competition. The best you can do is hold out some of the provided training data for testing/validation, try to replicate their scoring metric, and submit your best results. \n\n> And finally the clincher… I'll probably die of old age before the submission gets scored. It's really that slow.\n\nI am very dissatisfied with how long the scoring takes. I don't see why with a baseline model that took two minutes to train its takes over half an hour to deliver a scoring result.", "1574778": "The problems you encounter are the pendant of the good aspects of this competition. \nHaving a Time series API => this competition is better than Optiver competition\nHaving deanonymised data => this competition is better than Jane Street competition\nThe weird metrics are part of the challenge IMO.\nThe only thing that could really be changed are the public LB (using simulated data, putting the warning banner in bright red, deactivating it entirely) and the intro notebook. If the data are public there is no reason to not share the code used for computations. (Target, VWAP, base currency... etc.)", "1575236": "The public leaderboard can have utility, check this [notebook ](https://www.kaggle.com/julian3833/s-proposal-for-a-meaningful-lb) about making a meaningful LB substracting the test data from the train data.", "1575248": "Yes, we have some missing details here and there.", "1575367": "100%. I think though that the public LB makes all the difference.. It doesn't feel like a \"kaggle competition\" without the leaderboard. \n\nAnd yes the metric is weird but if they release the labeling code then.. we can live with that? 🤷", "1581011": "The individual weight is provided in asset_details.csv", "1581477": "> 3) \"Public leaderboard targets are publicly available and are provided as part of the competition dataset\". Weirdest of them all to make public LB useless. How are we supposed to perform validation when there are 100 ppl already with perfect scores?\n\nI find it a good strategy against people seeking for \"leakage\"." }, "source": "meta" }