Hi Kagglers!
\nPublic Leaderboard
\nWe wanted to take a moment to clarify some concerns/questions surrounding the public leaderboard. As this competition makes use of publically available data (or at least data which can be reasonably approximated using public sources), there is unfortunately no way for us to ‘hold out’ data to provide a public leaderboard without the risk of data leakage. This would be true regardless of the evaluation metric and could not be solved using simulated data (if we could accurately simulate crypto markets then we would already be able to predict them!).
For transparency, we therefore decided to explicitly include the targets used to calculate public scores in the released data such that all competitors had equal access to them. Unfortunately, this does mean that it is possible to submit unrealistic scores to the public leaderboard and you may not be able to directly compare your score to other competitors.
\nFor this reason we suggest that during your experimentation you implement your own validation procedure using best practices to avoid data leakage. The submission mechanism should only be used to verify that your code is able to interact with the time-series API correctly and produces a score commensurate with your own offline validation.
\nAPI
\nAs a side note, the API is expected to deliver data in ~10 minutes, thus if using a trivial model (e.g. submitting dummy forecasts), this should be roughly the total submission time of your solution. Keep in mind, however, that your inference function will get called once per time step, thus any inefficiencies here will cause large increases in the total run time of your model. Also note that at evaluation time you will still have access to all the training data. If you combine this with a copy of all data provided to you from the API, this will allow you to produce predictions using time series models from the very first step.
Public Leaderboard Precision
\nLastly, we have increased the precision of the public leaderboard given the range of expected scores.
Happy Kaggling!
", "messages": [ { "id": "1594090", "postDate": "11/24/2021 14:30:57", "content": "Hi Kagglers!
\nPublic Leaderboard
\nWe wanted to take a moment to clarify some concerns/questions surrounding the public leaderboard. As this competition makes use of publically available data (or at least data which can be reasonably approximated using public sources), there is unfortunately no way for us to ‘hold out’ data to provide a public leaderboard without the risk of data leakage. This would be true regardless of the evaluation metric and could not be solved using simulated data (if we could accurately simulate crypto markets then we would already be able to predict them!).
For transparency, we therefore decided to explicitly include the targets used to calculate public scores in the released data such that all competitors had equal access to them. Unfortunately, this does mean that it is possible to submit unrealistic scores to the public leaderboard and you may not be able to directly compare your score to other competitors.
\nFor this reason we suggest that during your experimentation you implement your own validation procedure using best practices to avoid data leakage. The submission mechanism should only be used to verify that your code is able to interact with the time-series API correctly and produces a score commensurate with your own offline validation.
\nAPI
\nAs a side note, the API is expected to deliver data in ~10 minutes, thus if using a trivial model (e.g. submitting dummy forecasts), this should be roughly the total submission time of your solution. Keep in mind, however, that your inference function will get called once per time step, thus any inefficiencies here will cause large increases in the total run time of your model. Also note that at evaluation time you will still have access to all the training data. If you combine this with a copy of all data provided to you from the API, this will allow you to produce predictions using time series models from the very first step.
Public Leaderboard Precision
\nLastly, we have increased the precision of the public leaderboard given the range of expected scores.
Happy Kaggling!
", "rawMarkdown": "Hi Kagglers!\n\n**Public Leaderboard**\nWe wanted to take a moment to clarify some concerns/questions surrounding the public leaderboard. As this competition makes use of publically available data (or at least data which can be reasonably approximated using public sources), there is unfortunately no way for us to ‘hold out’ data to provide a public leaderboard without the risk of data leakage. This would be true regardless of the evaluation metric and could not be solved using simulated data (if we could accurately simulate crypto markets then we would already be able to predict them!).\n \nFor transparency, we therefore decided to explicitly include the targets used to calculate public scores in the released data such that all competitors had equal access to them. Unfortunately, this does mean that it is possible to submit unrealistic scores to the public leaderboard and you may not be able to directly compare your score to other competitors.\n \nFor this reason we suggest that during your experimentation you implement your own validation procedure using best practices to avoid data leakage. The submission mechanism should only be used to verify that your code is able to interact with the time-series API correctly and produces a score commensurate with your own offline validation.\n\n**API** \nAs a side note, the API is expected to deliver data in ~10 minutes, thus if using a trivial model (e.g. submitting dummy forecasts), this should be roughly the total submission time of your solution. Keep in mind, however, that your inference function will get called once per time step, thus any inefficiencies here will cause large increases in the total run time of your model. Also note that at evaluation time you will still have access to all the training data. If you combine this with a copy of all data provided to you from the API, this will allow you to produce predictions using time series models from the very first step.\n\n**Public Leaderboard Precision**\nLastly, we have increased the precision of the public leaderboard given the range of expected scores.\n \nHappy Kaggling!", "votes": null }, { "id": "1594290", "postDate": "11/24/2021 17:10:50", "content": "@andrewscherer, On the evaluation, is a simple weighted mean of correlation or a weighted correlation?
\nBeing the former just a weighted mean using the correlation for each asset, and the last, the one that appears as a weighted correlation coeffient on Wikipedia.
\nDon't bother I found the answer, thanks @yamqwe.
\n", "rawMarkdown": "andrewscherer, On the evaluation, is a simple weighted mean of correlation or a weighted correlation? \n\nBeing the former just a weighted mean using the correlation for each asset, and the last, the one that appears as a weighted correlation coeffient on [Wikipedia](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient#Weighted_correlation_coefficient).\n\nDon't bother I found the answer, thanks @yamqwe.\n\n[Look here](https://www.kaggle.com/yamqwe/what-if-we-just-submit-the-target-median)", "votes": null }, { "id": "1594777", "postDate": "11/25/2021 05:57:37", "content": "Thank you for keeping addressing the issues directly, this is not an easy competition to be the host of..
", "rawMarkdown": "Thank you for keeping addressing the issues directly, this is not an easy competition to be the host of..", "votes": null }, { "id": "1603087", "postDate": "12/02/2021 10:26:44", "content": "Regarding the notebook runtime limits, Live test data after submission period is over will be around 3 months range (right?), the API will deliver one minute of data per iteration (right?) that's 90 days * 24 hrs * 60mins = 129600 iterations
\nwe have runtime limit of 9 hours (32400 seconds), if my understanding is correct, that means that my model should take at most 0.25 seconds between iterations (32400 seconds/ 129600 iterations)?
\nTo me, that doesn't sound like it's enough time, what am I missing?
", "rawMarkdown": "Regarding the notebook runtime limits, Live test data after submission period is over will be around 3 months range (right?), the API will deliver one minute of data per iteration (right?) that's 90 days * 24 hrs * 60mins = 129600 iterations\n\nwe have runtime limit of 9 hours (32400 seconds), if my understanding is correct, that means that my model should take at most 0.25 seconds between iterations (32400 seconds/ 129600 iterations)? \n\nTo me, that doesn't sound like it's enough time, what am I missing?", "votes": null }, { "id": "1604709", "postDate": "12/03/2021 16:20:26", "content": "Is this also the case for the current submission for leaderboard evaluation? My inference iteration take slightly longer than 0.25 seconds and it got Timeout after submitted. This makes constructing features on the fly really hard for me.
", "rawMarkdown": "Is this also the case for the current submission for leaderboard evaluation? My inference iteration take slightly longer than 0.25 seconds and it got Timeout after submitted. This makes constructing features on the fly really hard for me.", "votes": null }, { "id": "1614202", "postDate": "12/10/2021 18:38:49", "content": "Just to follow up with a further question raised by @shadymokh 's comment:
\nSaid another way, do we have to make consecutive API calls within 1 minute of run-time, or is this latency constraint not in place for this competition?
\n@shadymokh touches on the global run-time upper bound (implying an average API call upper bound of ~0.25s). My question is whether there will be a local upper bound too. i.e. the time between API calls must be less than 1 minute.
\nThis is relevant to the discussion around online learning and model retraining: in particular to, say once-per-week, retraining of models (possibly taking more than 1 minute of wall-time between API calls once per week of API minutes, and otherwise making calls much faster than once per second so as to meet the global run-time constraint).
", "rawMarkdown": "Just to follow up with a further question raised by @shadymokh 's [comment](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/290412#1603087):\n\n+ Do our models have to abide by the \"real-world\" latency constraint that predictions made after receiving each API call's data should be made within 1 minute of wall-time to be actionable?\n\nSaid another way, do we **have** to make consecutive API calls within 1 minute of run-time, or is this latency constraint not in place for this competition?\n\n@shadymokh touches on the **global** run-time upper bound (implying an average API call upper bound of ~0.25s). My question is whether there will be a **local** upper bound too. i.e. the time between API calls **must** be less than 1 minute.\n\nThis is relevant to the discussion around online learning and model retraining: in particular to, say once-per-week, retraining of models (possibly taking more than 1 minute of wall-time between API calls once per week of API minutes, and otherwise making calls much faster than once per second so as to meet the global run-time constraint).", "votes": null }, { "id": "1614208", "postDate": "12/10/2021 18:49:27", "content": "All API constraints are already documented; there is only a global runtime limit.
", "rawMarkdown": "All API constraints are already documented; there is only a global runtime limit.", "votes": null }, { "id": "1614215", "postDate": "12/10/2021 18:59:28", "content": "Great, thanks Sohier!
", "rawMarkdown": "Great, thanks Sohier!", "votes": null }, { "id": "1618998", "postDate": "12/15/2021 14:29:00", "content": "Question: In the training set, there will alway be at least 3750 timestamp preceding the timestamp of df_test?
\nThis is what I expect in order to use TSA models
This is the most important open question on the forums. It does seem that if you are standing on the final day of the 3m out-of-sample period, and you have to predict every minute in the 3months period at that time, then you are constrained to a wall time of 0.25 seconds per iteration. Is this right? If so, any submission using typical time series features will likely fail (time to concat the next minute and calculate features and make model pred is likely > 0.25 seconds). Also forget about re-training. This reduces the competition to mostly an engineering exercise (maybe that's fine for you, but it's boring for most Kagglers).
\nSince you are re-running the private leaderboard every two weeks, a solution to this problem is to limit the time to 9 hours per two-weeks of minute items. So for each two week period, you would provide train data (and realized target) up to the start of the period. Then the iteration would start from the beginning of the two week period. In this way each model re-run would only be for 14 days * 24hrs*60minutes = 20,160 minute segments.
\nAs is, every two weeks, you rerun the model from Feb 2, 2022 every time, right?
", "rawMarkdown": "andrewscherer \n\nThis is the most important open question on the forums. It does seem that if you are standing on the final day of the 3m out-of-sample period, and you have to predict every minute in the 3months period at that time, then you are constrained to a wall time of 0.25 seconds per iteration. Is this right? If so, any submission using typical time series features will likely fail (time to concat the next minute and calculate features and make model pred is likely > 0.25 seconds). Also forget about re-training. This reduces the competition to mostly an engineering exercise (maybe that's fine for you, but it's boring for most Kagglers). \n\nSince you are re-running the private leaderboard every two weeks, a solution to this problem is to limit the time to 9 hours per two-weeks of minute items. So for each two week period, you would provide train data (and realized target) up to the start of the period. Then the iteration would start from the beginning of the two week period. In this way each model re-run would only be for 14 days * 24hrs*60minutes = 20,160 minute segments. \n\nAs is, every two weeks, you rerun the model from Feb 2, 2022 every time, right?", "votes": null }, { "id": "1641877", "postDate": "01/07/2022 19:32:06", "content": "Would be nice to get official clarification, but here is some additional discussion on this here.
\nI think It's most likely you need to be able to process all 3 months within a single 9hr runtime. For the first LB update, the API will feed data that has accumulated through the first 2 weeks. For the second LB update the API will feed data that has accumulated through the first 4 weeks, … for the last LB update, the API will feed data that has accumulated through all 3 months.
\nTo take the other private LB approach (only 2 weeks of test data each LB update) would be too unnecessarily difficult and bug-prone to be sensible.
\nNot sure I agree on boring. The extra constraints are pretty cool imo, and open up a wealth of different strategies. Would be good to have some more precision, clarification, and interaction with hosts around the rules though - especially on the deviations between the de facto and the described Target calculation.
", "rawMarkdown": "Would be nice to get official clarification, but here is some additional discussion on this [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/297000#1637473). \n\nI think It's most likely you need to be able to process all 3 months within a single 9hr runtime. For the first LB update, the API will feed data that has accumulated through the first 2 weeks. For the second LB update the API will feed data that has accumulated through the first 4 weeks, ... for the last LB update, the API will feed data that has accumulated through all 3 months. \n\nTo take the other private LB approach (only 2 weeks of test data each LB update) would be too unnecessarily difficult and bug-prone to be sensible.\n\nNot sure I agree on boring. The extra constraints are pretty cool imo, and open up a wealth of different strategies. Would be good to have some more precision, clarification, and interaction with hosts around the rules though - especially on the deviations between the de facto and the described Target calculation.", "votes": null }, { "id": "1653827", "postDate": "01/17/2022 22:27:34", "content": "That’s right, the final score will be based on the run using the whole 3 months of future test data, with 9 hours runtime for the 3 months of data. The 2-week updates are simply partial LB updates to have an idea of how models are doing, and are not used in the final score.
", "rawMarkdown": "That’s right, the final score will be based on the run using the whole 3 months of future test data, with 9 hours runtime for the 3 months of data. The 2-week updates are simply partial LB updates to have an idea of how models are doing, and are not used in the final score.", "votes": null }, { "id": "1653860", "postDate": "01/17/2022 23:42:05", "content": "Great, thank you for the confirmation.
", "rawMarkdown": "Great, thank you for the confirmation.", "votes": null }, { "id": "1661571", "postDate": "01/23/2022 14:33:36", "content": "Question: Shouldn't the Live test be someting like: 90 days * 24 hrs * 60mins * 14 (they are 14 Asset_IDs) = 1,814,400 ?
", "rawMarkdown": "Question: Shouldn't the Live test be someting like: 90 days * 24 hrs * 60mins * 14 (they are 14 Asset_IDs) = 1,814,400 ?", "votes": null } ], "comments": [ { "id": 1594290, "author_name": "agenlu", "author_url": "", "post_date": "11/24/2021 17:10:50", "content": "@andrewscherer, On the evaluation, is a simple weighted mean of correlation or a weighted correlation?
\nBeing the former just a weighted mean using the correlation for each asset, and the last, the one that appears as a weighted correlation coeffient on Wikipedia.
\nDon't bother I found the answer, thanks @yamqwe.
\n", "votes": null, "replies": [] }, { "id": 1594777, "author_name": "", "author_url": "", "post_date": "11/25/2021 05:57:37", "content": "Thank you for keeping addressing the issues directly, this is not an easy competition to be the host of..
", "votes": null, "replies": [] }, { "id": 1603087, "author_name": "shadymokh", "author_url": "", "post_date": "12/02/2021 10:26:44", "content": "Regarding the notebook runtime limits, Live test data after submission period is over will be around 3 months range (right?), the API will deliver one minute of data per iteration (right?) that's 90 days * 24 hrs * 60mins = 129600 iterations
\nwe have runtime limit of 9 hours (32400 seconds), if my understanding is correct, that means that my model should take at most 0.25 seconds between iterations (32400 seconds/ 129600 iterations)?
\nTo me, that doesn't sound like it's enough time, what am I missing?
", "votes": null, "replies": [ { "id": 1604709, "author_name": "minshipeng1216", "author_url": "", "post_date": "12/03/2021 16:20:26", "content": "Is this also the case for the current submission for leaderboard evaluation? My inference iteration take slightly longer than 0.25 seconds and it got Timeout after submitted. This makes constructing features on the fly really hard for me.
", "votes": null, "replies": [] }, { "id": 1641615, "author_name": "marketneutral", "author_url": "", "post_date": "01/07/2022 15:22:31", "content": "\nThis is the most important open question on the forums. It does seem that if you are standing on the final day of the 3m out-of-sample period, and you have to predict every minute in the 3months period at that time, then you are constrained to a wall time of 0.25 seconds per iteration. Is this right? If so, any submission using typical time series features will likely fail (time to concat the next minute and calculate features and make model pred is likely > 0.25 seconds). Also forget about re-training. This reduces the competition to mostly an engineering exercise (maybe that's fine for you, but it's boring for most Kagglers).
\nSince you are re-running the private leaderboard every two weeks, a solution to this problem is to limit the time to 9 hours per two-weeks of minute items. So for each two week period, you would provide train data (and realized target) up to the start of the period. Then the iteration would start from the beginning of the two week period. In this way each model re-run would only be for 14 days * 24hrs*60minutes = 20,160 minute segments.
\nAs is, every two weeks, you rerun the model from Feb 2, 2022 every time, right?
", "votes": null, "replies": [] }, { "id": 1641877, "author_name": "jagofc", "author_url": "", "post_date": "01/07/2022 19:32:06", "content": "Would be nice to get official clarification, but here is some additional discussion on this here.
\nI think It's most likely you need to be able to process all 3 months within a single 9hr runtime. For the first LB update, the API will feed data that has accumulated through the first 2 weeks. For the second LB update the API will feed data that has accumulated through the first 4 weeks, … for the last LB update, the API will feed data that has accumulated through all 3 months.
\nTo take the other private LB approach (only 2 weeks of test data each LB update) would be too unnecessarily difficult and bug-prone to be sensible.
\nNot sure I agree on boring. The extra constraints are pretty cool imo, and open up a wealth of different strategies. Would be good to have some more precision, clarification, and interaction with hosts around the rules though - especially on the deviations between the de facto and the described Target calculation.
", "votes": null, "replies": [] }, { "id": 1653827, "author_name": "cstein06", "author_url": "", "post_date": "01/17/2022 22:27:34", "content": "That’s right, the final score will be based on the run using the whole 3 months of future test data, with 9 hours runtime for the 3 months of data. The 2-week updates are simply partial LB updates to have an idea of how models are doing, and are not used in the final score.
", "votes": null, "replies": [] }, { "id": 1653860, "author_name": "jagofc", "author_url": "", "post_date": "01/17/2022 23:42:05", "content": "Great, thank you for the confirmation.
", "votes": null, "replies": [] }, { "id": 1661571, "author_name": "roburadu", "author_url": "", "post_date": "01/23/2022 14:33:36", "content": "Question: Shouldn't the Live test be someting like: 90 days * 24 hrs * 60mins * 14 (they are 14 Asset_IDs) = 1,814,400 ?
", "votes": null, "replies": [] } ] }, { "id": 1614202, "author_name": "jagofc", "author_url": "", "post_date": "12/10/2021 18:38:49", "content": "Just to follow up with a further question raised by @shadymokh 's comment:
\nSaid another way, do we have to make consecutive API calls within 1 minute of run-time, or is this latency constraint not in place for this competition?
\n@shadymokh touches on the global run-time upper bound (implying an average API call upper bound of ~0.25s). My question is whether there will be a local upper bound too. i.e. the time between API calls must be less than 1 minute.
\nThis is relevant to the discussion around online learning and model retraining: in particular to, say once-per-week, retraining of models (possibly taking more than 1 minute of wall-time between API calls once per week of API minutes, and otherwise making calls much faster than once per second so as to meet the global run-time constraint).
", "votes": null, "replies": [ { "id": 1614208, "author_name": "sohier", "author_url": "", "post_date": "12/10/2021 18:49:27", "content": "All API constraints are already documented; there is only a global runtime limit.
", "votes": null, "replies": [] }, { "id": 1614215, "author_name": "jagofc", "author_url": "", "post_date": "12/10/2021 18:59:28", "content": "Great, thanks Sohier!
", "votes": null, "replies": [] } ] }, { "id": 1618998, "author_name": "elonore", "author_url": "", "post_date": "12/15/2021 14:29:00", "content": "Question: In the training set, there will alway be at least 3750 timestamp preceding the timestamp of df_test?
\nThis is what I expect in order to use TSA models