{ "id": 284942, "title": "Are we allowed to collect market data during these three months to extend the train set?", "url": "/competitions/g-research-crypto-forecasting/discussion/284942", "author_name": "", "post_date": "2021-11-03T00:09:54.572318200Z", "votes": 18, "comment_count": 19, "views": 0, "content": "

The train set is historical. So, are we allowed to collect market data during these three months to extend the train set?

", "messages": [ { "id": "1568649", "postDate": "11/03/2021 00:09:54", "content": "

The train set is historical. So, are we allowed to collect market data during these three months to extend the train set?

", "rawMarkdown": "The train set is historical. So, are we allowed to collect market data during these three months to extend the train set?", "votes": null }, { "id": "1568661", "postDate": "11/03/2021 00:28:30", "content": "

Please see Code Requirements section, \"Freely & publicly available external data is allowed, including pre-trained models\"

", "rawMarkdown": "Please see Code Requirements section, \"Freely & publicly available external data is allowed, including pre-trained models\"", "votes": null }, { "id": "1568672", "postDate": "11/03/2021 00:51:34", "content": "

\" Accordingly, THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code\"

\n

I think this competition is special because it also involves collecting your own data.
\nit is like working a full end-to-end pipeline string from data collection … doing the real job of data scientist

\n

it may also include the collection of aux data like tweets or news (this affects market sentiments)?
\n\"https://www.blockchainresearchlab.org/wp-content/uploads/2021/02/musk2_en.png\"

", "rawMarkdown": "\" Accordingly, THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code\"\n\nI think this competition is special because it also involves collecting your own data.\nit is like working a full end-to-end pipeline string from data collection ... doing the real job of data scientist\n\nit may also include the collection of aux data like tweets or news (this affects market sentiments)?\n![https://www.blockchainresearchlab.org/wp-content/uploads/2021/02/musk2_en.png](https://www.blockchainresearchlab.org/wp-content/uploads/2021/02/musk2_en.png)", "votes": null }, { "id": "1568680", "postDate": "11/03/2021 01:06:47", "content": "

But the internet is disabled during the forecasting period. How do we get real-time tweet 'features' then?

", "rawMarkdown": "But the internet is disabled during the forecasting period. How do we get real-time tweet 'features' then?", "votes": null }, { "id": "1568844", "postDate": "11/03/2021 04:32:56", "content": "

From the Data tab:

\n
\n

supplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.

\n
\n

This seems to indicate that the supplemental train file will be populated with market data from these 3 months, but only after the first stage ends. Perhaps we're meant to pretrain models offline using the currently available data and finetune them \"blindly\" in our submission notebooks on the supplemental training set.

", "rawMarkdown": "From the Data tab:\n\n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n\nThis seems to indicate that the supplemental train file will be populated with market data from these 3 months, but only after the first stage ends. Perhaps we're meant to pretrain models offline using the currently available data and finetune them \"blindly\" in our submission notebooks on the supplemental training set.", "votes": null }, { "id": "1568897", "postDate": "11/03/2021 05:56:39", "content": "

If this is possible the competition will become an arms race of \"who has the most data\" (and the biggest infrastructure) rather than who has the best predictive methods…

\n

If I were among the hosts, I would seriously consider not allowing this since it might make the competition lose meaning..
\n@maggiemd

", "rawMarkdown": "If this is possible the competition will become an arms race of \"who has the most data\" (**and the biggest infrastructure**) rather than who has the best predictive methods...\n\nIf I were among the hosts, I would seriously consider not allowing this since it might make the competition lose meaning..\n@maggiemd", "votes": null }, { "id": "1569012", "postDate": "11/03/2021 07:27:04", "content": "

You collect data on your own, build datasets, train models using them, build other datasets, etc. then use all your (additional resources) to prepare your submission.

", "rawMarkdown": "You collect data on your own, build datasets, train models using them, build other datasets, etc. then use all your (additional resources) to prepare your submission.", "votes": null }, { "id": "1569025", "postDate": "11/03/2021 07:46:09", "content": "

Publicly available when? If I start collecting market data today and upload it to kaggle pub dataset right before the submission deadline, would that be considered \"freely & publicly available\"?

", "rawMarkdown": "Publicly available when? If I start collecting market data today and upload it to kaggle pub dataset right before the submission deadline, would that be considered \"freely & publicly available\"?", "votes": null }, { "id": "1569031", "postDate": "11/03/2021 07:48:23", "content": "

This is confusing. What is the point of that file if it is only updated after the submission deadline?

", "rawMarkdown": "This is confusing. What is the point of that file if it is only updated after the submission deadline?", "votes": null }, { "id": "1569239", "postDate": "11/03/2021 10:25:26", "content": "

Well if you have market data from December, you probably shouldn't be using it for this competition.. 😏

", "rawMarkdown": "Well if you have market data from December, you probably shouldn't be using it for this competition.. 😏", "votes": null }, { "id": "1569530", "postDate": "11/03/2021 14:17:36", "content": "

@gogo827jz Seems like it means you can use external macro data to build your model then use forecasts for the predicting phase. Use case would be to exclude some data from the covid crisis… doubt it will really go further than this.

", "rawMarkdown": "gogo827jz Seems like it means you can use external macro data to build your model then use forecasts for the predicting phase. Use case would be to exclude some data from the covid crisis... doubt it will really go further than this.", "votes": null }, { "id": "1569534", "postDate": "11/03/2021 14:20:44", "content": "

Kaggle is already an arm race for infra. Given that we can't use live external data in the forecasting period, I doubt it will matter that much overall.

", "rawMarkdown": "Kaggle is already an arm race for infra. Given that we can't use live external data in the forecasting period, I doubt it will matter that much overall.", "votes": null }, { "id": "1569641", "postDate": "11/03/2021 16:03:11", "content": "
\n

What is the point

\n
\n

Re-training/finetuning, I guess. Probably submitted notebook will run max 9 hours every x days.
\nBut that might be written somewhere.

", "rawMarkdown": "> What is the point\n\nRe-training/finetuning, I guess. Probably submitted notebook will run max 9 hours every x days.\nBut that might be written somewhere.", "votes": null }, { "id": "1569673", "postDate": "11/03/2021 16:37:07", "content": "

My understanding is that no data other than price and volume will be available during private run. So, probably there is no advantage in using external data during training period (except for external data for price and volume before 2018 and current gap up to Feb 1).

", "rawMarkdown": "My understanding is that no data other than price and volume will be available during private run. So, probably there is no advantage in using external data during training period (except for external data for price and volume before 2018 and current gap up to Feb 1).", "votes": null }, { "id": "1569677", "postDate": "11/03/2021 16:40:19", "content": "

And by publicly available, you mean ANYWHERE, not just on Kaggle?

", "rawMarkdown": "And by publicly available, you mean ANYWHERE, not just on Kaggle?", "votes": null }, { "id": "1569759", "postDate": "11/03/2021 17:31:42", "content": "

I doubt that any specific extra data can help in this forecasting competition. While I am usually against external data, it totally makes sense here, as finding any signal somewhere would be valuable. Also, to not have the latest-up-to-date version of the data itself does not make much sense.

", "rawMarkdown": "I doubt that any specific extra data can help in this forecasting competition. While I am usually against external data, it totally makes sense here, as finding any signal somewhere would be valuable. Also, to not have the latest-up-to-date version of the data itself does not make much sense.", "votes": null }, { "id": "1569871", "postDate": "11/03/2021 19:28:05", "content": "

I just tought about it again. Getting the same data but on a longer scale / finer scale could help wildly to have a better training data set…

", "rawMarkdown": "I just tought about it again. Getting the same data but on a longer scale / finer scale could help wildly to have a better training data set...", "votes": null }, { "id": "1571332", "postDate": "11/04/2021 18:57:06", "content": "

But how can we use external information if internet access is disabled? we can't use any api to fetch data

", "rawMarkdown": "But how can we use external information if internet access is disabled? we can't use any api to fetch data", "votes": null }, { "id": "1572472", "postDate": "11/05/2021 17:39:58", "content": "

I believe it's meant that you can download data at home e.g. most recent few months, then upload to kaggle and import data, to use for your own training. (if that makes sense?!)

", "rawMarkdown": "I believe it's meant that you can download data at home e.g. most recent few months, then upload to kaggle and import data, to use for your own training. (if that makes sense?!)", "votes": null }, { "id": "1573121", "postDate": "11/06/2021 10:03:20", "content": "

Hi, I just implemented your Idea: created an auto-updating dataset that will refresh daily on Kaggle for everyone to use.
\nI would greatly appreciate it if you could give me some feedback about it.

\n

It wasn't easy pulling this whole thing off in just two days 😆

", "rawMarkdown": "Hi, I just implemented your Idea: created an [auto-updating dataset](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726) that will refresh daily on Kaggle for everyone to use. \nI would greatly appreciate it if you could give me some feedback about it. \n\nIt wasn't easy pulling this whole thing off in just two days 😆", "votes": null } ], "comments": [ { "id": 1568661, "author_name": "maggiemd", "author_url": "", "post_date": "11/03/2021 00:28:30", "content": "

Please see Code Requirements section, \"Freely & publicly available external data is allowed, including pre-trained models\"

", "votes": null, "replies": [ { "id": 1569025, "author_name": "wenrui29", "author_url": "", "post_date": "11/03/2021 07:46:09", "content": "

Publicly available when? If I start collecting market data today and upload it to kaggle pub dataset right before the submission deadline, would that be considered \"freely & publicly available\"?

", "votes": null, "replies": [] }, { "id": 1569239, "author_name": "julianmukaj", "author_url": "", "post_date": "11/03/2021 10:25:26", "content": "

Well if you have market data from December, you probably shouldn't be using it for this competition.. 😏

", "votes": null, "replies": [] }, { "id": 1569677, "author_name": "ahfretheim", "author_url": "", "post_date": "11/03/2021 16:40:19", "content": "

And by publicly available, you mean ANYWHERE, not just on Kaggle?

", "votes": null, "replies": [] }, { "id": 1571332, "author_name": "ebudia", "author_url": "", "post_date": "11/04/2021 18:57:06", "content": "

But how can we use external information if internet access is disabled? we can't use any api to fetch data

", "votes": null, "replies": [] }, { "id": 1572472, "author_name": "regipublic", "author_url": "", "post_date": "11/05/2021 17:39:58", "content": "

I believe it's meant that you can download data at home e.g. most recent few months, then upload to kaggle and import data, to use for your own training. (if that makes sense?!)

", "votes": null, "replies": [] } ] }, { "id": 1568672, "author_name": "hengck23", "author_url": "", "post_date": "11/03/2021 00:51:34", "content": "

\" Accordingly, THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code\"

\n

I think this competition is special because it also involves collecting your own data.
\nit is like working a full end-to-end pipeline string from data collection … doing the real job of data scientist

\n

it may also include the collection of aux data like tweets or news (this affects market sentiments)?
\n\"https://www.blockchainresearchlab.org/wp-content/uploads/2021/02/musk2_en.png\"

", "votes": null, "replies": [ { "id": 1568680, "author_name": "gogo827jz", "author_url": "", "post_date": "11/03/2021 01:06:47", "content": "

But the internet is disabled during the forecasting period. How do we get real-time tweet 'features' then?

", "votes": null, "replies": [] }, { "id": 1569012, "author_name": "gpreda", "author_url": "", "post_date": "11/03/2021 07:27:04", "content": "

You collect data on your own, build datasets, train models using them, build other datasets, etc. then use all your (additional resources) to prepare your submission.

", "votes": null, "replies": [] }, { "id": 1569530, "author_name": "lucasmorin", "author_url": "", "post_date": "11/03/2021 14:17:36", "content": "

@gogo827jz Seems like it means you can use external macro data to build your model then use forecasts for the predicting phase. Use case would be to exclude some data from the covid crisis… doubt it will really go further than this.

", "votes": null, "replies": [] }, { "id": 1569673, "author_name": "isakev", "author_url": "", "post_date": "11/03/2021 16:37:07", "content": "

My understanding is that no data other than price and volume will be available during private run. So, probably there is no advantage in using external data during training period (except for external data for price and volume before 2018 and current gap up to Feb 1).

", "votes": null, "replies": [] } ] }, { "id": 1568844, "author_name": "mingpan07", "author_url": "", "post_date": "11/03/2021 04:32:56", "content": "

From the Data tab:

\n
\n

supplemental_train.csv - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.

\n
\n

This seems to indicate that the supplemental train file will be populated with market data from these 3 months, but only after the first stage ends. Perhaps we're meant to pretrain models offline using the currently available data and finetune them \"blindly\" in our submission notebooks on the supplemental training set.

", "votes": null, "replies": [ { "id": 1569031, "author_name": "wenrui29", "author_url": "", "post_date": "11/03/2021 07:48:23", "content": "

This is confusing. What is the point of that file if it is only updated after the submission deadline?

", "votes": null, "replies": [] }, { "id": 1569641, "author_name": "cepheidq", "author_url": "", "post_date": "11/03/2021 16:03:11", "content": "
\n

What is the point

\n
\n

Re-training/finetuning, I guess. Probably submitted notebook will run max 9 hours every x days.
\nBut that might be written somewhere.

", "votes": null, "replies": [] } ] }, { "id": 1568897, "author_name": "", "author_url": "", "post_date": "11/03/2021 05:56:39", "content": "

If this is possible the competition will become an arms race of \"who has the most data\" (and the biggest infrastructure) rather than who has the best predictive methods…

\n

If I were among the hosts, I would seriously consider not allowing this since it might make the competition lose meaning..
\n@maggiemd

", "votes": null, "replies": [ { "id": 1569534, "author_name": "lucasmorin", "author_url": "", "post_date": "11/03/2021 14:20:44", "content": "

Kaggle is already an arm race for infra. Given that we can't use live external data in the forecasting period, I doubt it will matter that much overall.

", "votes": null, "replies": [] }, { "id": 1569759, "author_name": "philippsinger", "author_url": "", "post_date": "11/03/2021 17:31:42", "content": "

I doubt that any specific extra data can help in this forecasting competition. While I am usually against external data, it totally makes sense here, as finding any signal somewhere would be valuable. Also, to not have the latest-up-to-date version of the data itself does not make much sense.

", "votes": null, "replies": [] }, { "id": 1569871, "author_name": "lucasmorin", "author_url": "", "post_date": "11/03/2021 19:28:05", "content": "

I just tought about it again. Getting the same data but on a longer scale / finer scale could help wildly to have a better training data set…

", "votes": null, "replies": [] } ] }, { "id": 1573121, "author_name": "", "author_url": "", "post_date": "11/06/2021 10:03:20", "content": "

Hi, I just implemented your Idea: created an auto-updating dataset that will refresh daily on Kaggle for everyone to use.
\nI would greatly appreciate it if you could give me some feedback about it.

\n

It wasn't easy pulling this whole thing off in just two days 😆

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1568649": "The train set is historical. So, are we allowed to collect market data during these three months to extend the train set?", "1568661": "Please see Code Requirements section, \"Freely & publicly available external data is allowed, including pre-trained models\"", "1568672": "\" Accordingly, THE PUBLIC LEADERBOARD FOR THIS COMPETITION IS NOT MEANINGFUL and is only provided as a convenience for anyone who wants to test their code\"\n\nI think this competition is special because it also involves collecting your own data.\nit is like working a full end-to-end pipeline string from data collection ... doing the real job of data scientist\n\nit may also include the collection of aux data like tweets or news (this affects market sentiments)?\n![https://www.blockchainresearchlab.org/wp-content/uploads/2021/02/musk2_en.png](https://www.blockchainresearchlab.org/wp-content/uploads/2021/02/musk2_en.png)", "1568680": "But the internet is disabled during the forecasting period. How do we get real-time tweet 'features' then?", "1568844": "From the Data tab:\n\n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n\nThis seems to indicate that the supplemental train file will be populated with market data from these 3 months, but only after the first stage ends. Perhaps we're meant to pretrain models offline using the currently available data and finetune them \"blindly\" in our submission notebooks on the supplemental training set.", "1568897": "If this is possible the competition will become an arms race of \"who has the most data\" (**and the biggest infrastructure**) rather than who has the best predictive methods...\n\nIf I were among the hosts, I would seriously consider not allowing this since it might make the competition lose meaning..\n@maggiemd", "1569012": "You collect data on your own, build datasets, train models using them, build other datasets, etc. then use all your (additional resources) to prepare your submission.", "1569025": "Publicly available when? If I start collecting market data today and upload it to kaggle pub dataset right before the submission deadline, would that be considered \"freely & publicly available\"?", "1569031": "This is confusing. What is the point of that file if it is only updated after the submission deadline?", "1569239": "Well if you have market data from December, you probably shouldn't be using it for this competition.. 😏", "1569530": "gogo827jz Seems like it means you can use external macro data to build your model then use forecasts for the predicting phase. Use case would be to exclude some data from the covid crisis... doubt it will really go further than this.", "1569534": "Kaggle is already an arm race for infra. Given that we can't use live external data in the forecasting period, I doubt it will matter that much overall.", "1569641": "> What is the point\n\nRe-training/finetuning, I guess. Probably submitted notebook will run max 9 hours every x days.\nBut that might be written somewhere.", "1569673": "My understanding is that no data other than price and volume will be available during private run. So, probably there is no advantage in using external data during training period (except for external data for price and volume before 2018 and current gap up to Feb 1).", "1569677": "And by publicly available, you mean ANYWHERE, not just on Kaggle?", "1569759": "I doubt that any specific extra data can help in this forecasting competition. While I am usually against external data, it totally makes sense here, as finding any signal somewhere would be valuable. Also, to not have the latest-up-to-date version of the data itself does not make much sense.", "1569871": "I just tought about it again. Getting the same data but on a longer scale / finer scale could help wildly to have a better training data set...", "1571332": "But how can we use external information if internet access is disabled? we can't use any api to fetch data", "1572472": "I believe it's meant that you can download data at home e.g. most recent few months, then upload to kaggle and import data, to use for your own training. (if that makes sense?!)", "1573121": "Hi, I just implemented your Idea: created an [auto-updating dataset](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726) that will refresh daily on Kaggle for everyone to use. \nI would greatly appreciate it if you could give me some feedback about it. \n\nIt wasn't easy pulling this whole thing off in just two days 😆" }, "source": "meta" }