The competition was interesting. But there is still a sense of untapped opportunities.
\nI'm sure everyone would like to test their best decisions and not only the 2 selected. For this purpose, it is necessary to download similar test data via API, count on them the target, and compare with prediction. So, the whole question comes down to where to find minute data on the same currencies for the period February - April 2022. What are the ideas?
\nIn fact, there are problems with this. And the organizers are well done for eliminating them. But how to repeat it?
\nWhat are your ideas and comments?
", "messages": [ { "id": "1686706", "postDate": "02/12/2022 09:50:16", "content": "The competition was interesting. But there is still a sense of untapped opportunities.
\nI'm sure everyone would like to test their best decisions and not only the 2 selected. For this purpose, it is necessary to download similar test data via API, count on them the target, and compare with prediction. So, the whole question comes down to where to find minute data on the same currencies for the period February - April 2022. What are the ideas?
\nIn fact, there are problems with this. And the organizers are well done for eliminating them. But how to repeat it?
\nWhat are your ideas and comments?
", "rawMarkdown": "The competition was interesting. But there is still a sense of untapped opportunities.\n1. Only 2 decisions could be submitted for test verification.\n2. The organizers announced that the training data will be updated again, so those who have decided to use pre-trained models will lose. Markets are in a bit of a fever right now. They exhibit abnormal behavior.\n3. On the last day of the competition, many decisions were rejected (even with 30-second execution on pre-trained models) due to server overload.\n4. Training data is problematic - they have hundreds of missed data, even in the target.\n\nI'm sure everyone would like to test their best decisions and not only the 2 selected. For this purpose, it is necessary to **download similar test data via API, count on them the target, and compare with prediction**. So, the whole question comes down to where to find minute data on the same currencies for the period February - April 2022. What are the ideas?\n\nIn fact, there are problems with this. And the organizers are well done for eliminating them. But how to repeat it?\n1. Most resources provide data on a daily basis, not every minute.\n2. Some provide minute data, but prohibit downloading and posting somewhere. If you take the data via API, then they will first be downloaded to the intermediate file as data to the notebook, and then you will take that data from there. Will this be a violation of the license terms?\n3. I did not find similar datasets in Kaggle with actual modern data.\n\nWhat are your ideas and comments?", "votes": null }, { "id": "1686729", "postDate": "02/12/2022 10:00:15", "content": "I had set up two notebooks to dowload and build datasets: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance and https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance I think it can give you an idea of the path to go. It relies on the code used for: https://www.kaggle.com/jorijnsmit/binance-full-history.
\nHowever it wasn't an exact match: it wasn't clear what base currency was an exact match (BUSD / USDT ?) and the VWAP was missing. I've heard bitfinex offer VWAP but haven't got the time to try.
So. I also did not find open sources that contain VWAP. But finding at least OHLC-features would also be great!
", "rawMarkdown": "So. I also did not find open sources that contain VWAP. But finding at least OHLC-features would also be great!", "votes": null }, { "id": "1688417", "postDate": "02/13/2022 16:01:12", "content": "I did a quick comparison against binance vision for 1 minute data here.
\nI think one could get pretty close to vwap by calculating from the trade data but it's probably not worth the effort since the data is from multiple exchanges.
", "rawMarkdown": "I did a quick comparison against binance vision for 1 minute data [here](https://www.kaggle.com/devcon/g-research-crypto-eda-vs-binance).\n\nI think one could get pretty close to vwap by calculating from the trade data but it's probably not worth the effort since the data is from multiple exchanges.", "votes": null }, { "id": "1688958", "postDate": "02/13/2022 23:06:59", "content": "You can also just wait for the leaderboard to be updated instead of testing yourself? It makes no difference now anyway..
\nI also don't really get what you are asking? how to get minute data for crypto? its pretty easy to open a websocket and store the data yourself from the exchanges, aggregate it to minute and calculate the VWAP etc.. If you can't be bothered for that or don't have the infrastructure skills for it then there is also lots of paid APIs that will provide the data for you..
", "rawMarkdown": "You can also just wait for the leaderboard to be updated instead of testing yourself? It makes no difference now anyway..\n\nI also don't really get what you are asking? how to get minute data for crypto? its pretty easy to open a websocket and store the data yourself from the exchanges, aggregate it to minute and calculate the VWAP etc.. If you can't be bothered for that or don't have the infrastructure skills for it then there is also lots of paid APIs that will provide the data for you..", "votes": null } ], "comments": [ { "id": 1686729, "author_name": "lucasmorin", "author_url": "", "post_date": "02/12/2022 10:00:15", "content": "I had set up two notebooks to dowload and build datasets: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance and https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance I think it can give you an idea of the path to go. It relies on the code used for: https://www.kaggle.com/jorijnsmit/binance-full-history.
\nHowever it wasn't an exact match: it wasn't clear what base currency was an exact match (BUSD / USDT ?) and the VWAP was missing. I've heard bitfinex offer VWAP but haven't got the time to try.
So. I also did not find open sources that contain VWAP. But finding at least OHLC-features would also be great!
", "votes": null, "replies": [] } ] }, { "id": 1688417, "author_name": "devcon", "author_url": "", "post_date": "02/13/2022 16:01:12", "content": "I did a quick comparison against binance vision for 1 minute data here.
\nI think one could get pretty close to vwap by calculating from the trade data but it's probably not worth the effort since the data is from multiple exchanges.
", "votes": null, "replies": [] }, { "id": 1688958, "author_name": "julianmukaj", "author_url": "", "post_date": "02/13/2022 23:06:59", "content": "You can also just wait for the leaderboard to be updated instead of testing yourself? It makes no difference now anyway..
\nI also don't really get what you are asking? how to get minute data for crypto? its pretty easy to open a websocket and store the data yourself from the exchanges, aggregate it to minute and calculate the VWAP etc.. If you can't be bothered for that or don't have the infrastructure skills for it then there is also lots of paid APIs that will provide the data for you..
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1686706": "The competition was interesting. But there is still a sense of untapped opportunities.\n1. Only 2 decisions could be submitted for test verification.\n2. The organizers announced that the training data will be updated again, so those who have decided to use pre-trained models will lose. Markets are in a bit of a fever right now. They exhibit abnormal behavior.\n3. On the last day of the competition, many decisions were rejected (even with 30-second execution on pre-trained models) due to server overload.\n4. Training data is problematic - they have hundreds of missed data, even in the target.\n\nI'm sure everyone would like to test their best decisions and not only the 2 selected. For this purpose, it is necessary to **download similar test data via API, count on them the target, and compare with prediction**. So, the whole question comes down to where to find minute data on the same currencies for the period February - April 2022. What are the ideas?\n\nIn fact, there are problems with this. And the organizers are well done for eliminating them. But how to repeat it?\n1. Most resources provide data on a daily basis, not every minute.\n2. Some provide minute data, but prohibit downloading and posting somewhere. If you take the data via API, then they will first be downloaded to the intermediate file as data to the notebook, and then you will take that data from there. Will this be a violation of the license terms?\n3. I did not find similar datasets in Kaggle with actual modern data.\n\nWhat are your ideas and comments?", "1686729": "I had set up two notebooks to dowload and build datasets: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance and https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance I think it can give you an idea of the path to go. It relies on the code used for: https://www.kaggle.com/jorijnsmit/binance-full-history.\nHowever it wasn't an exact match: it wasn't clear what base currency was an exact match (BUSD / USDT ?) and the VWAP was missing. I've heard bitfinex offer VWAP but haven't got the time to try.", "1686740": "So. I also did not find open sources that contain VWAP. But finding at least OHLC-features would also be great!", "1688417": "I did a quick comparison against binance vision for 1 minute data [here](https://www.kaggle.com/devcon/g-research-crypto-eda-vs-binance).\n\nI think one could get pretty close to vwap by calculating from the trade data but it's probably not worth the effort since the data is from multiple exchanges.", "1688958": "You can also just wait for the leaderboard to be updated instead of testing yourself? It makes no difference now anyway..\n\nI also don't really get what you are asking? how to get minute data for crypto? its pretty easy to open a websocket and store the data yourself from the exchanges, aggregate it to minute and calculate the VWAP etc.. If you can't be bothered for that or don't have the infrastructure skills for it then there is also lots of paid APIs that will provide the data for you.." }, "source": "meta" }