{
  "id": 285913,
  "title": "Getting and Aggregating 1m Binance Data",
  "url": "/competitions/g-research-crypto-forecasting/discussion/285913",
  "author_name": "",
  "post_date": "2021-11-06T22:38:29.875478600Z",
  "votes": 30,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I finally managed to get some good external data &amp; spent some time handling the preprocessing.</p>\n<p>I share two notebooks: </p>\n<ul>\n<li>Getting all 1m data from Binance: <a href=\"https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance\" target=\"_blank\">https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance</a> :</li>\n</ul>\n<p>Using some code from <a href=\"https://www.kaggle.com/jorijnsmit\" target=\"_blank\">@jorijnsmit</a> available here: <a href=\"https://github.com/gosuto-ai/candlestick_retriever/blob/master/main.py\" target=\"_blank\">https://github.com/gosuto-ai/candlestick_retriever/blob/master/main.py</a> I manage to get all relevant crypto for this competition at 1m intervall. The main assumption I make is using Binance USD (BUSD) instead of USDT as base conversion. I get data from 09/19 to today. I'll try to get more.</p>\n<p>The raw output is available in this dataset: <a href=\"https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance\" target=\"_blank\">https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance</a></p>\n<ul>\n<li>Transforming 1m data From Binance: <a href=\"https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data\" target=\"_blank\">https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data</a></li>\n</ul>\n<p>Then I process the raw data from Binance. I pack the data in one dataset and ensure the content and the format is the same as the training data set. There seems to be some remaining work:</p>\n<ul>\n<li>VWAP is approximated from volumes and High / Low. I'll try to see If I can pull this data out of Binance</li>\n<li>Target: I use the code from: <a href=\"https://www.kaggle.com/alexfir/recreating-target\" target=\"_blank\">https://www.kaggle.com/alexfir/recreating-target</a> to build the target. It doesn't match exactly the existing data. There is an ongoing discussion about this.</li>\n<li>Filtering rows: there seems to be some rows with 0 volume or other problems (infinite / nan VWAP/return). I haven't cleaned them yet as they might be the key to explain target building errors. </li>\n</ul>\n<p>The preprocessed output is available in this dataset: <a href=\"https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance\" target=\"_blank\">https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance</a></p>",
  "messages": [
    {
      "id": "1573783",
      "postDate": "11/06/2021 22:38:29",
      "content": "<p>I finally managed to get some good external data &amp; spent some time handling the preprocessing.</p>\n<p>I share two notebooks: </p>\n<ul>\n<li>Getting all 1m data from Binance: <a href=\"https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance\" target=\"_blank\">https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance</a> :</li>\n</ul>\n<p>Using some code from <a href=\"https://www.kaggle.com/jorijnsmit\" target=\"_blank\">@jorijnsmit</a> available here: <a href=\"https://github.com/gosuto-ai/candlestick_retriever/blob/master/main.py\" target=\"_blank\">https://github.com/gosuto-ai/candlestick_retriever/blob/master/main.py</a> I manage to get all relevant crypto for this competition at 1m intervall. The main assumption I make is using Binance USD (BUSD) instead of USDT as base conversion. I get data from 09/19 to today. I'll try to get more.</p>\n<p>The raw output is available in this dataset: <a href=\"https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance\" target=\"_blank\">https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance</a></p>\n<ul>\n<li>Transforming 1m data From Binance: <a href=\"https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data\" target=\"_blank\">https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data</a></li>\n</ul>\n<p>Then I process the raw data from Binance. I pack the data in one dataset and ensure the content and the format is the same as the training data set. There seems to be some remaining work:</p>\n<ul>\n<li>VWAP is approximated from volumes and High / Low. I'll try to see If I can pull this data out of Binance</li>\n<li>Target: I use the code from: <a href=\"https://www.kaggle.com/alexfir/recreating-target\" target=\"_blank\">https://www.kaggle.com/alexfir/recreating-target</a> to build the target. It doesn't match exactly the existing data. There is an ongoing discussion about this.</li>\n<li>Filtering rows: there seems to be some rows with 0 volume or other problems (infinite / nan VWAP/return). I haven't cleaned them yet as they might be the key to explain target building errors. </li>\n</ul>\n<p>The preprocessed output is available in this dataset: <a href=\"https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance\" target=\"_blank\">https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance</a></p>",
      "rawMarkdown": "I finally managed to get some good external data & spent some time handling the preprocessing.\n\nI share two notebooks: \n\n  - Getting all 1m data from Binance: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance :\n\nUsing some code from @jorijnsmit available here: https://github.com/gosuto-ai/candlestick_retriever/blob/master/main.py I manage to get all relevant crypto for this competition at 1m intervall. The main assumption I make is using Binance USD (BUSD) instead of USDT as base conversion. I get data from 09/19 to today. I'll try to get more.\n\nThe raw output is available in this dataset: https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance\n\n - Transforming 1m data From Binance: https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data\n\nThen I process the raw data from Binance. I pack the data in one dataset and ensure the content and the format is the same as the training data set. There seems to be some remaining work:\n- VWAP is approximated from volumes and High / Low. I'll try to see If I can pull this data out of Binance\n- Target: I use the code from: https://www.kaggle.com/alexfir/recreating-target to build the target. It doesn't match exactly the existing data. There is an ongoing discussion about this.\n- Filtering rows: there seems to be some rows with 0 volume or other problems (infinite / nan VWAP/return). I haven't cleaned them yet as they might be the key to explain target building errors. \n\nThe preprocessed output is available in this dataset: https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance",
      "votes": null
    },
    {
      "id": "1576155",
      "postDate": "11/09/2021 00:19:37",
      "content": "<p>Great thanks was about to start on this and was looking to source the same . This will give a head start 👍</p>",
      "rawMarkdown": "Great thanks was about to start on this and was looking to source the same . This will give a head start 👍",
      "votes": null
    },
    {
      "id": "1576730",
      "postDate": "11/09/2021 12:24:25",
      "content": "<p>Just what I thought of doing <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a>. I just made a <a href=\"https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286491\" target=\"_blank\">discussion topic</a> to help me understand what VWAP is before doing what you're making. If I can help you with something just tell me. </p>",
      "rawMarkdown": "Just what I thought of doing @lucasmorin. I just made a [discussion topic](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286491) to help me understand what VWAP is before doing what you're making. If I can help you with something just tell me.",
      "votes": null
    },
    {
      "id": "1576880",
      "postDate": "11/09/2021 14:27:12",
      "content": "<p>Great resource, thanks for sharing!</p>\n<p>Side note: the first link is broken <a href=\"https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:\" target=\"_blank\">https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:</a>, it has the colon <br>\n(\":\") in it and gets a 404.</p>",
      "rawMarkdown": "Great resource, thanks for sharing!\n\nSide note: the first link is broken https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:, it has the colon \n(\":\") in it and gets a 404.",
      "votes": null
    },
    {
      "id": "1579042",
      "postDate": "11/11/2021 13:33:18",
      "content": "<p>Thanks for the mention <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a>. I updated the original dataset just now; <a href=\"https://www.kaggle.com/jorijnsmit/binance-full-history\" target=\"_blank\">https://www.kaggle.com/jorijnsmit/binance-full-history</a></p>\n<p>25GB of parquet files, containing 1m historical candles for the top 1000 markets on Binance, going back to 2017 :-)</p>",
      "rawMarkdown": "Thanks for the mention @lucasmorin. I updated the original dataset just now; https://www.kaggle.com/jorijnsmit/binance-full-history\n\n25GB of parquet files, containing 1m historical candles for the top 1000 markets on Binance, going back to 2017 :-)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1576155,
      "author_name": "gauravbrills",
      "author_url": "",
      "post_date": "11/09/2021 00:19:37",
      "content": "<p>Great thanks was about to start on this and was looking to source the same . This will give a head start 👍</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1576730,
      "author_name": "ajpass",
      "author_url": "",
      "post_date": "11/09/2021 12:24:25",
      "content": "<p>Just what I thought of doing <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a>. I just made a <a href=\"https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286491\" target=\"_blank\">discussion topic</a> to help me understand what VWAP is before doing what you're making. If I can help you with something just tell me. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1576880,
      "author_name": "julian3833",
      "author_url": "",
      "post_date": "11/09/2021 14:27:12",
      "content": "<p>Great resource, thanks for sharing!</p>\n<p>Side note: the first link is broken <a href=\"https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:\" target=\"_blank\">https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:</a>, it has the colon <br>\n(\":\") in it and gets a 404.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1579042,
      "author_name": "jorijnsmit",
      "author_url": "",
      "post_date": "11/11/2021 13:33:18",
      "content": "<p>Thanks for the mention <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a>. I updated the original dataset just now; <a href=\"https://www.kaggle.com/jorijnsmit/binance-full-history\" target=\"_blank\">https://www.kaggle.com/jorijnsmit/binance-full-history</a></p>\n<p>25GB of parquet files, containing 1m historical candles for the top 1000 markets on Binance, going back to 2017 :-)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1573783": "I finally managed to get some good external data & spent some time handling the preprocessing.\n\nI share two notebooks: \n\n  - Getting all 1m data from Binance: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance :\n\nUsing some code from @jorijnsmit available here: https://github.com/gosuto-ai/candlestick_retriever/blob/master/main.py I manage to get all relevant crypto for this competition at 1m intervall. The main assumption I make is using Binance USD (BUSD) instead of USDT as base conversion. I get data from 09/19 to today. I'll try to get more.\n\nThe raw output is available in this dataset: https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance\n\n - Transforming 1m data From Binance: https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data\n\nThen I process the raw data from Binance. I pack the data in one dataset and ensure the content and the format is the same as the training data set. There seems to be some remaining work:\n- VWAP is approximated from volumes and High / Low. I'll try to see If I can pull this data out of Binance\n- Target: I use the code from: https://www.kaggle.com/alexfir/recreating-target to build the target. It doesn't match exactly the existing data. There is an ongoing discussion about this.\n- Filtering rows: there seems to be some rows with 0 volume or other problems (infinite / nan VWAP/return). I haven't cleaned them yet as they might be the key to explain target building errors. \n\nThe preprocessed output is available in this dataset: https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance",
    "1576155": "Great thanks was about to start on this and was looking to source the same . This will give a head start 👍",
    "1576730": "Just what I thought of doing @lucasmorin. I just made a [discussion topic](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286491) to help me understand what VWAP is before doing what you're making. If I can help you with something just tell me.",
    "1576880": "Great resource, thanks for sharing!\n\nSide note: the first link is broken https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:, it has the colon \n(\":\") in it and gets a 404.",
    "1579042": "Thanks for the mention @lucasmorin. I updated the original dataset just now; https://www.kaggle.com/jorijnsmit/binance-full-history\n\n25GB of parquet files, containing 1m historical candles for the top 1000 markets on Binance, going back to 2017 :-)"
  },
  "source": "meta"
}