{ "id": 285726, "title": "[Extra Data] Auto updating dataset collecting market data | Baseline + CV ", "url": "/competitions/g-research-crypto-forecasting/discussion/285726", "author_name": "", "post_date": "2021-11-06T00:53:22.026849600Z", "votes": 117, "comment_count": 23, "views": 0, "content": "
\n

TL;DR: Baseline Notebook with CV | see baselines below ๐Ÿ‘‡

\n
\n

Auto updating - Full Price Datasets

\n

I collected an up-to-today [auto updating] dataset which contains the full historical data for all assets so you can easily build models that utilize it.
\nThe datasets are split to each asset since they are much heavier than the competition data.
\nThe datasets have also been labeled as described in the competition overview and had been organized in a way that they are at the exact format of the competition data.

\n
\n

The point is to provide a general dataset that:

\n
    \n
  1. Contains the FULL history for each asset. Currently, the competition data goes back to 2018. This dataset contains data from even earlier.
    2. \n
  2. Auto updating daily - Due to the high volatility of the cryptocurrency market, we should train our models on the most recent data available. These datasets have a backend pipeline for collecting, formatting, and reuploading to kaggle. They are scheduled to be updated daily, every single day until the end of the competition.
    3. \n
  3. Preprocessed - The datasets had been ffilled to overcome any missing values issue that is present in the original competition dataset.
    4. \n
\n
\n

The Datasets:

\n\n
\n

The Data
\nFor every asset in the competition, the following fields from Binance's official API endpoint for historical candlestick data are collected, saved, and processed.

\n
    \n
  1. timestamp - A timestamp for the minute covered by the row.
    2. \n
  2. Asset_ID - An ID code for the cryptoasset.
    3. \n
  3. Count - The number of trades that took place this minute.
    4. \n
  4. Open - The USD price at the beginning of the minute.
    5. \n
  5. High - The highest USD price during the minute.
    6. \n
  6. Low - The lowest USD price during the minute.
    7. \n
  7. Close - The USD price at the end of the minute.
    8. \n
  8. Volume - The number of cryptoasset u units traded during the minute.
    9. \n
  9. VWAP - The volume-weighted average price for the minute.
    \n10.Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.
    10. \n
  10. Weight - Weight, defined by the competition hosts here
    11. \n
  11. Asset_Name - Human readable Asset name.
    12. \n
\n

Indexing
\nThe dataframe is indexed by timestamp and sorted from oldest to newest.
\nThe first row starts at the first timestamp available on the exchange, which is July 2017 for the longest-running pairs.

\n

Loose-ends:
\nThis is a work in progress and will be updated constantly throughout the competition. At the moment, there are some known issues that still needed to be addressed:

\n\n

About auto-updates:

\n

I've updated the datasets once right now and so far the downloading process seems to be working. We can expect the datasets to be updated daily. [Note: this is still at an initial stage, there might be some set backs and delays on the first week]
\nWhen you download the dataset, the CSV files inside are of the exact same format at the competition.

\n

I have also created a starter notebook to set up Purged Group Time Series CV that is here.

\n
\n

Purged Group Time Series CV Flexible baseline

\n

This notebook is a simple starter notebook for Kaggle's Crypto Comp showing purged group time-series KFold with extra data.
\nPurged Times Series is explained here. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets in the same format as the competition, so you can input that into your model too.

\n

This notebook follows the ideas presented in my \"Initial Thoughts\" here. Some code sections have been reused from Chris' great (great great) notebook series on SIIM ISIC melanoma detection competition here

\n
\n

Baselines & Starter Notebooks

\n

I am working on making baselines package for this competition, this is a work in progress. These are the baselines that are currently ready:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CV + ModelHyperparam OptimizationTime Series ModelsFeature Engineering
Neural Network StarterMLP + AELSTMTechnical Analysis #1
LightGBM StarterLightGBMWavenetTechnical Analysis #2
Catboost StarterCatboostMultivariate-Transformer [written from scratch]Time Series Agg
XGBoost StarterXGboostN-BEATSNeutralization
Supervised AE [Janestreet 1st]Supervised AE [Janestreet 1st]DeepARQuant's Volatility Features
Transformer)TransformerโณTarget Engineering
TabNet StarterโณFourier Analysis
Reinforcement Learning (PPO) StarterโณWavelets
\n

About the validation: GroupTimeSeriesSplit

\n

(โณ - in the making..)

\n

Fork them as you please!
\nEnjoy Yourself!

\n
\n
\n

Bonus dataset: I've also uploaded an updated dataset of all Elon Musk's tweets ๐Ÿ˜‚. We can't use it for the private leaderboard but I just have to try! I am going to check if Elon Musk can move models up the leaderboard! ๐Ÿ‘Œ You can play with it yourself here.

\n
\n

Enjoy Yourself!
\nHappy Kaggling!

", "messages": [ { "id": "1572795", "postDate": "11/06/2021 00:53:22", "content": "
\n

TL;DR: Baseline Notebook with CV | see baselines below ๐Ÿ‘‡

\n
\n

Auto updating - Full Price Datasets

\n

I collected an up-to-today [auto updating] dataset which contains the full historical data for all assets so you can easily build models that utilize it.
\nThe datasets are split to each asset since they are much heavier than the competition data.
\nThe datasets have also been labeled as described in the competition overview and had been organized in a way that they are at the exact format of the competition data.

\n
\n

The point is to provide a general dataset that:

\n
    \n
  1. Contains the FULL history for each asset. Currently, the competition data goes back to 2018. This dataset contains data from even earlier.
    2. \n
  2. Auto updating daily - Due to the high volatility of the cryptocurrency market, we should train our models on the most recent data available. These datasets have a backend pipeline for collecting, formatting, and reuploading to kaggle. They are scheduled to be updated daily, every single day until the end of the competition.
    3. \n
  3. Preprocessed - The datasets had been ffilled to overcome any missing values issue that is present in the original competition dataset.
    4. \n
\n
\n

The Datasets:

\n\n
\n

The Data
\nFor every asset in the competition, the following fields from Binance's official API endpoint for historical candlestick data are collected, saved, and processed.

\n
    \n
  1. timestamp - A timestamp for the minute covered by the row.
    2. \n
  2. Asset_ID - An ID code for the cryptoasset.
    3. \n
  3. Count - The number of trades that took place this minute.
    4. \n
  4. Open - The USD price at the beginning of the minute.
    5. \n
  5. High - The highest USD price during the minute.
    6. \n
  6. Low - The lowest USD price during the minute.
    7. \n
  7. Close - The USD price at the end of the minute.
    8. \n
  8. Volume - The number of cryptoasset u units traded during the minute.
    9. \n
  9. VWAP - The volume-weighted average price for the minute.
    \n10.Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.
    10. \n
  10. Weight - Weight, defined by the competition hosts here
    11. \n
  11. Asset_Name - Human readable Asset name.
    12. \n
\n

Indexing
\nThe dataframe is indexed by timestamp and sorted from oldest to newest.
\nThe first row starts at the first timestamp available on the exchange, which is July 2017 for the longest-running pairs.

\n

Loose-ends:
\nThis is a work in progress and will be updated constantly throughout the competition. At the moment, there are some known issues that still needed to be addressed:

\n\n

About auto-updates:

\n

I've updated the datasets once right now and so far the downloading process seems to be working. We can expect the datasets to be updated daily. [Note: this is still at an initial stage, there might be some set backs and delays on the first week]
\nWhen you download the dataset, the CSV files inside are of the exact same format at the competition.

\n

I have also created a starter notebook to set up Purged Group Time Series CV that is here.

\n
\n

Purged Group Time Series CV Flexible baseline

\n

This notebook is a simple starter notebook for Kaggle's Crypto Comp showing purged group time-series KFold with extra data.
\nPurged Times Series is explained here. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets in the same format as the competition, so you can input that into your model too.

\n

This notebook follows the ideas presented in my \"Initial Thoughts\" here. Some code sections have been reused from Chris' great (great great) notebook series on SIIM ISIC melanoma detection competition here

\n
\n

Baselines & Starter Notebooks

\n

I am working on making baselines package for this competition, this is a work in progress. These are the baselines that are currently ready:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CV + ModelHyperparam OptimizationTime Series ModelsFeature Engineering
Neural Network StarterMLP + AELSTMTechnical Analysis #1
LightGBM StarterLightGBMWavenetTechnical Analysis #2
Catboost StarterCatboostMultivariate-Transformer [written from scratch]Time Series Agg
XGBoost StarterXGboostN-BEATSNeutralization
Supervised AE [Janestreet 1st]Supervised AE [Janestreet 1st]DeepARQuant's Volatility Features
Transformer)TransformerโณTarget Engineering
TabNet StarterโณFourier Analysis
Reinforcement Learning (PPO) StarterโณWavelets
\n

About the validation: GroupTimeSeriesSplit

\n

(โณ - in the making..)

\n

Fork them as you please!
\nEnjoy Yourself!

\n
\n
\n

Bonus dataset: I've also uploaded an updated dataset of all Elon Musk's tweets ๐Ÿ˜‚. We can't use it for the private leaderboard but I just have to try! I am going to check if Elon Musk can move models up the leaderboard! ๐Ÿ‘Œ You can play with it yourself here.

\n
\n

Enjoy Yourself!
\nHappy Kaggling!

", "rawMarkdown": "> **TL;DR:** [Baseline Notebook with CV](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | see baselines below ๐Ÿ‘‡\n\n\n\n### Auto updating - Full Price Datasets\n\nI collected an up-to-today [auto updating] dataset which contains the full historical data for all assets so you can easily build models that utilize it.\nThe datasets are split to each asset since they are much heavier than the competition data.\nThe datasets have also been labeled as described in the competition overview and had been organized in a way that they are at the exact format of the competition data.\n\n____\n\n**The point is to provide a general dataset that:**\n1. Contains the FULL history for each asset. Currently, the competition data goes back to 2018. This dataset contains data from even earlier.\n2. Auto updating daily - Due to the high volatility of the cryptocurrency market, we should train our models on the most recent data available. These datasets have a backend pipeline for collecting, formatting, and reuploading to kaggle. They are scheduled to be updated daily, every single day until the end of the competition.\n3. Preprocessed - The datasets had been ffilled to overcome any missing values issue that is present in the original competition dataset.\n\n____\n\n**The Datasets:**\n- [Binance Coin](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-binance-coin)\n- [Bitcoin Cash](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-bitcoin-cash)\n- [Bitcoin](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-bitcoin)\n- [Cardano](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-cardano)\n- [Dogecoin](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-dogecoin) \n- [Eos.io](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-eos-io)\n- [Ethereum](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-ethereum)\n- [Ethereum Classic](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-ethereum-classic)\n- [Iota](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-iota)\n- [Litecoin](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-litecoin)\n- [Monero](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-monero)\n- [Maker](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-maker)\n- [Stellar](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-stellar)\n- [TRON](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-tron)\n\n____\n\n**The Data**\nFor every asset in the competition, the following fields from [Binance's official API endpoint for historical candlestick data](https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md#klinecandlestick-data) are collected, saved, and processed.\n\n\n\n1. timestamp - A timestamp for the minute covered by the row.\n2. Asset_ID - An ID code for the cryptoasset.\n3. Count - The number of trades that took place this minute.\n4. Open - The USD price at the beginning of the minute.\n5. High - The highest USD price during the minute.\n6. Low - The lowest USD price during the minute.\n7. Close - The USD price at the end of the minute.\n8. Volume - The number of cryptoasset u units traded during the minute.\n9. VWAP - The volume-weighted average price for the minute.\n10.Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n12. Asset_Name - Human readable Asset name.\n\n\n**Indexing**\nThe dataframe is indexed by `timestamp` and sorted from oldest to newest. \nThe first row starts at the first timestamp available on the exchange, which is July 2017 for the longest-running pairs.\n\n**Loose-ends:**\nThis is a work in progress and will be updated constantly throughout the competition. At the moment, there are some known issues that still needed to be addressed:\n\n- **VWAP:** - At the moment VWAP calculation formula is still unclear. Currently the dataset uses an approximation calculated from the Open, High, Low, Close, Volume candlesticks. [Waiting for competition hosts input]\n- **Target Labeling:** There exist some mismatches to the original target provided by the hosts at some time intervals. On all the others - it is the same. The labeling code can be seen [here](https://www.kaggle.com/alexfir/recreating-target). [Waiting for competition hosts] input] \n- **Filtering:** No filtration of 0 volume data is taken place.\n\n**About auto-updates:**\n\nI've updated the datasets once right now and so far the downloading process seems to be working. We can expect the datasets to be updated daily. [Note: this is still at an initial stage, there might be some set backs and delays on the first week]\nWhen you download the dataset, the CSV files inside are of the exact same format at the competition.\n\nI have also created a starter notebook to set up Purged Group Time Series CV that is [here](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data).\n\n____\n\n### Purged Group Time Series CV Flexible baseline\n\nThis notebook is a simple starter notebook for Kaggle's Crypto Comp showing purged group time-series KFold with extra data. \nPurged Times Series is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets in the same format as the competition, so you can input that into your model too.\n\nThis notebook follows the ideas presented in my \"Initial Thoughts\" [here][1]. Some code sections have been reused from Chris' great (great great) notebook series on SIIM ISIC melanoma detection competition [here][3]\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n[3]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n\n____\n\n### Baselines & Starter Notebooks\n\nI am working on making baselines package for this competition, this is a work in progress. These are the baselines that are currently ready:\n| CV + Model | Hyperparam Optimization | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner) | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | [Technical Analysis #1](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features) |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna) | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | [Technical Analysis #2](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2) |\n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost) | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | [Time Series Agg](https://www.kaggle.com/yamqwe/features-all-time-series-aggregations-ever) | \n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data) | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | [N-BEATS](https://www.kaggle.com/yamqwe/crypto-forecasting-n-beats) | [Neutralization](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/) |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | [DeepAR](https://www.kaggle.com/yamqwe/probabilistic-forecasting-deepar/) | [Quant's Volatility Features](https://www.kaggle.com/yamqwe/crypto-prediction-volatility-features) |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer) | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline) | | โณTarget Engineering |\n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data) | | |โณFourier Analysis | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) | | | โณWavelets | \n\n\n[About the validation: GroupTimeSeriesSplit](https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit)\n\n(โณ - in the making..)\n\n\n\n\n\n\n\nFork them as you please!\nEnjoy Yourself! \n____\n\n\n> **Bonus dataset:** I've also uploaded an updated dataset of all Elon Musk's tweets ๐Ÿ˜‚. We can't use it for the private leaderboard but I just have to try! I am going to check if Elon Musk can move models up the leaderboard! ๐Ÿ‘Œ You can play with it yourself [here](https://www.kaggle.com/yamqwe/elon-musks-twitter-updated-031121).\n\n\n\nEnjoy Yourself!\nHappy Kaggling!", "votes": null }, { "id": "1573014", "postDate": "11/06/2021 07:07:04", "content": "

Quick update: Had some issues with the notebook crashing, fixed it and it currently committing.

", "rawMarkdown": "Quick update: Had some issues with the notebook crashing, fixed it and it currently committing.", "votes": null }, { "id": "1573032", "postDate": "11/06/2021 08:01:44", "content": "

Quick update: Had some issues with datasets uploading empty. reuploading it all now. They will be up and running in approx 1-Hour.

\n

Also, I have some suspicions about my labeling method. can you please point me to a labeling function you trust so I can validate mine?

", "rawMarkdown": "Quick update: Had some issues with datasets uploading empty. reuploading it all now. They will be up and running in approx 1-Hour. \n\nAlso, I have some suspicions about my labeling method. can you please point me to a labeling function you trust so I can validate mine?", "votes": null }, { "id": "1573065", "postDate": "11/06/2021 08:40:08", "content": "

Mind to share the code used to generate datasets ? the code used for labelling ? For the moment I don't think there is a 'validated' method to calculate beta.
\nEdit: I have found and shared some code (here: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance). I'd like to know if we have the same source and options (BUSD v.s. USDT).

", "rawMarkdown": "Mind to share the code used to generate datasets ? the code used for labelling ? For the moment I don't think there is a 'validated' method to calculate beta. \nEdit: I have found and shared some code (here: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance). I'd like to know if we have the same source and options (BUSD v.s. USDT).", "votes": null }, { "id": "1573118", "postDate": "11/06/2021 09:57:17", "content": "

Also you have 1440 instance per year. That's a 6h timestep. Can you confirm this ?

", "rawMarkdown": "Also you have 1440 instance per year. That's a 6h timestep. Can you confirm this ?", "votes": null }, { "id": "1573721", "postDate": "11/06/2021 21:10:27", "content": "

Looks nice, thanks.
\nCan I see which cells were initially empty?

", "rawMarkdown": "Looks nice, thanks.\nCan I see which cells were initially empty?", "votes": null }, { "id": "1573730", "postDate": "11/06/2021 21:18:05", "content": "

Thank you very much!
\nDo you plan on adding WVAP and Beta(target) calculations to the coins?
\nAlso, count is the volume right?

", "rawMarkdown": "Thank you very much!\nDo you plan on adding WVAP and Beta(target) calculations to the coins?\nAlso, count is the volume right?", "votes": null }, { "id": "1573877", "postDate": "11/07/2021 02:12:29", "content": "

Thank you so much for the datasets. Does it make more sense to save as parquet or feather for faster processing?

", "rawMarkdown": "Thank you so much for the datasets. Does it make more sense to save as parquet or feather for faster processing?", "votes": null }, { "id": "1574113", "postDate": "11/07/2021 08:50:20", "content": "

Sweet! Does more historical data really help with minute level data that already includes a few years though?

", "rawMarkdown": "Sweet! Does more historical data really help with minute level data that already includes a few years though?", "votes": null }, { "id": "1574197", "postDate": "11/07/2021 10:07:21", "content": "

Status update - 7.11.2021

\n
\n
    \n
  1. Tagging code: After struggling for hours implementing the target recreation from scratch, I've updated the tagging code of the dataset to be aligned with the proposed approach by @alexfir & @lucasmorin that you can read here and here.
    2. \n
  2. Automated collection: Last night we had our 2nd smooth update! All automated! so far, so good!
    3. \n
  3. Tabular model baseline: Similar to the Deep Learning Baseline, I made a version that implements a Tabular Baseline using LGBM for the problem.
    4. \n
  4. Reinforcement Learning baseline: Similar to the Supervised Learning Baseline, I made a version that implements a Reinforcement Learning Agent for the problem.
    5. \n
  5. Collection & Tagging code release: I plan to make the collection and tagging code public so we can all validate it together. Hopefully in the upcoming week!
    6. \n
\n

Thank you guys for the support! this isn't an easy project to maintain and your support makes the \"motivation issues\" slightly easier ๐Ÿ˜‚
\n[And special thanks to those that gave feedback! your feedback really helps make the dataset useful!]

\n

Peace โœŒ๏ธ

", "rawMarkdown": "Status update - 7.11.2021\n____\n\n1. **Tagging code:** After struggling for hours implementing the target recreation from scratch, I've updated the tagging code of the dataset to be aligned with the proposed approach by @alexfir & @lucasmorin that you can read [here](https://www.kaggle.com/alexfir/recreating-target) and [here](https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance). \n2. **Automated collection:** Last night we had our 2nd smooth update! All automated! so far, so good! \n3. **Tabular model baseline:** Similar to the [Deep Learning Baseline](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn), I made a version that implements a [Tabular Baseline using LGBM](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) for the problem.\n4. **Reinforcement Learning baseline:** Similar to the [Supervised Learning Baseline](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn), I made a version that implements a [Reinforcement Learning Agent](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) for the problem.\n5. **Collection & Tagging code release:** I plan to make the collection and tagging code public so we can all validate it together. Hopefully in the upcoming week! \n\n\nThank you guys for the support! this isn't an easy project to maintain and your support makes the \"motivation issues\" slightly easier ๐Ÿ˜‚\n[And special thanks to those that gave feedback! your feedback really helps make the dataset useful!]\n\nPeace โœŒ๏ธ", "votes": null }, { "id": "1574383", "postDate": "11/07/2021 13:54:15", "content": "

Thanks.
\nWhat exchange is this data collected from?

", "rawMarkdown": "Thanks.\nWhat exchange is this data collected from?", "votes": null }, { "id": "1575205", "postDate": "11/08/2021 08:31:29", "content": "

2 questions:

\n
    \n
  1. Have we confirmed that the original data provided by the host was from Binance?
    2. \n
  2. Is this auto-updating approach approved by the host or Kaggle staff? So far I've seen a lot of people
    \nasking this but have not seen any confirmation. How this data should be used in the forecasting phase when the Internet connection is disabled?
    3. \n
", "rawMarkdown": "2 questions:\n1. Have we confirmed that the original data provided by the host was from Binance?\n2. Is this auto-updating approach approved by the host or Kaggle staff? So far I've seen a lot of people\nasking this but have not seen any confirmation. How this data should be used in the forecasting phase when the Internet connection is disabled?", "votes": null }, { "id": "1575546", "postDate": "11/08/2021 13:08:06", "content": "

Thanks for sharing content.

", "rawMarkdown": "Thanks for sharing content.", "votes": null }, { "id": "1576207", "postDate": "11/09/2021 01:57:52", "content": "

I really appreciate your efforts, but all the numbers in each year are the same. In other words, the file with 2018 data is the same as the file for 2021.

", "rawMarkdown": "I really appreciate your efforts, but all the numbers in each year are the same. In other words, the file with 2018 data is the same as the file for 2021.", "votes": null }, { "id": "1577522", "postDate": "11/10/2021 06:50:37", "content": "

Status update 10-11-2021:

\n", "rawMarkdown": "**Status update 10-11-2021:**\n- Data collection had been going smoothly for the past 3 days. \n- Working on VWAP & Targets creation. Minimizing the mean distance to the originals. \n- Just updated the post with the full baselines table. \n- You guys rock! Many many forks of the baseline notebooks in just a couple of days! ๐Ÿคฉ", "votes": null }, { "id": "1579384", "postDate": "11/11/2021 20:31:01", "content": "

Thank you for your efforts. Is it just me or is there a fault in the timestamp columns. The 5 dataframes for each asset seem to have the exact (and therefore incorrect) timestamps. For example, the dataframes for Binance Coin all start at timestamp 1509939540 (2018-01-01 00:01) and (currently) end at 1636501380 (2021-11-09 23:43). Is there something I am missing?

", "rawMarkdown": "Thank you for your efforts. Is it just me or is there a fault in the timestamp columns. The 5 dataframes for each asset seem to have the exact (and therefore incorrect) timestamps. For example, the dataframes for Binance Coin all start at timestamp 1509939540 (2018-01-01 00:01) and (currently) end at 1636501380 (2021-11-09 23:43). Is there something I am missing?", "votes": null }, { "id": "1579427", "postDate": "11/11/2021 21:52:42", "content": "

The original submission code on the XGBoost starter notebook doesn't seem to be working for me. If you are having the same issue, please feel free to use this updated notebook that fixed this issue: https://www.kaggle.com/mingyangyi/xgb-extra-data-submit

", "rawMarkdown": "The original submission code on the XGBoost starter notebook doesn't seem to be working for me. If you are having the same issue, please feel free to use this updated notebook that fixed this issue: https://www.kaggle.com/mingyangyi/xgb-extra-data-submit", "votes": null }, { "id": "1582433", "postDate": "11/14/2021 22:19:18", "content": "

Thank you! Some notebooks (e.g. XGBoost) seem to run into an error in the submission stage though.

", "rawMarkdown": "Thank you! Some notebooks (e.g. XGBoost) seem to run into an error in the submission stage though.", "votes": null }, { "id": "1585982", "postDate": "11/17/2021 17:31:34", "content": "

Status update 17-11-2021:

\n", "rawMarkdown": "**Status update 17-11-2021:**\n- **Target Computation:** Targets creation had been updated to the [code](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778) released by the hosts. And was validated to make sure it is similar [~0.00012526]\n- **Bug fix:** one of the updates to the collection code introduced a bug where all the data frames were of one asset. This was pointed out in the discussions of the dataset and fixed within minutes. The datasets are fixed now and usable again. \n- **Feature engineering baselines:** Three new feature engineering baselines had been added.\n- **Baselines genera; update:** All the baselines had been updated to support submitting to the LB [compute time issues..] also many small bugs were fixed. I advise you to recheck them if you use code from them. \n- **Distance to the competition data:** Some pointed out that in some eras the data is slightly different than the competition data. I am checking this out now. [I mean, right now. the comparison runs in the background while I type this ๐Ÿ˜„]\n- **Uptime**: Data collection had been going smoothly for 9/10 days since the project started! \n\n- Thank you all for your support! ๐Ÿ˜", "votes": null }, { "id": "1590299", "postDate": "11/21/2021 04:36:47", "content": "
\n

Bonus dataset: I've also uploaded an updated dataset of all Elon Musk's tweets ๐Ÿ˜‚. We can't use it for the private leaderboard but I just have to try! I am going to check if Elon Musk can move models up the leaderboard! ๐Ÿ‘Œ You can play with it yourself here.

\n
\n

it is so cool ๐Ÿ˜‚, but I don't think it will work, because after the end of submit, our model can't predict by tweets. Musk's tweet only can be used to remove bias๐Ÿ˜‚๐Ÿ˜‚๐Ÿ˜‚

", "rawMarkdown": "> Bonus dataset: I've also uploaded an updated dataset of all Elon Musk's tweets ๐Ÿ˜‚. We can't use it for the private leaderboard but I just have to try! I am going to check if Elon Musk can move models up the leaderboard! ๐Ÿ‘Œ You can play with it yourself here.\n\nit is so cool ๐Ÿ˜‚, but I don't think it will work, because after the end of submit, our model can't predict by tweets. Musk's tweet only can be used to remove bias๐Ÿ˜‚๐Ÿ˜‚๐Ÿ˜‚", "votes": null }, { "id": "1590666", "postDate": "11/21/2021 14:45:13", "content": "

Hi @yamqwe - you say these notebooks \"are scheduled to be updated daily, every single day until the end of the competition.\" - does this mean that a notebook based on these will also continue to get daily updates, even after the submission deadline (i.e. in the testing phase of the competition) even though it is set to have internet turned off?

", "rawMarkdown": "Hi @yamqwe - you say these notebooks \"are scheduled to be updated daily, every single day until the end of the competition.\" - does this mean that a notebook based on these will also continue to get daily updates, even after the submission deadline (i.e. in the testing phase of the competition) even though it is set to have internet turned off?", "votes": null }, { "id": "1590728", "postDate": "11/21/2021 15:40:10", "content": "

Are you sure that the data sets are actually updating?

\n

For example, for the 2021 Binance Coin data, I'd have thought that it's last entry would have been from the day when the dataset was created, yet it seems to be in this range:

\n

Start = 2021-01-01 00:01:00
\nEnd = 2021-10-07 23:59:00

\n

i.e. it ends on the 7th October 2021, not 20th November 2021 when the dataset was created (Version 18 of the dataset)

\n

These values were obtained by doing:

\n
crypto_0_2021 = pd.read_csv(\"../input/cryptocurrency-extra-data-binance-coin/full_data__0__2021.csv\") \npd.to_datetime(crypto_0_2021['timestamp'], unit = 's') \n
\n

Also the bitcoin data seems to have had some extra columns introduced that aren't in the original training data:
\n'Unnamed: 0', 'Asset_Name' and 'Weight'

", "rawMarkdown": "Are you sure that the data sets are actually updating?\n\nFor example, for the [2021 Binance Coin data](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-binance-coin?select=full_data__0__2021.csv), I'd have thought that it's last entry would have been from the day when the dataset was created, yet it seems to be in this range:\n\nStart = 2021-01-01 00:01:00\nEnd = 2021-10-07 23:59:00\n\ni.e. it ends on the 7th October 2021, not 20th November 2021 when the dataset was created (Version 18 of the dataset)\n\nThese values were obtained by doing:\n\n```\ncrypto_0_2021 = pd.read_csv(\"../input/cryptocurrency-extra-data-binance-coin/full_data__0__2021.csv\") \npd.to_datetime(crypto_0_2021['timestamp'], unit = 's') \n```\n\nAlso the bitcoin data seems to have had some extra columns introduced that aren't in the original training data:\n'Unnamed: 0', 'Asset_Name' and 'Weight'", "votes": null }, { "id": "1590783", "postDate": "11/21/2021 16:41:46", "content": "

My bad! - the datasets are updating - I just needed to sort by date to see it:

\n
crypto_0_2021 = pd.read_csv(\"../input/cryptocurrency-extra-data-binance-coin/full_data__0__2021.csv\",low_memory=False) \ncrypto_0_2021['date'] = pd.to_datetime(crypto_0_2021['timestamp'], unit = 's')        \ncrypto_0_2021 = crypto_0_2021.sort_values('date')\ncrypto_0_2021.tail()\n
\n

Gives an end date: 2021-11-17 23:43:00

", "rawMarkdown": "My bad! - the datasets are updating - I just needed to sort by date to see it:\n\n```\ncrypto_0_2021 = pd.read_csv(\"../input/cryptocurrency-extra-data-binance-coin/full_data__0__2021.csv\",low_memory=False) \ncrypto_0_2021['date'] = pd.to_datetime(crypto_0_2021['timestamp'], unit = 's') \ncrypto_0_2021 = crypto_0_2021.sort_values('date')\ncrypto_0_2021.tail()\n```\n\nGives an end date: 2021-11-17 23:43:00", "votes": null }, { "id": "1733078", "postDate": "03/24/2022 02:04:03", "content": "

can some friend send me the main datasets

", "rawMarkdown": "can some friend send me the main datasets", "votes": null } ], "comments": [ { "id": 1573014, "author_name": "", "author_url": "", "post_date": "11/06/2021 07:07:04", "content": "

Quick update: Had some issues with the notebook crashing, fixed it and it currently committing.

", "votes": null, "replies": [] }, { "id": 1573032, "author_name": "", "author_url": "", "post_date": "11/06/2021 08:01:44", "content": "

Quick update: Had some issues with datasets uploading empty. reuploading it all now. They will be up and running in approx 1-Hour.

\n

Also, I have some suspicions about my labeling method. can you please point me to a labeling function you trust so I can validate mine?

", "votes": null, "replies": [ { "id": 1573065, "author_name": "lucasmorin", "author_url": "", "post_date": "11/06/2021 08:40:08", "content": "

Mind to share the code used to generate datasets ? the code used for labelling ? For the moment I don't think there is a 'validated' method to calculate beta.
\nEdit: I have found and shared some code (here: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance). I'd like to know if we have the same source and options (BUSD v.s. USDT).

", "votes": null, "replies": [] } ] }, { "id": 1573118, "author_name": "lucasmorin", "author_url": "", "post_date": "11/06/2021 09:57:17", "content": "

Also you have 1440 instance per year. That's a 6h timestep. Can you confirm this ?

", "votes": null, "replies": [] }, { "id": 1573721, "author_name": "manelab", "author_url": "", "post_date": "11/06/2021 21:10:27", "content": "

Looks nice, thanks.
\nCan I see which cells were initially empty?

", "votes": null, "replies": [] }, { "id": 1573730, "author_name": "eladdv", "author_url": "", "post_date": "11/06/2021 21:18:05", "content": "

Thank you very much!
\nDo you plan on adding WVAP and Beta(target) calculations to the coins?
\nAlso, count is the volume right?

", "votes": null, "replies": [] }, { "id": 1573877, "author_name": "johnyan1019", "author_url": "", "post_date": "11/07/2021 02:12:29", "content": "

Thank you so much for the datasets. Does it make more sense to save as parquet or feather for faster processing?

", "votes": null, "replies": [] }, { "id": 1574113, "author_name": "danofer", "author_url": "", "post_date": "11/07/2021 08:50:20", "content": "

Sweet! Does more historical data really help with minute level data that already includes a few years though?

", "votes": null, "replies": [] }, { "id": 1574197, "author_name": "", "author_url": "", "post_date": "11/07/2021 10:07:21", "content": "

Status update - 7.11.2021

\n
\n
    \n
  1. Tagging code: After struggling for hours implementing the target recreation from scratch, I've updated the tagging code of the dataset to be aligned with the proposed approach by @alexfir & @lucasmorin that you can read here and here.
    2. \n
  2. Automated collection: Last night we had our 2nd smooth update! All automated! so far, so good!
    3. \n
  3. Tabular model baseline: Similar to the Deep Learning Baseline, I made a version that implements a Tabular Baseline using LGBM for the problem.
    4. \n
  4. Reinforcement Learning baseline: Similar to the Supervised Learning Baseline, I made a version that implements a Reinforcement Learning Agent for the problem.
    5. \n
  5. Collection & Tagging code release: I plan to make the collection and tagging code public so we can all validate it together. Hopefully in the upcoming week!
    6. \n
\n

Thank you guys for the support! this isn't an easy project to maintain and your support makes the \"motivation issues\" slightly easier ๐Ÿ˜‚
\n[And special thanks to those that gave feedback! your feedback really helps make the dataset useful!]

\n

Peace โœŒ๏ธ

", "votes": null, "replies": [] }, { "id": 1574383, "author_name": "shahnawazakhtar", "author_url": "", "post_date": "11/07/2021 13:54:15", "content": "

Thanks.
\nWhat exchange is this data collected from?

", "votes": null, "replies": [] }, { "id": 1575205, "author_name": "wenrui29", "author_url": "", "post_date": "11/08/2021 08:31:29", "content": "

2 questions:

\n
    \n
  1. Have we confirmed that the original data provided by the host was from Binance?
    2. \n
  2. Is this auto-updating approach approved by the host or Kaggle staff? So far I've seen a lot of people
    \nasking this but have not seen any confirmation. How this data should be used in the forecasting phase when the Internet connection is disabled?
    3. \n
", "votes": null, "replies": [] }, { "id": 1575546, "author_name": "rfsilvadata", "author_url": "", "post_date": "11/08/2021 13:08:06", "content": "

Thanks for sharing content.

", "votes": null, "replies": [] }, { "id": 1576207, "author_name": "tapioca", "author_url": "", "post_date": "11/09/2021 01:57:52", "content": "

I really appreciate your efforts, but all the numbers in each year are the same. In other words, the file with 2018 data is the same as the file for 2021.

", "votes": null, "replies": [] }, { "id": 1577522, "author_name": "", "author_url": "", "post_date": "11/10/2021 06:50:37", "content": "

Status update 10-11-2021:

\n", "votes": null, "replies": [] }, { "id": 1579384, "author_name": "alexanderkern", "author_url": "", "post_date": "11/11/2021 20:31:01", "content": "

Thank you for your efforts. Is it just me or is there a fault in the timestamp columns. The 5 dataframes for each asset seem to have the exact (and therefore incorrect) timestamps. For example, the dataframes for Binance Coin all start at timestamp 1509939540 (2018-01-01 00:01) and (currently) end at 1636501380 (2021-11-09 23:43). Is there something I am missing?

", "votes": null, "replies": [] }, { "id": 1579427, "author_name": "mingyangyi", "author_url": "", "post_date": "11/11/2021 21:52:42", "content": "

The original submission code on the XGBoost starter notebook doesn't seem to be working for me. If you are having the same issue, please feel free to use this updated notebook that fixed this issue: https://www.kaggle.com/mingyangyi/xgb-extra-data-submit

", "votes": null, "replies": [] }, { "id": 1582433, "author_name": "mingyangyi", "author_url": "", "post_date": "11/14/2021 22:19:18", "content": "

Thank you! Some notebooks (e.g. XGBoost) seem to run into an error in the submission stage though.

", "votes": null, "replies": [] }, { "id": 1585982, "author_name": "", "author_url": "", "post_date": "11/17/2021 17:31:34", "content": "

Status update 17-11-2021:

\n", "votes": null, "replies": [] }, { "id": 1590299, "author_name": "huangzchao", "author_url": "", "post_date": "11/21/2021 04:36:47", "content": "
\n

Bonus dataset: I've also uploaded an updated dataset of all Elon Musk's tweets ๐Ÿ˜‚. We can't use it for the private leaderboard but I just have to try! I am going to check if Elon Musk can move models up the leaderboard! ๐Ÿ‘Œ You can play with it yourself here.

\n
\n

it is so cool ๐Ÿ˜‚, but I don't think it will work, because after the end of submit, our model can't predict by tweets. Musk's tweet only can be used to remove bias๐Ÿ˜‚๐Ÿ˜‚๐Ÿ˜‚

", "votes": null, "replies": [] }, { "id": 1590666, "author_name": "steveroberts", "author_url": "", "post_date": "11/21/2021 14:45:13", "content": "

Hi @yamqwe - you say these notebooks \"are scheduled to be updated daily, every single day until the end of the competition.\" - does this mean that a notebook based on these will also continue to get daily updates, even after the submission deadline (i.e. in the testing phase of the competition) even though it is set to have internet turned off?

", "votes": null, "replies": [] }, { "id": 1590728, "author_name": "steveroberts", "author_url": "", "post_date": "11/21/2021 15:40:10", "content": "

Are you sure that the data sets are actually updating?

\n

For example, for the 2021 Binance Coin data, I'd have thought that it's last entry would have been from the day when the dataset was created, yet it seems to be in this range:

\n

Start = 2021-01-01 00:01:00
\nEnd = 2021-10-07 23:59:00

\n

i.e. it ends on the 7th October 2021, not 20th November 2021 when the dataset was created (Version 18 of the dataset)

\n

These values were obtained by doing:

\n
crypto_0_2021 = pd.read_csv(\"../input/cryptocurrency-extra-data-binance-coin/full_data__0__2021.csv\") \npd.to_datetime(crypto_0_2021['timestamp'], unit = 's') \n
\n

Also the bitcoin data seems to have had some extra columns introduced that aren't in the original training data:
\n'Unnamed: 0', 'Asset_Name' and 'Weight'

", "votes": null, "replies": [ { "id": 1590783, "author_name": "steveroberts", "author_url": "", "post_date": "11/21/2021 16:41:46", "content": "

My bad! - the datasets are updating - I just needed to sort by date to see it:

\n
crypto_0_2021 = pd.read_csv(\"../input/cryptocurrency-extra-data-binance-coin/full_data__0__2021.csv\",low_memory=False) \ncrypto_0_2021['date'] = pd.to_datetime(crypto_0_2021['timestamp'], unit = 's')        \ncrypto_0_2021 = crypto_0_2021.sort_values('date')\ncrypto_0_2021.tail()\n
\n

Gives an end date: 2021-11-17 23:43:00

", "votes": null, "replies": [] } ] }, { "id": 1733078, "author_name": "mahdiasdzd", "author_url": "", "post_date": "03/24/2022 02:04:03", "content": "

can some friend send me the main datasets

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1572795": "> **TL;DR:** [Baseline Notebook with CV](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | see baselines below ๐Ÿ‘‡\n\n\n\n### Auto updating - Full Price Datasets\n\nI collected an up-to-today [auto updating] dataset which contains the full historical data for all assets so you can easily build models that utilize it.\nThe datasets are split to each asset since they are much heavier than the competition data.\nThe datasets have also been labeled as described in the competition overview and had been organized in a way that they are at the exact format of the competition data.\n\n____\n\n**The point is to provide a general dataset that:**\n1. Contains the FULL history for each asset. Currently, the competition data goes back to 2018. This dataset contains data from even earlier.\n2. Auto updating daily - Due to the high volatility of the cryptocurrency market, we should train our models on the most recent data available. These datasets have a backend pipeline for collecting, formatting, and reuploading to kaggle. They are scheduled to be updated daily, every single day until the end of the competition.\n3. Preprocessed - The datasets had been ffilled to overcome any missing values issue that is present in the original competition dataset.\n\n____\n\n**The Datasets:**\n- [Binance Coin](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-binance-coin)\n- [Bitcoin Cash](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-bitcoin-cash)\n- [Bitcoin](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-bitcoin)\n- [Cardano](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-cardano)\n- [Dogecoin](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-dogecoin) \n- [Eos.io](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-eos-io)\n- [Ethereum](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-ethereum)\n- [Ethereum Classic](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-ethereum-classic)\n- [Iota](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-iota)\n- [Litecoin](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-litecoin)\n- [Monero](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-monero)\n- [Maker](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-maker)\n- [Stellar](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-stellar)\n- [TRON](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-tron)\n\n____\n\n**The Data**\nFor every asset in the competition, the following fields from [Binance's official API endpoint for historical candlestick data](https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md#klinecandlestick-data) are collected, saved, and processed.\n\n\n\n1. timestamp - A timestamp for the minute covered by the row.\n2. Asset_ID - An ID code for the cryptoasset.\n3. Count - The number of trades that took place this minute.\n4. Open - The USD price at the beginning of the minute.\n5. High - The highest USD price during the minute.\n6. Low - The lowest USD price during the minute.\n7. Close - The USD price at the end of the minute.\n8. Volume - The number of cryptoasset u units traded during the minute.\n9. VWAP - The volume-weighted average price for the minute.\n10.Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n12. Asset_Name - Human readable Asset name.\n\n\n**Indexing**\nThe dataframe is indexed by `timestamp` and sorted from oldest to newest. \nThe first row starts at the first timestamp available on the exchange, which is July 2017 for the longest-running pairs.\n\n**Loose-ends:**\nThis is a work in progress and will be updated constantly throughout the competition. At the moment, there are some known issues that still needed to be addressed:\n\n- **VWAP:** - At the moment VWAP calculation formula is still unclear. Currently the dataset uses an approximation calculated from the Open, High, Low, Close, Volume candlesticks. [Waiting for competition hosts input]\n- **Target Labeling:** There exist some mismatches to the original target provided by the hosts at some time intervals. On all the others - it is the same. The labeling code can be seen [here](https://www.kaggle.com/alexfir/recreating-target). [Waiting for competition hosts] input] \n- **Filtering:** No filtration of 0 volume data is taken place.\n\n**About auto-updates:**\n\nI've updated the datasets once right now and so far the downloading process seems to be working. We can expect the datasets to be updated daily. [Note: this is still at an initial stage, there might be some set backs and delays on the first week]\nWhen you download the dataset, the CSV files inside are of the exact same format at the competition.\n\nI have also created a starter notebook to set up Purged Group Time Series CV that is [here](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data).\n\n____\n\n### Purged Group Time Series CV Flexible baseline\n\nThis notebook is a simple starter notebook for Kaggle's Crypto Comp showing purged group time-series KFold with extra data. \nPurged Times Series is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which years are loaded, which neural networks are used, and whether to use feature engineering. You can experiment with different data preprocessing, model architecture, loss, optimizers, and learning rate schedules. The extra datasets contain the full history of the assets in the same format as the competition, so you can input that into your model too.\n\nThis notebook follows the ideas presented in my \"Initial Thoughts\" [here][1]. Some code sections have been reused from Chris' great (great great) notebook series on SIIM ISIC melanoma detection competition [here][3]\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n[3]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n\n____\n\n### Baselines & Starter Notebooks\n\nI am working on making baselines package for this competition, this is a work in progress. These are the baselines that are currently ready:\n| CV + Model | Hyperparam Optimization | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner) | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | [Technical Analysis #1](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features) |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna) | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | [Technical Analysis #2](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2) |\n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost) | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | [Time Series Agg](https://www.kaggle.com/yamqwe/features-all-time-series-aggregations-ever) | \n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data) | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | [N-BEATS](https://www.kaggle.com/yamqwe/crypto-forecasting-n-beats) | [Neutralization](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/) |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | [DeepAR](https://www.kaggle.com/yamqwe/probabilistic-forecasting-deepar/) | [Quant's Volatility Features](https://www.kaggle.com/yamqwe/crypto-prediction-volatility-features) |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer) | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline) | | โณTarget Engineering |\n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data) | | |โณFourier Analysis | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) | | | โณWavelets | \n\n\n[About the validation: GroupTimeSeriesSplit](https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit)\n\n(โณ - in the making..)\n\n\n\n\n\n\n\nFork them as you please!\nEnjoy Yourself! \n____\n\n\n> **Bonus dataset:** I've also uploaded an updated dataset of all Elon Musk's tweets ๐Ÿ˜‚. We can't use it for the private leaderboard but I just have to try! I am going to check if Elon Musk can move models up the leaderboard! ๐Ÿ‘Œ You can play with it yourself [here](https://www.kaggle.com/yamqwe/elon-musks-twitter-updated-031121).\n\n\n\nEnjoy Yourself!\nHappy Kaggling!", "1573014": "Quick update: Had some issues with the notebook crashing, fixed it and it currently committing.", "1573032": "Quick update: Had some issues with datasets uploading empty. reuploading it all now. They will be up and running in approx 1-Hour. \n\nAlso, I have some suspicions about my labeling method. can you please point me to a labeling function you trust so I can validate mine?", "1573065": "Mind to share the code used to generate datasets ? the code used for labelling ? For the moment I don't think there is a 'validated' method to calculate beta. \nEdit: I have found and shared some code (here: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance). I'd like to know if we have the same source and options (BUSD v.s. USDT).", "1573118": "Also you have 1440 instance per year. That's a 6h timestep. Can you confirm this ?", "1573721": "Looks nice, thanks.\nCan I see which cells were initially empty?", "1573730": "Thank you very much!\nDo you plan on adding WVAP and Beta(target) calculations to the coins?\nAlso, count is the volume right?", "1573877": "Thank you so much for the datasets. Does it make more sense to save as parquet or feather for faster processing?", "1574113": "Sweet! Does more historical data really help with minute level data that already includes a few years though?", "1574197": "Status update - 7.11.2021\n____\n\n1. **Tagging code:** After struggling for hours implementing the target recreation from scratch, I've updated the tagging code of the dataset to be aligned with the proposed approach by @alexfir & @lucasmorin that you can read [here](https://www.kaggle.com/alexfir/recreating-target) and [here](https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance). \n2. **Automated collection:** Last night we had our 2nd smooth update! All automated! so far, so good! \n3. **Tabular model baseline:** Similar to the [Deep Learning Baseline](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn), I made a version that implements a [Tabular Baseline using LGBM](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) for the problem.\n4. **Reinforcement Learning baseline:** Similar to the [Supervised Learning Baseline](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn), I made a version that implements a [Reinforcement Learning Agent](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) for the problem.\n5. **Collection & Tagging code release:** I plan to make the collection and tagging code public so we can all validate it together. Hopefully in the upcoming week! \n\n\nThank you guys for the support! this isn't an easy project to maintain and your support makes the \"motivation issues\" slightly easier ๐Ÿ˜‚\n[And special thanks to those that gave feedback! your feedback really helps make the dataset useful!]\n\nPeace โœŒ๏ธ", "1574383": "Thanks.\nWhat exchange is this data collected from?", "1575205": "2 questions:\n1. Have we confirmed that the original data provided by the host was from Binance?\n2. Is this auto-updating approach approved by the host or Kaggle staff? So far I've seen a lot of people\nasking this but have not seen any confirmation. How this data should be used in the forecasting phase when the Internet connection is disabled?", "1575546": "Thanks for sharing content.", "1576207": "I really appreciate your efforts, but all the numbers in each year are the same. In other words, the file with 2018 data is the same as the file for 2021.", "1577522": "**Status update 10-11-2021:**\n- Data collection had been going smoothly for the past 3 days. \n- Working on VWAP & Targets creation. Minimizing the mean distance to the originals. \n- Just updated the post with the full baselines table. \n- You guys rock! Many many forks of the baseline notebooks in just a couple of days! ๐Ÿคฉ", "1579384": "Thank you for your efforts. Is it just me or is there a fault in the timestamp columns. The 5 dataframes for each asset seem to have the exact (and therefore incorrect) timestamps. For example, the dataframes for Binance Coin all start at timestamp 1509939540 (2018-01-01 00:01) and (currently) end at 1636501380 (2021-11-09 23:43). Is there something I am missing?", "1579427": "The original submission code on the XGBoost starter notebook doesn't seem to be working for me. If you are having the same issue, please feel free to use this updated notebook that fixed this issue: https://www.kaggle.com/mingyangyi/xgb-extra-data-submit", "1582433": "Thank you! Some notebooks (e.g. XGBoost) seem to run into an error in the submission stage though.", "1585982": "**Status update 17-11-2021:**\n- **Target Computation:** Targets creation had been updated to the [code](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778) released by the hosts. And was validated to make sure it is similar [~0.00012526]\n- **Bug fix:** one of the updates to the collection code introduced a bug where all the data frames were of one asset. This was pointed out in the discussions of the dataset and fixed within minutes. The datasets are fixed now and usable again. \n- **Feature engineering baselines:** Three new feature engineering baselines had been added.\n- **Baselines genera; update:** All the baselines had been updated to support submitting to the LB [compute time issues..] also many small bugs were fixed. I advise you to recheck them if you use code from them. \n- **Distance to the competition data:** Some pointed out that in some eras the data is slightly different than the competition data. I am checking this out now. [I mean, right now. the comparison runs in the background while I type this ๐Ÿ˜„]\n- **Uptime**: Data collection had been going smoothly for 9/10 days since the project started! \n\n- Thank you all for your support! ๐Ÿ˜", "1590299": "> Bonus dataset: I've also uploaded an updated dataset of all Elon Musk's tweets ๐Ÿ˜‚. We can't use it for the private leaderboard but I just have to try! I am going to check if Elon Musk can move models up the leaderboard! ๐Ÿ‘Œ You can play with it yourself here.\n\nit is so cool ๐Ÿ˜‚, but I don't think it will work, because after the end of submit, our model can't predict by tweets. Musk's tweet only can be used to remove bias๐Ÿ˜‚๐Ÿ˜‚๐Ÿ˜‚", "1590666": "Hi @yamqwe - you say these notebooks \"are scheduled to be updated daily, every single day until the end of the competition.\" - does this mean that a notebook based on these will also continue to get daily updates, even after the submission deadline (i.e. in the testing phase of the competition) even though it is set to have internet turned off?", "1590728": "Are you sure that the data sets are actually updating?\n\nFor example, for the [2021 Binance Coin data](https://www.kaggle.com/yamqwe/cryptocurrency-extra-data-binance-coin?select=full_data__0__2021.csv), I'd have thought that it's last entry would have been from the day when the dataset was created, yet it seems to be in this range:\n\nStart = 2021-01-01 00:01:00\nEnd = 2021-10-07 23:59:00\n\ni.e. it ends on the 7th October 2021, not 20th November 2021 when the dataset was created (Version 18 of the dataset)\n\nThese values were obtained by doing:\n\n```\ncrypto_0_2021 = pd.read_csv(\"../input/cryptocurrency-extra-data-binance-coin/full_data__0__2021.csv\") \npd.to_datetime(crypto_0_2021['timestamp'], unit = 's') \n```\n\nAlso the bitcoin data seems to have had some extra columns introduced that aren't in the original training data:\n'Unnamed: 0', 'Asset_Name' and 'Weight'", "1590783": "My bad! - the datasets are updating - I just needed to sort by date to see it:\n\n```\ncrypto_0_2021 = pd.read_csv(\"../input/cryptocurrency-extra-data-binance-coin/full_data__0__2021.csv\",low_memory=False) \ncrypto_0_2021['date'] = pd.to_datetime(crypto_0_2021['timestamp'], unit = 's') \ncrypto_0_2021 = crypto_0_2021.sort_values('date')\ncrypto_0_2021.tail()\n```\n\nGives an end date: 2021-11-17 23:43:00", "1733078": "can some friend send me the main datasets" }, "source": "meta" }