","metadata":{"_cell_guid":"241ce292-edca-4688-a665-c6321e80b438","_uuid":"79576405-5d2f-465f-bb01-10eee623172c","papermill":{"duration":0.036915,"end_time":"2021-10-29T13:38:47.184754","exception":false,"start_time":"2021-10-29T13:38:47.147839","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n","metadata":{"_cell_guid":"ca79faf7-ca73-4e9b-8552-be96bdc881f9","_uuid":"4e428101-c977-488b-aa10-fe96d228224f","papermill":{"duration":0.046159,"end_time":"2021-10-29T13:38:47.267941","exception":false,"start_time":"2021-10-29T13:38:47.221782","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":">### π₯π₯ Crypto Prediction: CatBoost Regressor π₯π₯\n>\n> **Just a simple pipeline going from zero to a valid submission**\n>\n> We train one `CatboostRegressor` for each asset over a very very naive set of features (the input dataframe `['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']`), we get the predictions correctly using the iterator and we submit. No validation for now, no cross validation... nothing at all lol: just the bare pipeline!\n>\n>\n>This notebook follows the ideas presented in my \"Initial Thoughts\" [here][1].\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284903\n[2]: https://www.kaggle.com/yamqwe/let-s-talk-validation-grouptimeseriessplit\n\n
\n\n| CV + Model | Hyperparam Optimization | Time Series Models | Feature Engineering |\n| --- | --- | --- | --- |\n| [Neural Network Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-nn) | [MLP + AE](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner) | [LSTM](https://www.kaggle.com/yamqwe/time-series-modeling-lstm) | [Technical Analysis #1](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features) |\n| [LightGBM Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-with-extra-data-lgbm) | [LightGBM](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna) | [Wavenet](https://www.kaggle.com/yamqwe/time-series-modeling-wavenet) | [Technical Analysis #2](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2) |\n| [Catboost Starter](https://www.kaggle.com/yamqwe/purgedgrouptimeseries-cv-extra-data-catboost) | [Catboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna) | [Multivariate-Transformer [written from scratch]](https://www.kaggle.com/yamqwe/time-series-modeling-multivariate-transformer) | [Time Series Agg](https://www.kaggle.com/yamqwe/features-all-time-series-aggregations-ever) | \n| [XGBoost Starter](https://www.kaggle.com/yamqwe/xgb-extra-data) | [XGboost](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna) | [N-BEATS](https://www.kaggle.com/yamqwe/crypto-forecasting-n-beats) | [Neutralization](https://www.kaggle.com/yamqwe/g-research-avoid-overfit-feature-neutralization/) |\n| [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-adapted-to-crypto) | [Supervised AE [Janestreet 1st]](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner) | [DeepAR](https://www.kaggle.com/yamqwe/probabilistic-forecasting-deepar/) | β³Target Engineering |\n| [Transformer)](https://www.kaggle.com/yamqwe/let-s-test-a-transformer) | [Transformer](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline) | |β³Quant's Volatility Features \n| [TabNet Starter](https://www.kaggle.com/yamqwe/tabnet-cv-extra-data) | | |β³Fourier Analysis | \n| [Reinforcement Learning (PPO) Starter](https://www.kaggle.com/yamqwe/g-research-reinforcement-learning-starter) | | | β³Wavelets | \n\n____","metadata":{"papermill":{"duration":0.016676,"end_time":"2021-11-10T08:29:54.452085","exception":false,"start_time":"2021-11-10T08:29:54.435409","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Table Of Content π\n","metadata":{"_cell_guid":"3208562a-4f7a-45d5-b8e2-67da427cd906","_uuid":"d7276e81-829f-4be5-967d-0a641fb1adbf","papermill":{"duration":0.037447,"end_time":"2021-10-29T13:38:47.950458","exception":false,"start_time":"2021-10-29T13:38:47.913011","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Table Of Content\n\n1. [π Introduction](#introduction)\n\n2. [π Table Of Content](#outline) \n\n3. [π€Ώ Diving into the Data](#diving) \n\n4. [π Imports](#imports) \n\n5. [ποΈ Training](#training)\n\n6. [π° Submit to Kaggle](#submit)","metadata":{"_cell_guid":"d70815a1-3398-43b2-be85-2230fffd89a9","_uuid":"e9b980c5-036c-4e25-992b-115169a95ba4","papermill":{"duration":0.040583,"end_time":"2021-10-29T13:38:48.028885","exception":false,"start_time":"2021-10-29T13:38:47.988302","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Diving into the Data π€Ώ\n","metadata":{"_cell_guid":"310c5bce-3fab-4a82-b70e-d5e8a0e5e594","_uuid":"ce509f90-2ae9-4ea3-94ad-39479b72ca12","papermill":{"duration":0.038699,"end_time":"2021-10-29T13:38:48.105343","exception":false,"start_time":"2021-10-29T13:38:48.066644","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n#### **Dataset Structure**\n\n> **train.csv** - The training set\n> \n> 1. timestamp - A timestamp for the minute covered by the row.\n> 2. Asset_ID - An ID code for the cryptoasset.\n> 3. Count - The number of trades that took place this minute.\n> 4. Open - The USD price at the beginning of the minute.\n> 5. High - The highest USD price during the minute.\n> 6. Low - The lowest USD price during the minute.\n> 7. Close - The USD price at the end of the minute.\n> 8. Volume - The number of cryptoasset u units traded during the minute.\n> 9. VWAP - The volume-weighted average price for the minute.\n> 10. Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n> 11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n> 12. Asset_Name - Human readable Asset name.\n> \n>\n> **example_test.csv** - An example of the data that will be delivered by the time series API.\n> \n> **example_sample_submission.csv** - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n> \n> **asset_details.csv** - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n> \n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n>\n> - π There are 14 coins in the dataset\n>\n> - π There are 4 years in the [full] dataset","metadata":{"_cell_guid":"ca6da936-954f-427b-af0c-f04517c4d4ed","_uuid":"a91494a0-a071-49e5-bcc0-9c7b7566ad75","papermill":{"duration":0.036962,"end_time":"2021-10-29T13:38:48.179784","exception":false,"start_time":"2021-10-29T13:38:48.142822","status":"completed"},"tags":[]}},{"cell_type":"code","source":"css_file = '''\ndiv #notebook {\nbackground-color: white;\nfont-family: 'Open Sans', Helvetica, sans-serif;\nline-height: 20px;\n}\n\n#notebook-container {\nmargin-top: 2em;\npadding-top: 2em;\nborder-top: 4px solid %s; /* light orange */\n-webkit-box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n box-shadow: 0px 0px 8px 2px rgba(224, 212, 226, 0.5); /* pink */\n}\n\ndiv .input {\nmargin-bottom: 1em;\n}\n\n.rendered_html h1, .rendered_html h2, .rendered_html h3, .rendered_html h4, .rendered_html h5, .rendered_html h6 {\ncolor: %s; /* light orange */\nfont-weight: 600;\n}\n\n.rendered_html code {\n background-color: #efefef; /* light gray */\n}\n\n.CodeMirror {\ncolor: #8c8c8c; /* dark gray */\npadding: 0.7em;\n}\n\ndiv.input_area {\nborder: none;\n background-color: %s; /* rgba(229, 143, 101, 0.1); light orange [exactly #E58F65] */\n border-top: 2px solid %s; /* light orange */\n}\n\ndiv.input_prompt {\ncolor: %s; /* light blue */\n}\n\ndiv.output_prompt {\ncolor: %s; /* strong orange */\n}\n\ndiv.cell.selected:before, div.cell.selected.jupyter-soft-selected:before {\nbackground: %s; /* light orange */\n}\n\ndiv.cell.selected, div.cell.selected.jupyter-soft-selected {\n border-color: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected:before {\nbackground: %s; /* light orange */\n}\n\n.edit_mode div.cell.selected {\nborder-color: %s; /* light orange */\n\n}\n'''\ndef to_rgb(h): return tuple(int(h[i:i+2], 16) for i in (0, 2, 4))\nmain_color_rgba = 'rgba(%s, %s, %s, 0.1)' % (to_rgb(main_color[1:])[0], to_rgb(main_color[1:])[1], to_rgb(main_color[1:])[2])\nopen('notebook.css', 'w').write(css_file % (main_color, main_color, main_color_rgba, main_color, prompt, strong_main_color, main_color, main_color, main_color, main_color))\nfrom IPython.core.display import display, HTML, Javascript\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75405d52-594f-4791-a45a-16354bc24cfa","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"775496bb-f633-46f4-a642-10e6cd29c1bb","jupyter":{"outputs_hidden":false},"papermill":{"duration":0.052444,"end_time":"2021-10-29T13:38:48.269664","exception":false,"start_time":"2021-10-29T13:38:48.21722","status":"completed"},"tags":[],"collapsed":false,"execution":{"iopub.status.busy":"2021-12-06T21:37:21.613753Z","iopub.execute_input":"2021-12-06T21:37:21.614416Z","iopub.status.idle":"2021-12-06T21:37:21.628166Z","shell.execute_reply.started":"2021-12-06T21:37:21.614373Z","shell.execute_reply":"2021-12-06T21:37:21.627392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.core.display import display, HTML, Javascript\n# def nb(): return HTML(\"\")\ndef nb(): return HTML(\"\")\nnb()","metadata":{"_cell_guid":"75d3e80e-0c31-48f5-88dd-5b6a3c79cf38","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"968efaca-ede8-4817-b3aa-baedaba3205e","jupyter":{"outputs_hidden":false},"papermill":{"duration":0.062624,"end_time":"2021-10-29T13:38:48.370222","exception":false,"start_time":"2021-10-29T13:38:48.307598","status":"completed"},"tags":[],"collapsed":false,"execution":{"iopub.status.busy":"2021-12-06T21:37:21.646505Z","iopub.execute_input":"2021-12-06T21:37:21.647026Z","iopub.status.idle":"2021-12-06T21:37:21.654368Z","shell.execute_reply.started":"2021-12-06T21:37:21.646903Z","shell.execute_reply":"2021-12-06T21:37:21.653474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Kaggle's G-Research Crypto Forecasting\n\nIn this competition, we need to forecast returns of cryptocurrency assets. Full description [here][1]. This is a very challenging time series task as seen by looking at the sample data below.\n\n[1]: https://www.kaggle.com/c/g-research-crypto-forecasting/overview","metadata":{"papermill":{"duration":0.049065,"end_time":"2021-11-16T08:32:53.065811","exception":false,"start_time":"2021-11-16T08:32:53.016746","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import gc\nimport pandas as pd\nimport numpy as np\npd.set_option(\"display.max_columns\", None)\nfrom datetime import datetime\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'font.size': 14})\ndata_folder = \"../input/g-research-crypto-forecasting/\"\n# data_folder = \"/t/Datasets/kaggle_crypto/\"\nasset_details = pd.read_csv(data_folder + 'asset_details.csv', low_memory=False)\ntrain = pd.read_csv(data_folder + 'train.csv', low_memory=False)\nrename_dict = {}\nfor a in asset_details['Asset_ID']: rename_dict[a] = asset_details[asset_details.Asset_ID == a].Asset_Name.values[0]\ntrain['timestamp'] = train['timestamp'].astype('datetime64[s]')\ntrain_daily = pd.DataFrame()\nfor asset_id in asset_details.Asset_ID:\n train_single = train[train.Asset_ID == asset_id].copy()\n train_single_new = train_single[['timestamp','Count']].resample('D', on='timestamp').sum()\n train_single_new['Open'] = train_single[['timestamp','Open']].resample('D', on='timestamp').first()['Open']\n train_single_new['High'] = train_single[['timestamp','High']].resample('D', on='timestamp').max()['High']\n train_single_new['Low'] = train_single[['timestamp','Low']].resample('D', on='timestamp').min()['Low']\n train_single_new['Close'] = train_single[['timestamp','Close']].resample('D', on='timestamp').last()['Close']\n train_single_new['Volume'] = train_single[['timestamp','Volume']].resample('D', on='timestamp').sum()['Volume']\n train_single_new['Asset_ID'] = asset_id\n train_daily = train_daily.append(train_single_new.reset_index(drop=False))\ntrain_daily = train_daily.sort_values(by = ['timestamp', 'Asset_ID']).reset_index(drop=True)\ntrain_daily = train_daily.pivot(index='timestamp', columns='Asset_ID')[['Count', 'Open', 'High', 'Low', 'Close', 'Volume']]\ntrain_daily = train_daily.reset_index(drop=False)\ntrain_daily['year'] = pd.DatetimeIndex(train_daily['timestamp']).year\nfig = make_subplots( rows=len(asset_details.Asset_ID), cols=1, subplot_titles=(asset_details.Asset_Name) )\nfor i, asset_id in enumerate(asset_details.Asset_ID):\n fig.append_trace(go.Candlestick(x=train_daily.timestamp, open=train_daily[('Open', asset_id)], high=train_daily[('High', asset_id)], low=train_daily[('Low', asset_id)], close=train_daily[('Close', asset_id)]),row=i+1, col=1,)\n fig.update_xaxes(range=[train_daily.timestamp.iloc[0], train_daily.timestamp.iloc[-1]], row=i+1, col=1)\nfig.update_layout(xaxis_rangeslider_visible = False, \n xaxis2_rangeslider_visible = False, \n xaxis3_rangeslider_visible = False,\n xaxis4_rangeslider_visible = False,\n xaxis5_rangeslider_visible = False,\n xaxis6_rangeslider_visible = False,\n xaxis7_rangeslider_visible = False,\n xaxis8_rangeslider_visible = False,\n xaxis9_rangeslider_visible = False,\n xaxis10_rangeslider_visible = False,\n xaxis11_rangeslider_visible = False,\n xaxis12_rangeslider_visible = False,\n xaxis13_rangeslider_visible = False,\n xaxis14_rangeslider_visible = False,\n height=3000, width=800, \n #title_text=\"Subplots with Annotations\"\n margin = dict(\n l = 0,\n r = 0,\n b = 0,\n t = 30,\n pad = 0)\n ) \nfig.show()\ndel train, train_daily,train_single, asset_details\ngc.collect()\nplt.close()","metadata":{"_kg_hide-input":true,"papermill":{"duration":83.51092,"end_time":"2021-11-16T08:34:16.627686","exception":false,"start_time":"2021-11-16T08:32:53.116766","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-12-06T21:37:21.89289Z","iopub.execute_input":"2021-12-06T21:37:21.893172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"papermill":{"duration":0.088608,"end_time":"2021-11-16T08:34:16.797007","exception":false,"start_time":"2021-11-16T08:34:16.708399","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df = pd.read_csv(\"../input/g-research-crypto-forecasting/\" + 'train.csv')\nassets = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nassets_names = dict(zip(assets.Asset_ID, assets.Asset_Name))\nassets_order = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv').Asset_ID[:14]\nassets_order = dict((t,i) for i,t in enumerate(assets_order))\nbtc = crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\")\nbtc_mini = btc.iloc[-200:]\n# fig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\n# fig.show()\ndel btc","metadata":{"_kg_hide-input":true,"papermill":{"duration":43.309053,"end_time":"2021-11-16T08:35:00.185889","exception":false,"start_time":"2021-11-16T08:34:16.876836","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = crypto_df.copy().set_index(\"timestamp\")\ntrain['assets'] = 1\ntrain['assets'] = train.groupby(by = train.index)['assets'].sum()\ntrain['asset_name'] = train.Asset_ID.map(assets_names)\ntrain['asset_name'].value_counts()\nall_same_time = train[train['assets'] == 14][['Asset_ID', 'Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'asset_name','VWAP','Target']]\ncorr_target = all_same_time.reset_index().pivot(index = 'asset_name', columns = 'timestamp')['Target'].transpose().corr()\ncorr_matrix = all_same_time.reset_index().drop(['Target', 'Asset_ID'], axis=1).pivot(index = 'asset_name', columns = 'timestamp').transpose().corr()\nfig, ax = plt.subplots(1, 2, figsize = (20, 8))\nsns.heatmap(np.round(corr_target, 2), annot = True, ax = ax[0], square = True)\nsns.heatmap(np.round(corr_matrix, 2), annot = True, ax = ax[1], square = True)\nax[0].title.set_text('Asset Targets correlation')\nax[1].title.set_text('Asset Features correlation')","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"papermill":{"duration":58.6563,"end_time":"2021-11-16T08:35:58.921091","exception":false,"start_time":"2021-11-16T08:35:00.264791","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try: plt.close()\nexcept: pass\nfig = make_subplots(rows=7, cols=2,shared_xaxes=True, vertical_spacing=0.03, subplot_titles=tuple([assets_names[i] for i in range(14)]))\ndata = all_same_time[1000:2400]\ndata['time'] = [pd.to_datetime(x, unit='s') for x in data.index]\nfor i in range(14): \n coin = data[data.Asset_ID == i]\n name = assets_names[i]\n fig.add_trace(go.Scatter(x=coin['time'], y=coin['VWAP'], name = name + ', VWAP'),row=i//2+1, col= i%2 +1)\nfig.update_layout(height=1000, title_text=' Weighted average prices')\nfig.show()\ndel data, train, crypto_df, all_same_time, corr_target","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"papermill":{"duration":0.779451,"end_time":"2021-11-16T08:35:59.788228","exception":false,"start_time":"2021-11-16T08:35:59.008777","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Libraries π\n","metadata":{"_cell_guid":"347324aa-490a-4556-bcc4-36379bdbec7c","_uuid":"03824ce3-fc08-4d19-926d-27edbb773466","papermill":{"duration":0.092916,"end_time":"2021-11-16T08:35:59.975497","exception":false,"start_time":"2021-11-16T08:35:59.882581","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"#### Code starts here β¬","metadata":{"_cell_guid":"f5bdcf7b-aa74-4204-9ff2-f20042b819e0","_uuid":"78d84509-539f-4a94-a546-e228b28ea70a","papermill":{"duration":0.093823,"end_time":"2021-11-16T08:36:00.164648","exception":false,"start_time":"2021-11-16T08:36:00.070825","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport gresearch_crypto\nfrom lightgbm import LGBMRegressor\n\nTRAIN_CSV = '/kaggle/input/g-research-crypto-forecasting/train.csv'\nASSET_DETAILS_CSV = '/kaggle/input/g-research-crypto-forecasting/asset_details.csv'","metadata":{"papermill":{"duration":1.912046,"end_time":"2021-11-05T07:01:11.946788","exception":false,"start_time":"2021-11-05T07:01:10.034742","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = pd.read_csv(TRAIN_CSV)\ndf_train.head()","metadata":{"papermill":{"duration":60.150258,"end_time":"2021-11-05T07:02:12.109237","exception":false,"start_time":"2021-11-05T07:01:11.958979","status":"completed"},"scrolled":true,"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_asset_details = pd.read_csv(ASSET_DETAILS_CSV).sort_values(\"Asset_ID\")\ndf_asset_details","metadata":{"papermill":{"duration":0.042168,"end_time":"2021-11-05T07:02:12.163047","exception":false,"start_time":"2021-11-05T07:02:12.120879","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training ποΈ\n","metadata":{"papermill":{"duration":0.012008,"end_time":"2021-11-05T07:02:12.187737","exception":false,"start_time":"2021-11-05T07:02:12.175729","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Utility functions to train a model for one asset","metadata":{"papermill":{"duration":0.011958,"end_time":"2021-11-05T07:02:12.21184","exception":false,"start_time":"2021-11-05T07:02:12.199882","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from catboost import Pool, CatBoostClassifier, cv, CatBoostRegressor\n\ndef upper_shadow(df): return df['High'] - np.maximum(df['Close'], df['Open'])\ndef lower_shadow(df): return np.minimum(df['Close'], df['Open']) - df['Low']\n\ndef get_features(df,row=False):\n df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP', \"timestamp\"]].copy()\n df_feat['Upper_Shadow'] = upper_shadow(df_feat)\n df_feat['Lower_Shadow'] = lower_shadow(df_feat)\n \n ## Add some more feats\n df_feat[\"high_div_low\"] = df_feat[\"High\"]/df_feat[\"Low\"]\n df_feat[\"open_sub_close\"] = df_feat[\"Open\"]-df_feat[\"Close\"]\n\n ## possible seasonality, datetime features (unlikely to me meaningful, given very short time-frames)\n times = pd.to_datetime(df_feat[\"timestamp\"],unit=\"s\",infer_datetime_format=True)\n if row:\n df_feat[\"hour\"] = times.hour\n df_feat[\"dayofweek\"] = times.dayofweek \n df_feat[\"day\"] = times.day \n else:\n df_feat[\"hour\"] = times.dt.hour\n df_feat[\"dayofweek\"] = times.dt.dayofweek \n df_feat[\"day\"] = times.dt.day \n df_feat = df_feat.drop(columns = \"timestamp\")\n return df_feat\n\n\ndef get_Xy_and_model_for_asset(df_train, asset_id):\n df = df_train[df_train[\"Asset_ID\"] == asset_id]\n \n df_proc = get_features(df)\n df_proc['y'] = df['Target']\n df_proc = df_proc.dropna(how=\"any\")\n \n X = df_proc.drop(\"y\", axis=1)\n y = df_proc[\"y\"]\n \n model = CatBoostRegressor(iterations=1000, learning_rate=0.05, depth=10, random_seed=42)\n model.fit(X, y)\n\n return X, y, model","metadata":{"papermill":{"duration":0.279296,"end_time":"2021-11-05T07:02:12.50337","exception":false,"start_time":"2021-11-05T07:02:12.224074","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loop over all assets","metadata":{"papermill":{"duration":0.012351,"end_time":"2021-11-05T07:02:12.528095","exception":false,"start_time":"2021-11-05T07:02:12.515744","status":"completed"},"tags":[]}},{"cell_type":"code","source":"Xs = {}\nys = {}\nmodels = {}\n\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n X, y, model = get_Xy_and_model_for_asset(df_train, asset_id) \n Xs[asset_id], ys[asset_id], models[asset_id] = X, y, model","metadata":{"papermill":{"duration":4268.026719,"end_time":"2021-11-05T08:13:20.566987","exception":false,"start_time":"2021-11-05T07:02:12.540268","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the model interface\nx = get_features(df_train.iloc[1],row=True)\ny_pred = models[0].predict([x])\ny_pred[0]","metadata":{"papermill":{"duration":4.042371,"end_time":"2021-11-05T08:13:28.632104","exception":false,"start_time":"2021-11-05T08:13:24.589733","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit To Kaggle π°\n\n\nReferences: [Detailed API Introduction](https://www.kaggle.com/sohier/detailed-api-introduction)\n\nSomething that helped me understand this iterator was adding a pdb checkpoint inside of the for loop:\n\n```python\nimport pdb; pdb.set_trace()\n```\n\nSee [Python Debugging With Pdb](https://realpython.com/python-debugging-pdb/) if you want to use it and you don't know how to.\n","metadata":{"execution":{"iopub.status.busy":"2021-11-02T20:57:49.349459Z","iopub.status.idle":"2021-11-02T20:57:49.349757Z","shell.execute_reply":"2021-11-02T20:57:49.349613Z","shell.execute_reply.started":"2021-11-02T20:57:49.349596Z"},"papermill":{"duration":4.04781,"end_time":"2021-11-05T08:13:36.629986","exception":false,"start_time":"2021-11-05T08:13:32.582176","status":"completed"},"tags":[]}},{"cell_type":"code","source":"env = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n for j , row in df_test.iterrows():\n \n model = models[row['Asset_ID']]\n x_test = get_features(row,row=True)\n y_pred = model.predict([x_test])[0]\n \n df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n \n \n # Print just one sample row to get a feeling of what it looks like\n if i == 0 and j == 0:\n display(x_test)\n\n # Display the first prediction dataframe\n if i == 0:\n display(df_pred)\n\n # Send submissions\n env.predict(df_pred)","metadata":{"papermill":{"duration":4.436258,"end_time":"2021-11-05T08:13:45.053529","exception":false,"start_time":"2021-11-05T08:13:40.617271","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"papermill":{"duration":4.026799,"end_time":"2021-11-05T08:13:53.099036","exception":false,"start_time":"2021-11-05T08:13:49.072237","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]}]}
![](\"https://i.ibb.co/hHpTy3c/g-research-logo6.png\"){kind=logo}
![](\"https://i.ibb.co/9YFyhT8/Bitcoin2.gif\")
\n