{"nbformat_minor":4,"nbformat":4,"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-21T05:10:11.508281Z","iopub.execute_input":"2022-01-21T05:10:11.50864Z","iopub.status.idle":"2022-01-21T05:10:11.541457Z","shell.execute_reply.started":"2022-01-21T05:10:11.508555Z","shell.execute_reply":"2022-01-21T05:10:11.540667Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# # LightGBM for prediction\n\nLightGBM is a version of gradient boosting developed and maintained by Microsoft, it has become a goto algorithm in kaggle competitions . LightGBM algorithm has become bigger challenger along with CATboost algorithm to traditional dominate XBBoost Algorithm in gaggle competitions. \n\n\nLightGBM Advantages\n* Faster training speed and higher efficiency.\n* Lower memory usage.\n* Better accuracy.\n* Support of parallel and GPU learning.\n* Capable of handling large-scale data.\n\nLightGBM tree tends to grows leaf-wise  which will converge faster than depth wise ones trees. But they can be more prone to overfitting.\n\nOne significant advantage of LightGBM algorithm is the way it works seamlessly with categorical features, in general for categorical features we do one-hot encoding of the features which will make unbalanced trees which can impact the accuracy. With LightGBM algorithm we have categorical_feature attribute, can specify categorical features (without one-hot encoding) for the model. It will enable Categorical features to be encoded as non-negative integers less than Int32.MaxValue starting form zero. ","metadata":{}},{"cell_type":"markdown","source":"# # **Data Description**\n\n**train.csv**\n\ntimestamp: All timestamps are returned as second Unix timestamps (the number of seconds elapsed since 1970-01-01 00:00:00.000 UTC). Timestamps in this dataset are multiple of 60, indicating minute-by-minute data.\n\nAsset_ID: The asset ID corresponding to one of the crytocurrencies (e.g. Asset_ID = 1 for Bitcoin). The mapping from Asset_ID to crypto asset is contained in asset_details.csv.\n\nCount: Total number of trades in the time interval (last minute).\n\nOpen: Opening price of the time interval (in USD).\n\nHigh: Highest price reached during time interval (in USD).\n\nLow: Lowest price reached during time interval (in USD).\n\nClose: Closing price of the time interval (in USD).\n\nVolume: Quantity of asset bought or sold, displayed in base currency USD.\n\nVWAP: The average price of the asset over the time interval, weighted by volume. VWAP is an aggregated form of trade data.\n\nTarget: Residual log-returns for the asset over a 15 minute horizon.\nsupplemental_train.csv After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n\nasset_details.csv Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric. Weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact the models.\n\nexample_sample_submission.csv An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n\nexample_test.csv An example of the data that will be delivered by the time series API.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom datetime import datetime\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom lightgbm import LGBMRegressor\nfrom sklearn.model_selection import GridSearchCV\n","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:10:11.545418Z","iopub.execute_input":"2022-01-21T05:10:11.545597Z","iopub.status.idle":"2022-01-21T05:10:14.654133Z","shell.execute_reply.started":"2022-01-21T05:10:11.545574Z","shell.execute_reply":"2022-01-21T05:10:14.653336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### Loading datasets\n\npath = \"/kaggle/input/g-research-crypto-forecasting/\"\ndf_train = pd.read_csv(path + \"train.csv\")\ndf_test = pd.read_csv(path + \"example_test.csv\")\ndf_asset_details = pd.read_csv(path + \"asset_details.csv\")\ndf_supp_train = pd.read_csv(path + \"supplemental_train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:10:14.655732Z","iopub.execute_input":"2022-01-21T05:10:14.65629Z","iopub.status.idle":"2022-01-21T05:11:13.359975Z","shell.execute_reply.started":"2022-01-21T05:10:14.656252Z","shell.execute_reply":"2022-01-21T05:11:13.359155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:13.361502Z","iopub.execute_input":"2022-01-21T05:11:13.361751Z","iopub.status.idle":"2022-01-21T05:11:13.385514Z","shell.execute_reply.started":"2022-01-21T05:11:13.361717Z","shell.execute_reply":"2022-01-21T05:11:13.384825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#df_asset_details.head().sort()\ndf_asset_details.sort_values(by='Asset_ID')","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:13.387606Z","iopub.execute_input":"2022-01-21T05:11:13.387855Z","iopub.status.idle":"2022-01-21T05:11:13.404001Z","shell.execute_reply.started":"2022-01-21T05:11:13.387821Z","shell.execute_reply":"2022-01-21T05:11:13.402976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Summary Statistics of overall coins.\ndf_train.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:13.405675Z","iopub.execute_input":"2022-01-21T05:11:13.406074Z","iopub.status.idle":"2022-01-21T05:11:21.327421Z","shell.execute_reply.started":"2022-01-21T05:11:13.406038Z","shell.execute_reply":"2022-01-21T05:11:21.32674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:21.328738Z","iopub.execute_input":"2022-01-21T05:11:21.329091Z","iopub.status.idle":"2022-01-21T05:11:21.743194Z","shell.execute_reply.started":"2022-01-21T05:11:21.329047Z","shell.execute_reply":"2022-01-21T05:11:21.742372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define plot space\nfig, ax = plt.subplots(figsize=(10, 6))\n\n# Create bar plot\nax.bar(df_asset_details['Asset_Name'], \n       df_asset_details['Weight'])\nplt.xlabel('Asset Name')\nplt.ylabel('Weight')\nplt.xticks(rotation=70)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:21.744503Z","iopub.execute_input":"2022-01-21T05:11:21.744824Z","iopub.status.idle":"2022-01-21T05:11:22.003247Z","shell.execute_reply.started":"2022-01-21T05:11:21.744788Z","shell.execute_reply":"2022-01-21T05:11:22.002598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#setting date format\nds_train_copy = df_train\nds_train_copy['date'] = pd.to_datetime(ds_train_copy['timestamp'], unit='s')","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:22.004622Z","iopub.execute_input":"2022-01-21T05:11:22.004872Z","iopub.status.idle":"2022-01-21T05:11:23.520126Z","shell.execute_reply.started":"2022-01-21T05:11:22.004838Z","shell.execute_reply":"2022-01-21T05:11:23.519403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bnc = ds_train_copy[ds_train_copy['Asset_ID']==0].set_index('timestamp')\nbtc = ds_train_copy[ds_train_copy['Asset_ID']==1].set_index('timestamp')\nbtcsh = ds_train_copy[ds_train_copy['Asset_ID']==2].set_index('timestamp')\ncar = ds_train_copy[ds_train_copy['Asset_ID']==3].set_index('timestamp')\ndog = ds_train_copy[ds_train_copy['Asset_ID']==4].set_index('timestamp')\neos = ds_train_copy[ds_train_copy['Asset_ID']==5].set_index('timestamp')\neth = ds_train_copy[ds_train_copy['Asset_ID']==6].set_index('timestamp')\neth_csc = ds_train_copy[ds_train_copy['Asset_ID']==7].set_index('timestamp')\niot = ds_train_copy[ds_train_copy['Asset_ID']==8].set_index('timestamp')\nltc = ds_train_copy[ds_train_copy['Asset_ID']==9].set_index('timestamp')\nmak = ds_train_copy[ds_train_copy['Asset_ID']==10].set_index('timestamp')\nmon = ds_train_copy[ds_train_copy['Asset_ID']==11].set_index('timestamp')\nste = ds_train_copy[ds_train_copy['Asset_ID']==12].set_index('timestamp')\ntro = ds_train_copy[ds_train_copy['Asset_ID']==13].set_index('timestamp')\neth.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:23.521496Z","iopub.execute_input":"2022-01-21T05:11:23.521733Z","iopub.status.idle":"2022-01-21T05:11:27.859847Z","shell.execute_reply.started":"2022-01-21T05:11:23.521701Z","shell.execute_reply":"2022-01-21T05:11:27.859113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check the train dataset start and end for each asset","metadata":{}},{"cell_type":"code","source":"beg_btcsh = btcsh.index[0].astype('datetime64[s]')\nend_btcsh = btcsh.index[-1].astype('datetime64[s]')\nbeg_bnc = bnc.index[0].astype('datetime64[s]')\nend_bnc = bnc.index[-1].astype('datetime64[s]')\nbeg_btc = btc.index[0].astype('datetime64[s]')\nend_btc = btc.index[-1].astype('datetime64[s]')\nbeg_eos = eos.index[0].astype('datetime64[s]')\nend_eos = eos.index[-1].astype('datetime64[s]')\nbeg_eth_csc = eth_csc.index[0].astype('datetime64[s]')\nend_eth_csc = eth_csc.index[-1].astype('datetime64[s]')\nbeg_ltc = ltc.index[0].astype('datetime64[s]')\nend_ltc = ltc.index[-1].astype('datetime64[s]')\nbeg_mon = mon.index[0].astype('datetime64[s]')\nend_mon = mon.index[-1].astype('datetime64[s]')\nbeg_tro = tro.index[0].astype('datetime64[s]')\nend_tro = tro.index[-1].astype('datetime64[s]')\nbeg_eth = eth.index[0].astype('datetime64[s]')\nend_eth = eth.index[-1].astype('datetime64[s]')\nbeg_ste = ste.index[0].astype('datetime64[s]')\nend_ste = ste.index[-1].astype('datetime64[s]')\nbeg_car = car.index[0].astype('datetime64[s]')\nend_car = car.index[-1].astype('datetime64[s]')\nbeg_iot = iot.index[0].astype('datetime64[s]')\nend_iot = iot.index[-1].astype('datetime64[s]')\nbeg_mak = mak.index[0].astype('datetime64[s]')\nend_mak = mak.index[-1].astype('datetime64[s]')\nbeg_dog = dog.index[0].astype('datetime64[s]')\nend_dog = dog.index[-1].astype('datetime64[s]')\nprint('Bitcoin Cash     :', beg_btcsh, 'to', end_btcsh)\nprint('Binance Coin     :', beg_bnc, 'to', end_bnc)\nprint('Bitcoin          :', beg_btc, 'to', end_btc)\nprint('EOS IO           :', beg_eos, 'to', end_eos)\nprint('Etherium Classic :', beg_eth_csc, 'to', end_eth_csc)\nprint('Ethereum         :', beg_eth, 'to', end_eth)\nprint('Lite Coin        :', beg_ltc, 'to', end_ltc)\nprint('Monero           :', beg_mon, 'to', end_mon)\nprint('TRON             :', beg_tro, 'to', end_tro)\nprint('Stellar          :', beg_ste, 'to', end_ste)\nprint('Cardano          :', beg_car, 'to', end_car)\nprint('IOTA             :', beg_iot, 'to', end_iot)\nprint('Maker            :', beg_mak, 'to', end_mak)\nprint('Dogecoin         :', beg_dog, 'to', end_dog)","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:27.863204Z","iopub.execute_input":"2022-01-21T05:11:27.863443Z","iopub.status.idle":"2022-01-21T05:11:27.891796Z","shell.execute_reply.started":"2022-01-21T05:11:27.863408Z","shell.execute_reply":"2022-01-21T05:11:27.890844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc=df_train[df_train['Asset_ID']==1].set_index('timestamp')\nbtc_mini = btc.iloc[-200:]\n","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:27.89331Z","iopub.execute_input":"2022-01-21T05:11:27.893565Z","iopub.status.idle":"2022-01-21T05:11:28.232075Z","shell.execute_reply.started":"2022-01-21T05:11:27.89353Z","shell.execute_reply":"2022-01-21T05:11:28.231325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check the missing timestamp for each data","metadata":{}},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:28.23317Z","iopub.execute_input":"2022-01-21T05:11:28.233399Z","iopub.status.idle":"2022-01-21T05:11:28.257331Z","shell.execute_reply.started":"2022-01-21T05:11:28.233368Z","shell.execute_reply":"2022-01-21T05:11:28.256535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reindex all asset to remove the missing data in each timestamp","metadata":{}},{"cell_type":"code","source":"eth     = eth.reindex(range(eth.index[0],eth.index[-1]+60,60),method='pad')\nbtc     = btc.reindex(range(btc.index[0],btc.index[-1]+60,60),method='pad')\nbtcsh   = btcsh.reindex(range(btcsh.index[0],btcsh.index[-1]+60,60),method='pad')\nbnc     = bnc.reindex(range(bnc.index[0],bnc.index[-1]+60,60),method='pad')\neos     = eos.reindex(range(eos.index[0],eos.index[-1]+60,60),method='pad')\neth_csc = eth_csc.reindex(range(eth_csc.index[0],eth_csc.index[-1]+60,60),method='pad')\nltc     = ltc.reindex(range(ltc.index[0],ltc.index[-1]+60,60),method='pad')\nmon     = mon.reindex(range(mon.index[0],mon.index[-1]+60,60),method='pad')\ntro     = tro.reindex(range(tro.index[0],tro.index[-1]+60,60),method='pad')\nste     = ste.reindex(range(ste.index[0],ste.index[-1]+60,60),method='pad')\ncar     = car.reindex(range(car.index[0],car.index[-1]+60,60),method='pad')\niot     = iot.reindex(range(iot.index[0],iot.index[-1]+60,60),method='pad')\nmak     = mak.reindex(range(mak.index[0],mak.index[-1]+60,60),method='pad')\ndog     = dog.reindex(range(dog.index[0],dog.index[-1]+60,60),method='pad')","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:28.258581Z","iopub.execute_input":"2022-01-21T05:11:28.25892Z","iopub.status.idle":"2022-01-21T05:11:31.590052Z","shell.execute_reply.started":"2022-01-21T05:11:28.258871Z","shell.execute_reply":"2022-01-21T05:11:31.589285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Show all close value for each asset from beginning until end","metadata":{}},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:31.591352Z","iopub.execute_input":"2022-01-21T05:11:31.591604Z","iopub.status.idle":"2022-01-21T05:11:31.623078Z","shell.execute_reply.started":"2022-01-21T05:11:31.591559Z","shell.execute_reply":"2022-01-21T05:11:31.622232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define plot space\nfig, ax = plt.subplots(5, 3, figsize=(18, 22))\n\n# Bitcoin Cash\nax[0, 0].plot(btcsh['Close'], label='BTCSH')\nax[0, 0].set_title('Bitcoin Cash')\nax[0, 1].plot(bnc['Close'], label='BNC')\nax[0, 1].set_title('Binance Coin')\nax[0, 2].plot(btc['Close'], label='BTC')\nax[0, 2].set_title('Bitcoin')\nax[1, 0].plot(eos['Close'], label='EOS')\nax[1, 0].set_title('EOS.IO')\nax[1, 1].plot(eth_csc['Close'], label='ETH_CSC')\nax[1, 1].set_title('Etherium Cash')\nax[1, 2].plot(ltc['Close'], label='LTC')\nax[1, 2].set_title('Lite Coin')\nax[2, 0].plot(mon['Close'], label='MON')\nax[2, 0].set_title('Monero')\nax[2, 1].plot(tro['Close'], label='TRO')\nax[2, 1].set_title('TRON')\nax[2, 2].plot(eth['Close'], label='ETH')\nax[2, 2].set_title('Etherium')\nax[3, 0].plot(ste['Close'], label='STE')\nax[3, 0].set_title('Stelar')\nax[3, 1].plot(car['Close'], label='CAR')\nax[3, 1].set_title('Cardano')\nax[3, 2].plot(iot['Close'], label='IOT')\nax[3, 2].set_title('IOTA')\nax[4, 0].plot(mak['Close'], label='MAK')\nax[4, 0].set_title('Maker')\nax[4, 1].plot(dog['Close'], label='DOG')\nax[4, 1].set_title('Dogecoin')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:31.62447Z","iopub.execute_input":"2022-01-21T05:11:31.624733Z","iopub.status.idle":"2022-01-21T05:11:36.174938Z","shell.execute_reply.started":"2022-01-21T05:11:31.624699Z","shell.execute_reply":"2022-01-21T05:11:36.174232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Bit coin candle stick diagram\nimport plotly.graph_objects as go\n\nfig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:36.175988Z","iopub.execute_input":"2022-01-21T05:11:36.176242Z","iopub.status.idle":"2022-01-21T05:11:36.267006Z","shell.execute_reply.started":"2022-01-21T05:11:36.176203Z","shell.execute_reply":"2022-01-21T05:11:36.266202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:36.268457Z","iopub.execute_input":"2022-01-21T05:11:36.268708Z","iopub.status.idle":"2022-01-21T05:11:36.724767Z","shell.execute_reply.started":"2022-01-21T05:11:36.268673Z","shell.execute_reply":"2022-01-21T05:11:36.723935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth.tail()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:36.726108Z","iopub.execute_input":"2022-01-21T05:11:36.726433Z","iopub.status.idle":"2022-01-21T05:11:36.746221Z","shell.execute_reply.started":"2022-01-21T05:11:36.726394Z","shell.execute_reply":"2022-01-21T05:11:36.745407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:36.74766Z","iopub.execute_input":"2022-01-21T05:11:36.748101Z","iopub.status.idle":"2022-01-21T05:11:36.785467Z","shell.execute_reply.started":"2022-01-21T05:11:36.748061Z","shell.execute_reply":"2022-01-21T05:11:36.784737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# plot vwap time series for both chosen assets\nf = plt.figure(figsize=(15,4))\n\n# fill missing values for BTC\nbtc = btc.reindex(range(btc.index[0],btc.index[-1]+60,60),method='pad')\n\nax = f.add_subplot(121)\nplt.plot(btc['Close'], label='BTC')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin')\n\nax2 = f.add_subplot(122)\nax2.plot(eth['Close'], color='brown', label='ETH')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Ethereum')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:36.786582Z","iopub.execute_input":"2022-01-21T05:11:36.787201Z","iopub.status.idle":"2022-01-21T05:11:39.291377Z","shell.execute_reply.started":"2022-01-21T05:11:36.787166Z","shell.execute_reply":"2022-01-21T05:11:39.290704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Coin Correlation for last 12000 Minutes","metadata":{}},{"cell_type":"code","source":"data =df_train[-12000:]\ncheck = pd.DataFrame()\nfor i in data.Asset_ID.unique():\n    check[i] = data[data.Asset_ID==i]['Target'].reset_index(drop=True) \n    \nplt.figure(figsize=(10,8))\nsns.heatmap(check.dropna().corr(), vmin=-1.0, vmax=1.0, annot=True, cmap='coolwarm', linewidths=0.1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:39.292604Z","iopub.execute_input":"2022-01-21T05:11:39.293132Z","iopub.status.idle":"2022-01-21T05:11:40.252617Z","shell.execute_reply.started":"2022-01-21T05:11:39.293095Z","shell.execute_reply":"2022-01-21T05:11:40.251933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# creating the ratio\n\ndef hlco_ratio(df): \n    return (df['High'] - df['Low'])/(df['Close']-df['Open'])\ndef upper_shadow(df):\n    return df['High'] - np.maximum(df['Close'], df['Open'])\ndef lower_shadow(df):\n    return np.minimum(df['Close'], df['Open']) - df['Low']\n\ndef get_features(df):\n    df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n    df_feat['Upper_Shadow'] = upper_shadow(df_feat)\n    df_feat['hlco_ratio'] = hlco_ratio(df_feat)\n    df_feat['Lower_Shadow'] = lower_shadow(df_feat)\n    return df_feat","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:40.25383Z","iopub.execute_input":"2022-01-21T05:11:40.254249Z","iopub.status.idle":"2022-01-21T05:11:40.260849Z","shell.execute_reply.started":"2022-01-21T05:11:40.25421Z","shell.execute_reply":"2022-01-21T05:11:40.260075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train test split df_train into 80% train rows and 20% valid rows\ntrain_data = df_train\n# train_data = df_train.sample(frac = 0.8)\n# valid_data = df_train.drop(train_data.index)\n\ndef get_Xy_and_model_for_asset(df_train, asset_id):\n    df = df_train[df_train[\"Asset_ID\"] == asset_id]\n    \n    df = df.sample(frac=0.2)\n    df_proc = get_features(df)\n    df_proc['y'] = df['Target']\n    df_proc.replace([np.inf, -np.inf], np.nan, inplace=True)\n    df_proc = df_proc.dropna(how=\"any\")\n    \n    \n    X = df_proc.drop(\"y\", axis=1)\n    y = df_proc[\"y\"]   \n    model = LGBMRegressor()\n    model.fit(X, y)\n    return X, y, model\n\nXs = {}\nys = {}\nmodels = {}\n\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n    X, y, model = get_Xy_and_model_for_asset(train_data, asset_id)       \n    try:\n        Xs[asset_id], ys[asset_id], models[asset_id] = X, y, model\n    except: \n        Xs[asset_id], ys[asset_id], models[asset_id] = None, None, None ","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:11:40.262082Z","iopub.execute_input":"2022-01-21T05:11:40.262521Z","iopub.status.idle":"2022-01-21T05:12:04.705262Z","shell.execute_reply.started":"2022-01-21T05:11:40.262469Z","shell.execute_reply":"2022-01-21T05:12:04.704497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nparameters = {\n    # 'max_depth': range (2, 10, 1),\n    'num_leaves': range(21, 161, 10),\n    'learning_rate': [0.1, 0.01, 0.05]\n}\n\nnew_models = {}\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(\"GridSearchCV for: \" + asset_name)\n    grid_search = GridSearchCV(\n        estimator=get_Xy_and_model_for_asset(df_train, asset_id)[2], # bitcoin\n        param_grid=parameters,\n        n_jobs = -1,\n        cv = 5,\n        verbose=True\n    )\n    grid_search.fit(Xs[asset_id], ys[asset_id])\n    new_models[asset_id] = grid_search.best_estimator_\n    grid_search.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2022-01-21T05:12:04.709668Z","iopub.execute_input":"2022-01-21T05:12:04.711192Z","iopub.status.idle":"2022-01-21T06:25:27.381501Z","shell.execute_reply.started":"2022-01-21T05:12:04.71116Z","shell.execute_reply":"2022-01-21T06:25:27.38082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(f\"Tuned model for {asset_name:<1} (ID={asset_id:})\")\n    print(new_models[asset_id])","metadata":{"execution":{"iopub.status.busy":"2022-01-21T06:25:27.385252Z","iopub.execute_input":"2022-01-21T06:25:27.387133Z","iopub.status.idle":"2022-01-21T06:25:27.405673Z","shell.execute_reply.started":"2022-01-21T06:25:27.387089Z","shell.execute_reply":"2022-01-21T06:25:27.405019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gresearch_crypto\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    for j , row in df_test.iterrows():        \n        if new_models[row['Asset_ID']] is not None:\n            try:\n                model = new_models[row['Asset_ID']]\n                x_test = get_features(row)\n                y_pred = model.predict(pd.DataFrame([x_test]))[0]\n                df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n            except:\n                df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n                traceback.print_exc()\n        else: \n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0  \n    \n    env.predict(df_pred)","metadata":{"execution":{"iopub.status.busy":"2022-01-21T06:33:12.935232Z","iopub.execute_input":"2022-01-21T06:33:12.935558Z","iopub.status.idle":"2022-01-21T06:33:13.24408Z","shell.execute_reply.started":"2022-01-21T06:33:12.935525Z","shell.execute_reply":"2022-01-21T06:33:13.243488Z"},"trusted":true},"execution_count":null,"outputs":[]}]}