{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **LBGM**\n* LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed and efficient with the following advantages:\n* Faster training speed and higher efficiency (6 times faster than XGBoost)\n* Lower memory usage.\n* Better accuracy.\n* Support of parallel, distributed, and GPU learning.\n* Capable of handling large-scale data.","metadata":{}},{"cell_type":"markdown","source":"\n### **<span>Dataset Structure</span>**\n\n> **train.csv** - The training set\n> \n> 1.  timestamp - A timestamp for the minute covered by the row.\n> 2.  Asset_ID - An ID code for the cryptoasset.\n> 3.  Count - The number of trades that took place this minute.\n> 4.  Open - The USD price at the beginning of the minute.\n> 5.  High - The highest USD price during the minute.\n> 6.  Low - The lowest USD price during the minute.\n> 7.  Close - The USD price at the end of the minute.\n> 8.  Volume - The number of cryptoasset u units traded during the minute.\n> 9.  VWAP - The volume-weighted average price for the minute.\n> 10. Target - 15 minute residualized returns. See the 'Prediction and Evaluation section of this notebook for details of how the target is calculated.\n> 11. Weight - Weight, defined by the competition hosts [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n> 12. Asset_Name - Human readable Asset name.\n> \n>\n> **example_test.csv** - An example of the data that will be delivered by the time series API.\n> \n> **example_sample_submission.csv** - An example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n> \n> **asset_details.csv** - Provides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric.\n> \n> **supplemental_train.csv** - After the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n>\n> - 📌 There are 14 coins in the dataset\n>\n> - 📌 There are 4 years  in the [full] dataset","metadata":{}},{"cell_type":"markdown","source":"# **Importing Libraries**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom datetime import datetime\nfrom lightgbm import LGBMRegressor\nimport gresearch_crypto\nimport traceback\nimport time\nfrom datetime import datetime\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nfrom sklearn.model_selection import GridSearchCV\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:24:46.098397Z","iopub.execute_input":"2022-01-15T02:24:46.098767Z","iopub.status.idle":"2022-01-15T02:24:48.114118Z","shell.execute_reply.started":"2022-01-15T02:24:46.098676Z","shell.execute_reply":"2022-01-15T02:24:48.113273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Importing Data**","metadata":{}},{"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-15T02:25:03.200151Z","iopub.execute_input":"2022-01-15T02:25:03.200419Z","iopub.status.idle":"2022-01-15T02:25:03.210271Z","shell.execute_reply.started":"2022-01-15T02:25:03.200389Z","shell.execute_reply":"2022-01-15T02:25:03.209370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Loading Data**","metadata":{}},{"cell_type":"code","source":"path = \"/kaggle/input/g-research-crypto-forecasting/\"\ndf_train = pd.read_csv(path + \"train.csv\")\ndf_test = pd.read_csv(path + \"example_test.csv\")\ndf_asset_details = pd.read_csv(path + \"asset_details.csv\")\ndf_supp_train = pd.read_csv(path + \"supplemental_train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:25:25.843591Z","iopub.execute_input":"2022-01-15T02:25:25.843865Z","iopub.status.idle":"2022-01-15T02:26:33.534865Z","shell.execute_reply.started":"2022-01-15T02:25:25.843835Z","shell.execute_reply":"2022-01-15T02:26:33.533830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking Train data \ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:26:33.537009Z","iopub.execute_input":"2022-01-15T02:26:33.537265Z","iopub.status.idle":"2022-01-15T02:26:33.567333Z","shell.execute_reply.started":"2022-01-15T02:26:33.537219Z","shell.execute_reply":"2022-01-15T02:26:33.566537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.shape","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:26:33.568366Z","iopub.execute_input":"2022-01-15T02:26:33.568595Z","iopub.status.idle":"2022-01-15T02:26:33.574579Z","shell.execute_reply.started":"2022-01-15T02:26:33.568569Z","shell.execute_reply":"2022-01-15T02:26:33.573592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking Test data set\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:26:33.576523Z","iopub.execute_input":"2022-01-15T02:26:33.577189Z","iopub.status.idle":"2022-01-15T02:26:33.595849Z","shell.execute_reply.started":"2022-01-15T02:26:33.577159Z","shell.execute_reply":"2022-01-15T02:26:33.594905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.shape","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:26:33.597170Z","iopub.execute_input":"2022-01-15T02:26:33.597380Z","iopub.status.idle":"2022-01-15T02:26:33.607148Z","shell.execute_reply.started":"2022-01-15T02:26:33.597355Z","shell.execute_reply":"2022-01-15T02:26:33.606360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* By visualising Train and Test data set (train.csv) except contains less rows.","metadata":{}},{"cell_type":"markdown","source":"### **EDA**\n* Let's continue describing and analyzing the data ","metadata":{}},{"cell_type":"code","source":"df_train.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:26:53.320852Z","iopub.execute_input":"2022-01-15T02:26:53.321151Z","iopub.status.idle":"2022-01-15T02:27:01.246654Z","shell.execute_reply.started":"2022-01-15T02:26:53.321122Z","shell.execute_reply":"2022-01-15T02:27:01.245811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking Test data sets\ndf_test.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:01.248250Z","iopub.execute_input":"2022-01-15T02:27:01.248449Z","iopub.status.idle":"2022-01-15T02:27:01.287834Z","shell.execute_reply.started":"2022-01-15T02:27:01.248425Z","shell.execute_reply":"2022-01-15T02:27:01.287053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:01.289054Z","iopub.execute_input":"2022-01-15T02:27:01.289256Z","iopub.status.idle":"2022-01-15T02:27:01.709120Z","shell.execute_reply.started":"2022-01-15T02:27:01.289232Z","shell.execute_reply":"2022-01-15T02:27:01.708409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:01.710673Z","iopub.execute_input":"2022-01-15T02:27:01.710873Z","iopub.status.idle":"2022-01-15T02:27:01.717929Z","shell.execute_reply.started":"2022-01-15T02:27:01.710848Z","shell.execute_reply":"2022-01-15T02:27:01.717024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* seems like the VWAP column has NaN values we will check it later","metadata":{}},{"cell_type":"code","source":"df_train.info()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:01.719310Z","iopub.execute_input":"2022-01-15T02:27:01.719719Z","iopub.status.idle":"2022-01-15T02:27:01.738142Z","shell.execute_reply.started":"2022-01-15T02:27:01.719664Z","shell.execute_reply":"2022-01-15T02:27:01.737335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.info()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:03.278819Z","iopub.execute_input":"2022-01-15T02:27:03.279066Z","iopub.status.idle":"2022-01-15T02:27:03.291190Z","shell.execute_reply.started":"2022-01-15T02:27:03.279040Z","shell.execute_reply":"2022-01-15T02:27:03.290376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* By checking train data and test data there is some missing values in train data set","metadata":{}},{"cell_type":"markdown","source":"## Timestamp\n* We define a helper function that will turn a date format into a timestamp to use for indexing.","metadata":{}},{"cell_type":"code","source":"df_asset_details","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:09.570681Z","iopub.execute_input":"2022-01-15T02:27:09.571353Z","iopub.status.idle":"2022-01-15T02:27:09.582419Z","shell.execute_reply.started":"2022-01-15T02:27:09.571315Z","shell.execute_reply":"2022-01-15T02:27:09.581532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* There are total 14 unque coins","metadata":{}},{"cell_type":"code","source":"df_supp_train","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:11.571674Z","iopub.execute_input":"2022-01-15T02:27:11.571944Z","iopub.status.idle":"2022-01-15T02:27:11.593984Z","shell.execute_reply.started":"2022-01-15T02:27:11.571913Z","shell.execute_reply":"2022-01-15T02:27:11.593030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_supp_train.info()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:12.683262Z","iopub.execute_input":"2022-01-15T02:27:12.684042Z","iopub.status.idle":"2022-01-15T02:27:12.693913Z","shell.execute_reply.started":"2022-01-15T02:27:12.684005Z","shell.execute_reply":"2022-01-15T02:27:12.693052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_supp_train.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:14.067976Z","iopub.execute_input":"2022-01-15T02:27:14.068807Z","iopub.status.idle":"2022-01-15T02:27:14.117372Z","shell.execute_reply.started":"2022-01-15T02:27:14.068762Z","shell.execute_reply":"2022-01-15T02:27:14.116553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# auxiliary function, from datetime to timestamp\ntotimestamp = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:15.217461Z","iopub.execute_input":"2022-01-15T02:27:15.218227Z","iopub.status.idle":"2022-01-15T02:27:15.222491Z","shell.execute_reply.started":"2022-01-15T02:27:15.218189Z","shell.execute_reply":"2022-01-15T02:27:15.221678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ****Checking Time range****","metadata":{}},{"cell_type":"code","source":"df_train.Asset_ID.unique()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:32.356135Z","iopub.execute_input":"2022-01-15T02:27:32.356793Z","iopub.status.idle":"2022-01-15T02:27:32.502626Z","shell.execute_reply.started":"2022-01-15T02:27:32.356746Z","shell.execute_reply":"2022-01-15T02:27:32.501787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.Asset_ID.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:33.601807Z","iopub.execute_input":"2022-01-15T02:27:33.602127Z","iopub.status.idle":"2022-01-15T02:27:33.731640Z","shell.execute_reply.started":"2022-01-15T02:27:33.602081Z","shell.execute_reply":"2022-01-15T02:27:33.730677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Bitcoin**","metadata":{}},{"cell_type":"code","source":"df_train[df_train['Asset_ID'] == 1].set_index(\"timestamp\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:36.768601Z","iopub.execute_input":"2022-01-15T02:27:36.768887Z","iopub.status.idle":"2022-01-15T02:27:37.091513Z","shell.execute_reply.started":"2022-01-15T02:27:36.768855Z","shell.execute_reply":"2022-01-15T02:27:37.090709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc = df_train[df_train['Asset_ID'] == 1].set_index(\"timestamp\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:38.869009Z","iopub.execute_input":"2022-01-15T02:27:38.869273Z","iopub.status.idle":"2022-01-15T02:27:39.170368Z","shell.execute_reply.started":"2022-01-15T02:27:38.869244Z","shell.execute_reply":"2022-01-15T02:27:39.169523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datetime.fromtimestamp(btc.index[0])","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:41.693117Z","iopub.execute_input":"2022-01-15T02:27:41.693662Z","iopub.status.idle":"2022-01-15T02:27:41.699550Z","shell.execute_reply.started":"2022-01-15T02:27:41.693626Z","shell.execute_reply":"2022-01-15T02:27:41.698801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Starting Date Time\n# %A= day,%B = Month, %d = month number, %Y  = Year,%I =hours , %M =minutes starting date, %S = seconds\ndatetime.fromtimestamp(btc.index[0]).strftime(\"%A, %B %d, %Y  %I: %M: %S\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:43.315230Z","iopub.execute_input":"2022-01-15T02:27:43.315936Z","iopub.status.idle":"2022-01-15T02:27:43.321559Z","shell.execute_reply.started":"2022-01-15T02:27:43.315891Z","shell.execute_reply":"2022-01-15T02:27:43.320722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Above function store in a starting date formate\nbeg_btc = datetime.fromtimestamp(btc.index[0]).strftime(\"%A, %B %d, %Y  %I: %M: %S\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:44.690313Z","iopub.execute_input":"2022-01-15T02:27:44.690992Z","iopub.status.idle":"2022-01-15T02:27:44.695806Z","shell.execute_reply.started":"2022-01-15T02:27:44.690940Z","shell.execute_reply":"2022-01-15T02:27:44.694852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Ending Date time\ndatetime.fromtimestamp(btc.index[-1]).strftime(\"%A, %B %d, %Y %I : %M : %S\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:46.276222Z","iopub.execute_input":"2022-01-15T02:27:46.276761Z","iopub.status.idle":"2022-01-15T02:27:46.282766Z","shell.execute_reply.started":"2022-01-15T02:27:46.276711Z","shell.execute_reply":"2022-01-15T02:27:46.282019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"end_btc = datetime.fromtimestamp(btc.index[-1]).strftime(\"%A, %B %d, %Y %I : %M : %S\")","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:47.462408Z","iopub.execute_input":"2022-01-15T02:27:47.463029Z","iopub.status.idle":"2022-01-15T02:27:47.467747Z","shell.execute_reply.started":"2022-01-15T02:27:47.462978Z","shell.execute_reply":"2022-01-15T02:27:47.466937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Bitcoin data goes from ', beg_btc, ' to ', end_btc)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:49.796110Z","iopub.execute_input":"2022-01-15T02:27:49.796382Z","iopub.status.idle":"2022-01-15T02:27:49.801105Z","shell.execute_reply.started":"2022-01-15T02:27:49.796353Z","shell.execute_reply":"2022-01-15T02:27:49.800512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Selecting 4-Coins\n* Bitcoin  = btc\n* Etherium = eth\n* Binance  = bnb\n* Cardano  = ada","metadata":{}},{"cell_type":"code","source":"## Checking Time Range\nbtc = df_train[df_train[\"Asset_ID\"]==1].set_index(\"timestamp\") # Asset_ID = 1 for Bitcoin\neth = df_train[df_train[\"Asset_ID\"]==6].set_index(\"timestamp\") # Asset_ID = 6 for Ethereum\nbnb = df_train[df_train[\"Asset_ID\"]==0].set_index(\"timestamp\") # Asset_ID = 0 for Binance Coin\nada = df_train[df_train[\"Asset_ID\"]==3].set_index(\"timestamp\") # Asset_ID = 3 for Cardano\n\nbeg_btc = datetime.fromtimestamp(btc.index[0]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nend_btc = datetime.fromtimestamp(btc.index[-1]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nbeg_eth = datetime.fromtimestamp(eth.index[0]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nend_eth = datetime.fromtimestamp(eth.index[-1]).strftime(\"%A, %B %d, %Y %I:%M:%S\")\nbeg_bnb = datetime.fromtimestamp(eth.index[0]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nend_bnb = datetime.fromtimestamp(eth.index[-1]).strftime(\"%A, %B %d, %Y %I:%M:%S\")\nbeg_ada = datetime.fromtimestamp(eth.index[0]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nend_ada = datetime.fromtimestamp(eth.index[-1]).strftime(\"%A, %B %d, %Y %I:%M:%S\")\n\nprint('Bitcoin data goes from ', beg_btc, ' to ', end_btc) \nprint('Ethereum data goes from ', beg_eth, ' to ', end_eth)\nprint('Binance coin data goes from ', beg_bnb, ' to ', end_bnb) \nprint('Cardano data goes from ', beg_ada, ' to ', end_ada)","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:27:57.057746Z","iopub.execute_input":"2022-01-15T02:27:57.058299Z","iopub.status.idle":"2022-01-15T02:27:58.207958Z","shell.execute_reply.started":"2022-01-15T02:27:57.058263Z","shell.execute_reply":"2022-01-15T02:27:58.207014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Here we have 4-Years of data range 2018-2021","metadata":{}},{"cell_type":"markdown","source":"# Checking corelation of each individual coin\n* By using heatmap","metadata":{}},{"cell_type":"markdown","source":"#### **Btc**","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.heatmap(btc[['Count','Open','High','Low','Close','Volume','VWAP','Target']].corr(), \n            vmin=-1.0, vmax=1.0, annot=True, cmap='coolwarm', linewidths=0.1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:28:14.362099Z","iopub.execute_input":"2022-01-15T02:28:14.362468Z","iopub.status.idle":"2022-01-15T02:28:15.594470Z","shell.execute_reply.started":"2022-01-15T02:28:14.362435Z","shell.execute_reply":"2022-01-15T02:28:15.593641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Checking candle chart b/w 2-coins Btc,Eth","metadata":{}},{"cell_type":"code","source":"btc.iloc[-1440:]","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:28:20.776439Z","iopub.execute_input":"2022-01-15T02:28:20.777022Z","iopub.status.idle":"2022-01-15T02:28:20.796512Z","shell.execute_reply.started":"2022-01-15T02:28:20.776988Z","shell.execute_reply":"2022-01-15T02:28:20.795462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Checking candle stick for Etherium for last 24 hrs","metadata":{}},{"cell_type":"code","source":"eth.iloc[-1440:]","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:28:25.821225Z","iopub.execute_input":"2022-01-15T02:28:25.821765Z","iopub.status.idle":"2022-01-15T02:28:25.843760Z","shell.execute_reply.started":"2022-01-15T02:28:25.821716Z","shell.execute_reply":"2022-01-15T02:28:25.842938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking 1 day candle stick chart 24hrs = 1440 mins\nbtc_mini = btc.iloc[-1440:]\neth_mini = eth.iloc[-1440:]\n\n# index = timestamp,\nfig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\nfig.update_xaxes(title_text=\"$\")\nfig.update_yaxes(title_text=\"Index\")\nfig.update_layout(title=\"Bitcoin Price, Last 24 hours\")\nfig.show()\n\nfig = go.Figure(data=[go.Candlestick(x=eth_mini.index, open=eth_mini['Open'], high=eth_mini['High'], low=eth_mini['Low'], close=eth_mini['Close'])])\nfig.update_xaxes(title_text=\"$\")\nfig.update_yaxes(title_text=\"Index\")\nfig.update_layout(title=\"Ethereum Price,Last 24 hours\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:28:28.746534Z","iopub.execute_input":"2022-01-15T02:28:28.747081Z","iopub.status.idle":"2022-01-15T02:28:28.940378Z","shell.execute_reply.started":"2022-01-15T02:28:28.747042Z","shell.execute_reply":"2022-01-15T02:28:28.939685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* We can Observe the corelation b/w two coins","metadata":{}},{"cell_type":"markdown","source":"# Ploting the Btc & Etherium","metadata":{}},{"cell_type":"code","source":"f = plt.figure(figsize=(15,4))\n\n# fill NAs for BTC and ETH\nbtc = btc.reindex(range(btc.index[0],btc.index[-1]+60,60),method='pad')\neth = eth.reindex(range(eth.index[0],eth.index[-1]+60,60),method='pad')\n\nax = f.add_subplot(121)\nplt.plot(btc['Close'], color='yellow', label='BTC')\nplt.legend()\nplt.xlabel('Time (timestamp)')\nplt.ylabel('Bitcoin')\n\nax2 = f.add_subplot(122)\nax2.plot(eth['Close'], color='purple', label='ETH')\nplt.legend()\nplt.xlabel('Time (timestamp)')\nplt.ylabel('Ethereum')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:28:35.057428Z","iopub.execute_input":"2022-01-15T02:28:35.058203Z","iopub.status.idle":"2022-01-15T02:28:38.473488Z","shell.execute_reply.started":"2022-01-15T02:28:35.058145Z","shell.execute_reply":"2022-01-15T02:28:38.472700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* We can also confirm through another visual observation that, within the 4 recent years, BTC and ETH prices are correlated.","metadata":{}},{"cell_type":"markdown","source":"# Coin corelation for 1-weak","metadata":{}},{"cell_type":"code","source":"data =df_train[-10080:]\ncheck = pd.DataFrame()\nfor i in data.Asset_ID.unique():\n    check[i] = data[data.Asset_ID==i]['Target'].reset_index(drop=True) \n    \nplt.figure(figsize=(10,8))\nsns.heatmap(check.dropna().corr(), vmin=-1.0, vmax=1.0, annot=True, cmap='coolwarm', linewidths=0.1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:28:48.337361Z","iopub.execute_input":"2022-01-15T02:28:48.337658Z","iopub.status.idle":"2022-01-15T02:28:49.407224Z","shell.execute_reply.started":"2022-01-15T02:28:48.337622Z","shell.execute_reply":"2022-01-15T02:28:49.406348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Interestingly, in the last 1-weak we have several coins that are highly correlated with one another.","metadata":{}},{"cell_type":"markdown","source":"# Feature Extraction\n* we add some features for our future predection","metadata":{}},{"cell_type":"code","source":"def hlco_ratio(df): \n    return (df['High'] - df['Low'])/(df['Close']-df['Open'])\ndef upper_shadow(df):\n    return df['High'] - np.maximum(df['Close'], df['Open'])\ndef lower_shadow(df):\n    return np.minimum(df['Close'], df['Open']) - df['Low']\n\ndef get_features(df):\n    df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n    df_feat['Upper_Shadow'] = upper_shadow(df_feat)\n    df_feat['hlco_ratio'] = hlco_ratio(df_feat)\n    df_feat['Lower_Shadow'] = lower_shadow(df_feat)\n    return df_feat","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:28:54.503574Z","iopub.execute_input":"2022-01-15T02:28:54.503917Z","iopub.status.idle":"2022-01-15T02:28:54.511328Z","shell.execute_reply.started":"2022-01-15T02:28:54.503882Z","shell.execute_reply":"2022-01-15T02:28:54.510497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Splitting","metadata":{}},{"cell_type":"code","source":"# Training data set\ntrain_data = df_train\ntrain_data","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:28:57.954504Z","iopub.execute_input":"2022-01-15T02:28:57.954844Z","iopub.status.idle":"2022-01-15T02:28:57.976251Z","shell.execute_reply.started":"2022-01-15T02:28:57.954810Z","shell.execute_reply":"2022-01-15T02:28:57.975510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_Xy_and_model_for_asset(df_train, asset_id):\n    df = df_train[df_train[\"Asset_ID\"] == asset_id]\n    \n    df = df.sample(frac=0.2)\n    df_proc = get_features(df)\n    df_proc['y'] = df['Target']\n    df_proc.replace([np.inf, -np.inf], np.nan, inplace=True)\n    df_proc = df_proc.dropna(how=\"any\")\n    \n    \n    X = df_proc.drop(\"y\", axis=1)\n    y = df_proc[\"y\"]   \n    model = LGBMRegressor()\n    model.fit(X, y)\n    return X, y, model\n\n\nXs = {}\nys = {}\nmodels = {}\n\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n    X, y, model = get_Xy_and_model_for_asset(train_data, asset_id)       \n    try:\n        Xs[asset_id], ys[asset_id], models[asset_id] = X, y, model\n    except: \n        Xs[asset_id], ys[asset_id], models[asset_id] = None, None, None ","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:28:59.009854Z","iopub.execute_input":"2022-01-15T02:28:59.010117Z","iopub.status.idle":"2022-01-15T02:29:17.221630Z","shell.execute_reply.started":"2022-01-15T02:28:59.010090Z","shell.execute_reply":"2022-01-15T02:29:17.220885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Hyperparam Tuning\n* We will perform GridSearch for each LGBM model of 14 coins.","metadata":{}},{"cell_type":"code","source":"parameters = {\n    # 'max_depth': range (2, 10, 1),\n    'num_leaves': range(21, 161, 10),\n    'learning_rate': [0.1, 0.01, 0.05]\n}\n\nnew_models = {}\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(\"GridSearchCV for: \" + asset_name)\n    grid_search = GridSearchCV(\n        estimator=get_Xy_and_model_for_asset(df_train, asset_id)[2], # bitcoin\n        param_grid=parameters,\n        n_jobs = -1,\n        cv = 5,\n        verbose=True\n    )\n    grid_search.fit(Xs[asset_id], ys[asset_id])\n    new_models[asset_id] = grid_search.best_estimator_\n    grid_search.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2022-01-15T02:29:23.628121Z","iopub.execute_input":"2022-01-15T02:29:23.628558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking the Model interface\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(f\"Tuned model for {asset_name:<1} (ID={asset_id:})\")\n    print(new_models[asset_id])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"env = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    for j , row in df_test.iterrows():        \n        if new_models[row['Asset_ID']] is not None:\n            try:\n                model = new_models[row['Asset_ID']]\n                x_test = get_features(row)\n                y_pred = model.predict(pd.DataFrame([x_test]))[0]\n                df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n            except:\n                df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n                traceback.print_exc()\n        else: \n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0  \n    \n    env.predict(df_pred)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}