{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G-Research Crypto Forecasting\n\nG-ReSearch Crypto Forecasting is a time series data, which contains daily price of cryptocurrencies. Our task is to  forecast short term returns in 14 popular cryptocurrencies.\n\nThe purpose of this notebook is to share basic approach to time series data analysis. ","metadata":{}},{"cell_type":"markdown","source":"## Contents\n1. [Basic EDA](#eda)\n2. [Time series feature engineering](#fe)\n3. [Modeling](#model)\n + [Time series cross validation](#tscv)\n + [Hyperparameter tuning with Optuna](#optuna)","metadata":{}},{"cell_type":"markdown","source":"## Load packages and data","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport warnings\nwarnings.filterwarnings('ignore')\n\nfrom lightgbm import LGBMRegressor\nfrom xgboost import XGBRegressor\nfrom sklearn.model_selection import TimeSeriesSplit\n\nfrom statsmodels.tsa.arima_model import ARIMA\nfrom statsmodels.tsa.statespace.sarimax import SARIMAX\nfrom pandas.plotting import autocorrelation_plot\nfrom statsmodels.tsa.stattools import adfuller, acf, pacf, arma_order_select_ic\nimport statsmodels.formula.api as smf\nimport statsmodels.tsa.api as smt\nimport statsmodels.api as sm\nimport scipy.stats as scs\nfrom scipy.stats.stats import pearsonr\n\nimport time\nfrom datetime import datetime\nfrom tqdm import tqdm","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-10T11:43:54.561141Z","iopub.execute_input":"2022-02-10T11:43:54.561484Z","iopub.status.idle":"2022-02-10T11:43:58.43202Z","shell.execute_reply.started":"2022-02-10T11:43:54.561393Z","shell.execute_reply":"2022-02-10T11:43:58.431326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/g-research-crypto-forecasting/train.csv')\nasset_details = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nsup_train = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv')\n\nprint(train.shape)\nprint(sup_train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:43:58.434142Z","iopub.execute_input":"2022-02-10T11:43:58.434748Z","iopub.status.idle":"2022-02-10T11:45:07.590796Z","shell.execute_reply.started":"2022-02-10T11:43:58.434696Z","shell.execute_reply":"2022-02-10T11:45:07.589693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. Basic EDA <a class=\"anchor\" id=\"eda\"></a>\nThis part contains simple exploration of G-Research data. Before we get into EDA, we need to change format of timestamp feature and create some new features.","metadata":{}},{"cell_type":"code","source":"# change format of timestamp variable\ntrain['timestamp'] = pd.to_datetime(train['timestamp'], unit = 's')\n\n# Diff: difference between close and open price of crypto\ntrain['Diff'] = train['Close'] - train['Open']","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:07.595873Z","iopub.execute_input":"2022-02-10T11:45:07.596144Z","iopub.status.idle":"2022-02-10T11:45:09.509858Z","shell.execute_reply.started":"2022-02-10T11:45:07.596102Z","shell.execute_reply":"2022-02-10T11:45:09.509088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# order by asset_id (ascending)\nasset_details = asset_details.sort_values(by = 'Asset_ID').reset_index(drop = True)\nasset_details","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:09.52574Z","iopub.execute_input":"2022-02-10T11:45:09.526262Z","iopub.status.idle":"2022-02-10T11:45:09.543785Z","shell.execute_reply.started":"2022-02-10T11:45:09.526228Z","shell.execute_reply":"2022-02-10T11:45:09.543236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Comparison of average trading volume of assets\nThis graph shows average trading volume of each asset. **Bitcoin, Dodgecoin and Ethereum** are 3 most frequently traded assets among 14 cryptos.","metadata":{}},{"cell_type":"code","source":"assets = train.groupby('Asset_ID')['Count'].mean().reset_index()\n\ncolors = ['lightgrey']*14\ncolors[1] = colors[4] = colors[6] = '#3366ff'\nassets_bar = go.Bar(x = assets['Asset_ID'], y = assets['Count'], marker_color = colors)\ndata = [assets_bar]\nlayout = go.Layout(title = 'Average trading volume of each asset')\nfig = go.Figure(data = data, layout = layout)\n\nfig.update_traces(marker_line_width = 1,marker_line_color = \"black\")\nfig.update_layout(\n    title = {\n        'text': 'Average trading volume of each asset',\n        'y':0.90,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'} , \n    \n    xaxis = dict(\n        tickvals = list(range(0, 14)),\n        ticktext = asset_details.Asset_Name\n    ),\n    template = \"plotly_white\")\n\nfig.update_xaxes(title_text = 'Asset')\nfig.update_yaxes(title_text = 'Trading volume')\n\nfig","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:09.54475Z","iopub.execute_input":"2022-02-10T11:45:09.545464Z","iopub.status.idle":"2022-02-10T11:45:10.997438Z","shell.execute_reply.started":"2022-02-10T11:45:09.545406Z","shell.execute_reply":"2022-02-10T11:45:10.996569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Candlestick chart of recent 1 year asset price\nThis is candlestick chart of recent 1 year price of specific asset. I plotted chart of 3 most frequently traded assets, Bitcoin, Dogecoin and Ethereum. Overall trend of price of those 3 assets are quite similar.\n\nReference: <https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition/notebook#Building-your-prediction-model>","metadata":{}},{"cell_type":"code","source":"def candelstick_chart(data, id_, title):\n    data = data[data['Asset_ID'] == id_].reset_index(drop = True)\n    data = data.set_index('timestamp')\n    data = data.iloc[-365:,:]  # recent 1 year\n    data['MA5'] = data['Close'].rolling(window = 5, min_periods = 1).mean()\n    data['MA30'] = data['Close'].rolling(window = 30, min_periods = 1).mean()\n    data['MA120'] = data['Close'].rolling(window = 120, min_periods = 1).mean()\n    \n    candlestick = go.Figure(data = [go.Candlestick(x =data.index, \n                                               open = data[('Open')], \n                                               high = data[('High')], \n                                               low = data[('Low')], \n                                               close = data[('Close')])])\n    candlestick.update_xaxes(title_text = 'Time')\n\n    candlestick.update_layout(\n    title = {\n        'text': '{:} Candelstick Chart'.format(title),\n        'y':0.90,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'} , \n    template=\"plotly_white\")\n\n    candlestick.update_yaxes(title_text = 'Price in USD', ticksuffix = '$')\n    return candlestick","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:10.998843Z","iopub.execute_input":"2022-02-10T11:45:10.999566Z","iopub.status.idle":"2022-02-10T11:45:11.010512Z","shell.execute_reply.started":"2022-02-10T11:45:10.999521Z","shell.execute_reply":"2022-02-10T11:45:11.009165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"candelstick_chart(train, 1, 'Bitcoin')","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:11.012919Z","iopub.execute_input":"2022-02-10T11:45:11.013725Z","iopub.status.idle":"2022-02-10T11:45:13.087407Z","shell.execute_reply.started":"2022-02-10T11:45:11.013683Z","shell.execute_reply":"2022-02-10T11:45:13.086507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"candelstick_chart(train, 4, 'Dodgecoin')","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:13.088707Z","iopub.execute_input":"2022-02-10T11:45:13.088964Z","iopub.status.idle":"2022-02-10T11:45:13.441374Z","shell.execute_reply.started":"2022-02-10T11:45:13.088933Z","shell.execute_reply":"2022-02-10T11:45:13.440527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"candelstick_chart(train, 6, 'Ethereum')","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:13.442543Z","iopub.execute_input":"2022-02-10T11:45:13.442784Z","iopub.status.idle":"2022-02-10T11:45:13.938947Z","shell.execute_reply.started":"2022-02-10T11:45:13.442754Z","shell.execute_reply":"2022-02-10T11:45:13.938023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Check overall trend: Moving Average\nTo check overall trend of asset price, we use **moving average(MA)** in general. In finance, a moving average (MA) is a stock indicator that is commonly used in technical analysis. The reason for calculating the moving average of a stock is to help smooth out the price data by creating a constantly updated average price.\n\n + Simple moving average(SMA): calculation that takes the arithmetic mean of a given set of prices over the specific number of days in the past\n + Exponential moving average(EMA): weighted average that gives greater importance to the price of a stock in more recent days, making it an indicator that is more responsive to new information.\n \n \nIn this part, I'll calculate SMA of close price of asset for 5(weekly), 30(monthly) and 120 days. MA5 and MA30 shows short-term trend and MA120 shows long-term trend of asset price.","metadata":{}},{"cell_type":"code","source":"def ma_chart(data, id_, title):\n    data = data[data['Asset_ID'] == id_].reset_index(drop = True)\n    data = data.set_index('timestamp')\n    data = data.iloc[-365:,:]  # recent 1 year\n    data['MA5'] = data['Close'].rolling(window = 5, min_periods = 1).mean()\n    data['MA30'] = data['Close'].rolling(window = 30, min_periods = 1).mean()\n    data['MA120'] = data['Close'].rolling(window = 120, min_periods = 1).mean()\n    \n    ma = go.Figure(data = [go.Scatter(x = data.index, y = data['Close'], mode='lines', \n                                     name = 'Close', line = dict(color = 'black', width = 2))])\n    ma.update_xaxes(title_text = 'Time')\n\n    ma.update_layout(\n    title = {\n        'text': '{:} Moving Average'.format(title),\n        'y':0.90,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'} , \n    template=\"plotly_white\")\n    \n    ma.add_trace(go.Scatter(x = data.index, y = data['MA5'], mode='lines', \n                                     name='MA5', line = dict(color = 'red', width = 2)))\n    ma.add_trace(go.Scatter(x = data.index, y = data['MA30'], mode='lines', \n                                     name='MA30', line = dict(color = 'blue', width = 2)))\n    ma.add_trace(go.Scatter(x = data.index, y = data['MA120'], mode='lines', \n                                     name='MA120', line = dict(color = 'orange', width = 2)))\n\n    ma.update_yaxes(title_text = 'Price in USD', ticksuffix = '$')\n    return ma","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:13.940461Z","iopub.execute_input":"2022-02-10T11:45:13.940722Z","iopub.status.idle":"2022-02-10T11:45:13.95486Z","shell.execute_reply.started":"2022-02-10T11:45:13.940691Z","shell.execute_reply":"2022-02-10T11:45:13.953808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Moving average chart of Bitcoin\nma_chart(train, 1, 'Bitcoin')","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:13.956259Z","iopub.execute_input":"2022-02-10T11:45:13.956516Z","iopub.status.idle":"2022-02-10T11:45:14.512374Z","shell.execute_reply.started":"2022-02-10T11:45:13.956485Z","shell.execute_reply":"2022-02-10T11:45:14.509369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Missing values\nThere are some missing values in **Target** variable and **VWAP**. All missing values in VWAP variable belong to Asset 10, Maker. Since there are only 9 missings in this variable, we'll just remove those data. Missing in Target will not be included in modeling.","metadata":{}},{"cell_type":"code","source":"# Target - missing 3% \ntrain.isna().sum().sort_values(ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:14.513744Z","iopub.execute_input":"2022-02-10T11:45:14.514341Z","iopub.status.idle":"2022-02-10T11:45:15.083942Z","shell.execute_reply.started":"2022-02-10T11:45:14.514302Z","shell.execute_reply":"2022-02-10T11:45:15.083364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check missing in VWAP\ntrain[train['VWAP'].isna()]","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:15.084981Z","iopub.execute_input":"2022-02-10T11:45:15.085338Z","iopub.status.idle":"2022-02-10T11:45:15.140017Z","shell.execute_reply.started":"2022-02-10T11:45:15.085307Z","shell.execute_reply":"2022-02-10T11:45:15.139137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# remove NA values in VWAP\ntrain = train[train['VWAP'].isna() == False]","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:15.141811Z","iopub.execute_input":"2022-02-10T11:45:15.142683Z","iopub.status.idle":"2022-02-10T11:45:16.481939Z","shell.execute_reply.started":"2022-02-10T11:45:15.142633Z","shell.execute_reply":"2022-02-10T11:45:16.481051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Time series Feature Engineering <a class=\"anchor\" id=\"fe\"></a>\nWe will use asset Bitcoin data only for feature engineering and modeling in this notebook. For other 13 assets, you can repeat same process.","metadata":{}},{"cell_type":"code","source":"# Bitcoin only\ndata = train[train['Asset_ID'] == 1]","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:16.483222Z","iopub.execute_input":"2022-02-10T11:45:16.483492Z","iopub.status.idle":"2022-02-10T11:45:16.814984Z","shell.execute_reply.started":"2022-02-10T11:45:16.48346Z","shell.execute_reply":"2022-02-10T11:45:16.814212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# remove rows without target \ndata = data[data['Target'].isna() == False]\ndata","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:16.816035Z","iopub.execute_input":"2022-02-10T11:45:16.816298Z","iopub.status.idle":"2022-02-10T11:45:16.942748Z","shell.execute_reply.started":"2022-02-10T11:45:16.816269Z","shell.execute_reply":"2022-02-10T11:45:16.941824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Additional features\nI created several additional features for modeling, including **time related features**. In time series analysis, creating time related features can significantly increase model performance. I used 2 mostly used technique for time series analysis: **moving average** and **lagged features**.\n\n1. Moving average \n + Above explanation with graph\n<br>\n2. Lagged feature\n + In time series analysis, future value is greatly affected by past values. Those past values are **lags** and we use thos lag features to enhance model performance.","metadata":{}},{"cell_type":"code","source":"# df should be particular asset\ndef get_feats(df):\n    df['upper_shadow'] = df['High'] - np.maximum(df['Close'], df['Open'])\n    df['lower_shadow'] = np.minimum(df['Close'], df['Open']) - df['Low']\n    \n    # average trading volume\n    df['avg_volume'] = df['Volume'] / df['Count']\n    \n    # average price\n    df['avg_price'] = (df['Open'] + df['High'] + df['Low'] + df['Close']) / 4\n    \n    # difference between open and close price\n    df.rename(columns = {'Diff': 'diff_open_close'})\n    \n    # moving average features - short term and long term\n    df['ma_20'] = df['Close'].rolling(window = 20, min_periods = 1).mean()\n    df['ma_120'] = df['Close'].rolling(window = 120, min_periods = 1).mean()\n    \n    # lagged features\n    lags_ = [5, 20, 60, 120]\n    for lag in lags_:\n        df['lag_' + str(lag)] = df['Target'].shift(lag)\n        \n    df = df.fillna(0)\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:16.943901Z","iopub.execute_input":"2022-02-10T11:45:16.944217Z","iopub.status.idle":"2022-02-10T11:45:16.954593Z","shell.execute_reply.started":"2022-02-10T11:45:16.944186Z","shell.execute_reply":"2022-02-10T11:45:16.95353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lagged features\ndata = get_feats(data)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:16.95602Z","iopub.execute_input":"2022-02-10T11:45:16.956312Z","iopub.status.idle":"2022-02-10T11:45:32.453042Z","shell.execute_reply.started":"2022-02-10T11:45:16.956278Z","shell.execute_reply":"2022-02-10T11:45:32.452212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### FE function\nBelow function creates time-series features of each assets. You can just enter asset_id in **asset_feats** function to get features for that particular asset.","metadata":{}},{"cell_type":"code","source":"# feature engineering for other assets \ndef asset_feats(df, asset_id):\n    data = df[df['Asset_ID'] == asset_id]\n    data = data[data['Target'].isna() == False]\n    data = get_feats(data)\n    return data","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:32.454491Z","iopub.execute_input":"2022-02-10T11:45:32.454748Z","iopub.status.idle":"2022-02-10T11:45:32.46112Z","shell.execute_reply.started":"2022-02-10T11:45:32.454717Z","shell.execute_reply":"2022-02-10T11:45:32.460257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Modeling <a class=\"anchor\" id=\"model\"></a>","metadata":{}},{"cell_type":"markdown","source":"### (1) Time Series Cross Validation <a class=\"anchor\" id=\"tscv\"></a>\nFor time series modeling, I will use boosting regressors, LGBMRegressor and XGBRegressor, which show great performance in general. First step is to compare performance of two models with time series cross validation. In time series analysis, it's not recommended to apply KFold or Stratified KFold since observations in past influences current values. Instead, we use **Time series split** in this case. Below image shows how data is splitted if you apply time series split. Observations from the training set occur before their corresponding validation set. We will use 10-fold time series cross validation in this notebook. (Evaluation metric is Pearson Correlation Coefficient)\n<br>\n![tcsv](https://miro.medium.com/max/1204/1*qvdnPF8ETV9mFdMT0Y_BBA.png)","metadata":{}},{"cell_type":"code","source":"# create X and y(target)\nX = data.drop(['timestamp', 'Asset_ID'], axis = 1)\ny = data['Target']","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:32.462453Z","iopub.execute_input":"2022-02-10T11:45:32.463001Z","iopub.status.idle":"2022-02-10T11:45:32.579762Z","shell.execute_reply.started":"2022-02-10T11:45:32.462964Z","shell.execute_reply":"2022-02-10T11:45:32.578885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 10-fold time series cross validation\ndef timecv_model(model, X, y):\n    tfold = TimeSeriesSplit(n_splits = 10)\n    pcc_list = []\n    for _, (train_index, test_index) in tqdm(enumerate(tfold.split(X), start=1)):\n        X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n        y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n        clf = model.fit(X_train, y_train)\n        pred = clf.predict(X_test)\n        pcc = pearsonr(pred, y_test) \n        pcc_list.append(pcc[0])\n    \n    return pcc_list\n\ndef cv_result(model, X, y):\n    model_name = model.__class__.__name__\n    pcc_ = timecv_model(model, X, y)\n    for i, pcc in enumerate(pcc_):\n        print(f'{i}th fold: {model_name} PCC: {pcc:.4f}')\n    print(f'\\n{model_name} average PCC: {np.mean(pcc_):.4f}')","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:32.58357Z","iopub.execute_input":"2022-02-10T11:45:32.583827Z","iopub.status.idle":"2022-02-10T11:45:32.593582Z","shell.execute_reply.started":"2022-02-10T11:45:32.583798Z","shell.execute_reply":"2022-02-10T11:45:32.592568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below are results of 10-fold time series cross validation(TSCV) of LGBMRegressor and XGBRegressor. It takes much longer time in XGBRegressor. Parameters are randomly selected, just to compare general performance of two models. We will choose better model with TSCV and then conduct hyperparameter tuning.","metadata":{}},{"cell_type":"code","source":"lgb_model = LGBMRegressor(n_estimators = 1500,\n                      max_depth = 10,\n                      num_leaves = 20,\n                      colsample_bytree = 0.8,\n                      subsample = 0.7,\n                      seed = 0)\n\ncv_result(lgb_model, X, y)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:45:32.594796Z","iopub.execute_input":"2022-02-10T11:45:32.595112Z","iopub.status.idle":"2022-02-10T11:51:30.700618Z","shell.execute_reply.started":"2022-02-10T11:45:32.595063Z","shell.execute_reply":"2022-02-10T11:51:30.699742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#xgb_model = XGBRegressor(n_estimators = 1500,\n                         #max_depth = 10,\n                         #min_child_weight = 5,\n                         #gamma = 0.1)\n\n#cv_result(xgb_model, X, y)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:51:30.70279Z","iopub.execute_input":"2022-02-10T11:51:30.703372Z","iopub.status.idle":"2022-02-10T11:51:30.715291Z","shell.execute_reply.started":"2022-02-10T11:51:30.70332Z","shell.execute_reply":"2022-02-10T11:51:30.714276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### (2) Optuna\nNow, tune parameters to get optimal parameter for LGBMRegressor. There are many hyperparameter tuning techinques: GridSearchCV, RandomSearchCV, Bayesian Optimization..etc..\n\nBut I will try **Optuna** for hyperparameter tuning. Optuna is widely used in many data analysis platforms these days to get optimal parameters for model. Information about optuna is well explained [here](https://medium.com/@kalyaniavhale7/understanding-of-optuna-a-machine-learning-hyperparameter-optimization-framework-ed31ebb335b9). (I summarized explanations about optuna referring to this link.)","metadata":{}},{"cell_type":"code","source":"data['timestamp'] = data['timestamp'].dt.strftime('%Y-%m-%d')\n\nprint('Earliest date: ', min(data['timestamp']))\nprint('Lastest date: ', max(data['timestamp']))","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:51:30.716553Z","iopub.execute_input":"2022-02-10T11:51:30.718109Z","iopub.status.idle":"2022-02-10T11:51:45.659237Z","shell.execute_reply.started":"2022-02-10T11:51:30.718031Z","shell.execute_reply":"2022-02-10T11:51:45.658349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Bitcoin data starts from 2018-01-01 to 2021-09-20. I diveded full dataset into train and valid dataset first: Train data contains data before 2020-09-20 and validation data contains data after 2020-09-20. The ratio of train and valid size is apporixmately 73:27.","metadata":{}},{"cell_type":"code","source":"train = data[data['timestamp'] < '2020-09-20']\nvalid = data[data['timestamp'] >= '2020-09-20']\n\ntrain.drop(['timestamp', 'Asset_ID'], axis = 1, inplace = True)\nvalid.drop(['timestamp', 'Asset_ID'], axis = 1, inplace = True)\n\nX_train = train.drop(['Target'], axis = 1)\ny_train = train['Target']\nX_valid = valid.drop(['Target'], axis = 1)\ny_valid = valid['Target']\n\nprint(X_train.shape)\nprint(X_valid.shape)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:51:45.660345Z","iopub.execute_input":"2022-02-10T11:51:45.660568Z","iopub.status.idle":"2022-02-10T11:51:46.663679Z","shell.execute_reply.started":"2022-02-10T11:51:45.660541Z","shell.execute_reply":"2022-02-10T11:51:46.662931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Objective function <a class=\"anchor\" id=\"optuna\"></a>\nOur objective of hyperparameter tuning is to find parameter that maximizes or minimized output of objective function. If evaluation metric is logloss, we have to minimized objective function but if it is accuracy, we have to maximize objective function. During the optimization, Optuna repeatedly calls and evaluates the objective function with different parameters.\n\nIn objective function, we define parameter search space.\n + categorical parameter: optuna.trial.Trial.suggest_categorical()\n + integer parameter: optuna.trial.Trial.suggest_int()\n + float parameter: optuna.trial.Trial.suggest_float()\n \n \n \nIn Optuna, we use the study object to manage optimization. Method create_study() returns a study object. A study object has useful properties for analyzing the optimization outcome. \n + create_study()","metadata":{}},{"cell_type":"code","source":"from optuna.samplers import TPESampler\nimport optuna\n\nsampler = TPESampler(seed = 0)\n\ndef objective(trial):\n    params = {\n        'objective': 'regression',\n        'verbose': -1,\n        'max_depth': trial.suggest_int('max_depth',5, 20),\n        'num_leaves': trial.suggest_int('num_leaves', 10, 40),\n        'learning_rate': trial.suggest_float(\"learning_rate\", 1e-5, 0.1),\n        'n_estimators': trial.suggest_int('n_estimators', 500, 2500),\n        'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),\n        'subsample': trial.suggest_float('subsample', 0.4, 1)}\n    \n    model = LGBMRegressor(**params)\n    model.fit(X_train, y_train, eval_set = [(X_train, y_train), (X_valid, y_valid)],\n         verbose = 0, early_stopping_rounds = 50)\n    pred = model.predict(X_valid)\n    pcc = pearsonr(pred, y_valid)[0]\n    return pcc\n\nstudy_model = optuna.create_study(direction = 'maximize', sampler = sampler)\nstudy_model.optimize(objective, n_trials = 20) ","metadata":{"execution":{"iopub.status.busy":"2022-02-10T11:56:17.87011Z","iopub.execute_input":"2022-02-10T11:56:17.871672Z","iopub.status.idle":"2022-02-10T12:00:37.048097Z","shell.execute_reply.started":"2022-02-10T11:56:17.871618Z","shell.execute_reply":"2022-02-10T12:00:37.047329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# select best trial and parameter\ntrial = study_model.best_trial\nbest_params = trial.params\n\nprint('Best params from optuna: \\n', best_params)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T12:02:40.835533Z","iopub.execute_input":"2022-02-10T12:02:40.836172Z","iopub.status.idle":"2022-02-10T12:02:40.842359Z","shell.execute_reply.started":"2022-02-10T12:02:40.836131Z","shell.execute_reply":"2022-02-10T12:02:40.841619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Plots from optuna\nOptuna provides visualization of result from hyperparmeter tuning.\n\n1. plot_optimization_history\n  + plot optimization history of all trials in a study\n<br>\n2. plot_slice\n  + plot the parameter relationship as slice plot in a study.\n<br>  \n3. plot_param_importances\n  + plot hyperparameter importances ","metadata":{}},{"cell_type":"code","source":"optuna.visualization.plot_optimization_history(study_model)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T12:02:45.02119Z","iopub.execute_input":"2022-02-10T12:02:45.021645Z","iopub.status.idle":"2022-02-10T12:02:45.041522Z","shell.execute_reply.started":"2022-02-10T12:02:45.0216Z","shell.execute_reply":"2022-02-10T12:02:45.040668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"optuna.visualization.plot_slice(study_model)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T12:02:48.522527Z","iopub.execute_input":"2022-02-10T12:02:48.522861Z","iopub.status.idle":"2022-02-10T12:02:49.070714Z","shell.execute_reply.started":"2022-02-10T12:02:48.522814Z","shell.execute_reply":"2022-02-10T12:02:49.069887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"optuna.visualization.plot_param_importances(study_model)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T12:02:53.354836Z","iopub.execute_input":"2022-02-10T12:02:53.355155Z","iopub.status.idle":"2022-02-10T12:02:53.965057Z","shell.execute_reply.started":"2022-02-10T12:02:53.355117Z","shell.execute_reply":"2022-02-10T12:02:53.964175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### (3) Prediction with selected optimal parameter","metadata":{}},{"cell_type":"code","source":"opt_model = LGBMRegressor(**best_params)\n\ncv_result(opt_model, X, y)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T12:03:12.128334Z","iopub.execute_input":"2022-02-10T12:03:12.12865Z","iopub.status.idle":"2022-02-10T12:05:53.57765Z","shell.execute_reply.started":"2022-02-10T12:03:12.128616Z","shell.execute_reply":"2022-02-10T12:05:53.577005Z"},"trusted":true},"execution_count":null,"outputs":[]}]}