{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G-Research Crypto forecasting competition","metadata":{"papermill":{"duration":0.022176,"end_time":"2022-01-03T13:27:50.797544","exception":false,"start_time":"2022-01-03T13:27:50.775368","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### コンペ概要\n\n目的：暗号通貨によるリターンの予測<br>\n暗号通貨に関する時系列の価格履歴を使用して、価格が上がるか下がるか、どれだけの資産がリターンするかを予測します。<br>\n本コンペでは、14種の暗号通貨についてのデータが提供されています。<br>\n\nデータの特性\n- 時系列データ。\n- 暗号通貨の変動は、他の暗号通貨や暗号通貨市場全体に対して影響している。\n- 株価と同様に、通貨の価格はランダムウォークパターンに従っているようにも見える。","metadata":{"tags":[]}},{"cell_type":"markdown","source":"ベースとなる、時系列データ用CVのLGBMモデルを作成します。<br>\ntrain.csvよりもさらに直近のデータが更新されているようです。[supplemental_train.csv](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/303302)を学習データに追加しました。<br>\n\n**参考**\n- [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n- [Data Description](https://www.kaggle.com/c/g-research-crypto-forecasting/data)\n- [Detailed API Introduction](https://www.kaggle.com/sohier/detailed-api-introduction)\n\nEDA：[【日本語】G-research_Data_Outline](https://www.kaggle.com/kitopl/g-research-data-outline)<br>\nBase_Model：[【日本語】Bigginner_G-Research_Base_LGBM_Model](https://www.kaggle.com/kitopl/bigginner-g-research-base-lgbm-model)<br>","metadata":{}},{"cell_type":"code","source":"import os\nimport random\nimport pandas as pd\nimport numpy as np\nfrom lightgbm import LGBMRegressor\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":1.526291,"end_time":"2022-01-03T13:27:52.390995","exception":false,"start_time":"2022-01-03T13:27:50.864704","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-31T14:23:34.391176Z","iopub.execute_input":"2022-01-31T14:23:34.391454Z","iopub.status.idle":"2022-01-31T14:23:36.015314Z","shell.execute_reply.started":"2022-01-31T14:23:34.391376Z","shell.execute_reply":"2022-01-31T14:23:36.014643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### データ読み込み","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv('../input/g-research-crypto-forecasting/train.csv')\ndata.head()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2022-01-31T14:23:36.016728Z","iopub.execute_input":"2022-01-31T14:23:36.017234Z","iopub.status.idle":"2022-01-31T14:24:31.675752Z","shell.execute_reply.started":"2022-01-31T14:23:36.017203Z","shell.execute_reply":"2022-01-31T14:24:31.675102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **timestamp**：1970-01-01 00:00:00.000 UTCからの経過秒数を示す。データは分スケールのため、60の倍数になっているようです。\n- **Asset_ID**：暗号通貨を識別するID(例：ビットコインの場合はAsset_ID = 1)。 Asset_IDと暗号通貨の対応関係はasset_details.csvで参照できます。\n- **Count**：時間間隔（直前）の取引の総数。\n- **Open**：指定時間間隔における始値(USD)。\n- **High**：指定時間間隔における最高価格(USD)。\n- **Low**：指定時間間隔における最低価格(USD)。\n- **Close**：指定時間間隔における終値(USD)。\n- **Volume**：購入または販売された通貨の数量。\n- **VWAP**：指定時間間隔における通貨の重みづけ平均価格。\n- **Target**：15分間の対数収益率から算出。\ntimestamp、Asset_IDの複合キーが、データを識別するインデックになります。","metadata":{"papermill":{"duration":0.021851,"end_time":"2022-01-03T13:28:55.815829","exception":false,"start_time":"2022-01-03T13:28:55.793978","status":"completed"},"tags":[]}},{"cell_type":"code","source":"add_data = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv')\nadd_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-31T14:24:31.676605Z","iopub.execute_input":"2022-01-31T14:24:31.676774Z","iopub.status.idle":"2022-01-31T14:24:38.021363Z","shell.execute_reply.started":"2022-01-31T14:24:31.676751Z","shell.execute_reply":"2022-01-31T14:24:38.020078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pd.concat([data, add_data], axis=0)\ndata.describe()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2022-01-31T14:24:38.022911Z","iopub.execute_input":"2022-01-31T14:24:38.023577Z","iopub.status.idle":"2022-01-31T14:24:43.965670Z","shell.execute_reply.started":"2022-01-31T14:24:38.023546Z","shell.execute_reply":"2022-01-31T14:24:43.964791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Asset_IDと暗号通貨の対応<br>\nWeightは、目的変数となっているLogReturnを算出する際に使用します。","metadata":{}},{"cell_type":"code","source":"asset_tmp = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nasset = asset_tmp.set_index('Asset_ID').sort_index()\nasset","metadata":{"tags":[],"execution":{"iopub.status.busy":"2022-01-31T14:24:43.966562Z","iopub.execute_input":"2022-01-31T14:24:43.966735Z","iopub.status.idle":"2022-01-31T14:24:43.983504Z","shell.execute_reply.started":"2022-01-31T14:24:43.966712Z","shell.execute_reply":"2022-01-31T14:24:43.982781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 特徴量エンジニアリング\nチュートリアルで紹介されているupper_shadow、lower_shadowとhigh_div_low、open_sub_closeを算出します。","metadata":{}},{"cell_type":"code","source":"# from the competition tutorial\ndef upper_shadow(df):\n    return df['High'] - np.maximum(df['Close'], df['Open'])\n\ndef lower_shadow(df):\n    return np.minimum(df['Close'], df['Open']) - df['Low']\n\ndef cal_features(df):\n    df_cal = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n    df_cal['upper_Shadow'] = upper_shadow(df_cal)\n    df_cal['lower_Shadow'] = lower_shadow(df_cal)\n    df_cal[\"high_div_low\"] = df_cal[\"High\"] / df_cal[\"Low\"]\n    df_cal[\"open_sub_close\"] = df_cal[\"Open\"] - df_cal[\"Close\"]\n    return df_cal\n\ndef Xy_split(df_train):    \n    X = cal_features(df_train)\n    y = df_train['Target']\n    return X, y","metadata":{"tags":[],"execution":{"iopub.status.busy":"2022-01-31T14:24:43.984449Z","iopub.execute_input":"2022-01-31T14:24:43.984631Z","iopub.status.idle":"2022-01-31T14:24:43.990942Z","shell.execute_reply.started":"2022-01-31T14:24:43.984608Z","shell.execute_reply":"2022-01-31T14:24:43.990406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 学習\n検証データに未来のデータを使用しないよう、[TimeSeriesSplit](https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split)でCVを作成します。","metadata":{}},{"cell_type":"code","source":"# weighted correlation\ndef corr(a, b, w):\n    cov = lambda x, y: np.sum(w * (x - np.average(x, weights=w)) * (y - np.average(y, weights=w))) / np.sum(w)\n    return cov(a, b) / np.sqrt(cov(a, a) * cov(b, b))","metadata":{"execution":{"iopub.status.busy":"2022-01-31T14:24:43.991741Z","iopub.execute_input":"2022-01-31T14:24:43.992319Z","iopub.status.idle":"2022-01-31T14:24:43.999565Z","shell.execute_reply.started":"2022-01-31T14:24:43.992293Z","shell.execute_reply":"2022-01-31T14:24:43.998950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import TimeSeriesSplit\nn_splits = 3\ntscv = TimeSeriesSplit(n_splits=n_splits)\n\nX_train_dict = {}\ny_train_dict = {}\nX_valid_dict = {}\ny_valid_dict = {}\nmodel_dict = {}\n\nfor asset_id, asset_name in zip(asset.index, asset['Asset_Name']):\n    print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n    df_asset = data[data[\"Asset_ID\"] == asset_id].reset_index(drop=True)\n    df_asset = df_asset.dropna(how='any')\n    \n    X_trains = []\n    y_trains = []\n    X_valids = []\n    y_valids = []\n    scores = []\n    models = []\n    importance = np.array([])\n    \n    for i, (train, valid) in enumerate(tscv.split(df_asset)):\n        X_train, y_train = Xy_split(df_asset.iloc[train,:])\n        X_valid, y_valid = Xy_split(df_asset.iloc[valid,:])\n\n        model = LGBMRegressor(\n            n_estimators = 1000, num_leaves = 500, max_depth = 10, learning_rate = 0.09\n        )\n        model.fit(X_train, y_train)\n        \n        scores.append(corr(y_valid, model.predict(X_valid), np.array([asset.loc[asset_id,'Weight']] * len(y_valid))))\n        \n        X_trains.append(X_train)\n        y_trains.append(y_train)\n        X_valids.append(X_valid)\n        y_valids.append(y_valid)\n        models.append(model)\n        importance = np.append(importance, model.feature_importances_)\n    \n    print('CORR:{}'.format(np.mean(scores)))\n    \n    df_feature = pd.DataFrame({\n        'features' :X_valid.columns,\n        'importance' : np.mean(importance.reshape(n_splits, -1), axis=0)})\n    df_feature = df_feature.sort_values(by='importance',ascending = False)\n\n    plt.figure(figsize=(14, 4))\n    sns.barplot(x='features', y='importance', data=df_feature)\n    plt.title('Feature Importance')\n    plt.tight_layout()\n    plt.show()\n    \n    print('-' * 90)\n    X_train_dict[asset_id] = X_trains\n    y_train_dict[asset_id] = y_trains\n    X_valid_dict[asset_id] = X_valids\n    y_valid_dict[asset_id] = y_valids\n    model_dict[asset_id] = models\n    \n","metadata":{"tags":[],"execution":{"iopub.status.busy":"2022-01-31T14:57:44.027008Z","iopub.execute_input":"2022-01-31T14:57:44.027346Z","iopub.status.idle":"2022-01-31T15:40:56.226602Z","shell.execute_reply.started":"2022-01-31T14:57:44.027303Z","shell.execute_reply":"2022-01-31T15:40:56.225642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### テストデータの予測\nCryptoコンペでは、データと一緒に提供されているgresearch_crypto Python moduleを使用する必要があります。<br>\nこれは、将来のデータを使用して予測を行わないようにするためであるようです。<br>\n[API Introduction](https://www.kaggle.com/sohier/detailed-api-introduction)に記載の通り、predictionを行います。<br>","metadata":{}},{"cell_type":"code","source":"import gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()","metadata":{"execution":{"iopub.status.busy":"2022-01-22T12:31:37.369545Z","iopub.status.idle":"2022-01-22T12:31:37.3699Z","shell.execute_reply.started":"2022-01-22T12:31:37.369721Z","shell.execute_reply":"2022-01-22T12:31:37.369742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test_entire = []\n\n# from the competition tutorial\nfor i, (df_test, sample_prediction_df) in enumerate(iter_test):\n    for j, test_row in df_test.iterrows():\n        try:\n            cur_models = model_dict[test_row['Asset_ID']]\n            X_test = cal_features(test_row).values.reshape(1, -1)\n            y_pred = np.mean([model.predict(X_test) for model in cur_models])\n        except: \n            y_pred = 0.0\n        sample_prediction_df.loc[sample_prediction_df['row_id'] == test_row['row_id'], 'Target'] = y_pred\n    df_test_entire.append(df_test)\n\n    # Send submissions\n    env.predict(sample_prediction_df)","metadata":{"execution":{"iopub.status.busy":"2022-01-22T12:31:37.371528Z","iopub.status.idle":"2022-01-22T12:31:37.371828Z","shell.execute_reply.started":"2022-01-22T12:31:37.371673Z","shell.execute_reply":"2022-01-22T12:31:37.371689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}