{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G-Research Crypto forecasting competition","metadata":{"papermill":{"duration":0.022176,"end_time":"2022-01-03T13:27:50.797544","exception":false,"start_time":"2022-01-03T13:27:50.775368","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### コンペ概要\n\n目的：暗号通貨によるリターンの予測<br>\n暗号通貨に関する時系列の価格履歴を使用して、価格が上がるか下がるか、どれだけの資産がリターンするかを予測します。<br>\n本コンペでは、14種の暗号通貨についてのデータが提供されています。<br>\n\nデータの特性\n- 時系列データ。\n- 暗号通貨の変動は、他の暗号通貨や暗号通貨市場全体に対して影響している。\n- 株価と同様に、通貨の価格はランダムウォークパターンに従っているようにも見える。\n\n<br>","metadata":{"tags":[]}},{"cell_type":"markdown","source":"チュートリアルを参考に、submission提出までをシンプルなモデルを作成します。<br>\n\n**参考**\n- [Tutorial to the G-Research Crypto Competition](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition)\n- [Data Description](https://www.kaggle.com/c/g-research-crypto-forecasting/data)\n- [Detailed API Introduction](https://www.kaggle.com/sohier/detailed-api-introduction)\n\n日本語EDA：[【日本語】G-research_Data_Outline](https://www.kaggle.com/kitopl/g-research-data-outline)<br>","metadata":{}},{"cell_type":"code","source":"import os\nimport random\nimport pandas as pd\nimport numpy as np\nfrom lightgbm import LGBMRegressor","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":1.526291,"end_time":"2022-01-03T13:27:52.390995","exception":false,"start_time":"2022-01-03T13:27:50.864704","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-01-19T06:38:51.142695Z","iopub.execute_input":"2022-01-19T06:38:51.143153Z","iopub.status.idle":"2022-01-19T06:38:52.737395Z","shell.execute_reply.started":"2022-01-19T06:38:51.143004Z","shell.execute_reply":"2022-01-19T06:38:52.736538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### データ概要確認","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv('../input/g-research-crypto-forecasting/train.csv')\ndata.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T06:38:52.739039Z","iopub.execute_input":"2022-01-19T06:38:52.739385Z","iopub.status.idle":"2022-01-19T06:39:30.417493Z","shell.execute_reply.started":"2022-01-19T06:38:52.739344Z","shell.execute_reply":"2022-01-19T06:39:30.416488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **timestamp**：1970-01-01 00:00:00.000 UTCからの経過秒数を示す。データは分スケールのため、60の倍数になっているようです。\n- **Asset_ID**：暗号通貨を識別するID(例：ビットコインの場合はAsset_ID = 1)。 Asset_IDと暗号通貨の対応関係はasset_details.csvで参照できます。\n- **Count**：時間間隔（直前）の取引の総数。\n- **Open**：指定時間間隔における始値(USD)。\n- **High**：指定時間間隔における最高価格(USD)。\n- **Low**：指定時間間隔における最低価格(USD)。\n- **Close**：指定時間間隔における終値(USD)。\n- **Volume**：購入または販売された通貨の数量。\n- **VWAP**：指定時間間隔における通貨の重みづけ平均価格。\n- **Target**：15分間の対数収益率から算出。\ntimestamp、Asset_IDの複合キーが、データを識別するインデックになります。","metadata":{"papermill":{"duration":0.021851,"end_time":"2022-01-03T13:28:55.815829","exception":false,"start_time":"2022-01-03T13:28:55.793978","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T06:39:30.418853Z","iopub.execute_input":"2022-01-19T06:39:30.419193Z","iopub.status.idle":"2022-01-19T06:39:39.121815Z","shell.execute_reply.started":"2022-01-19T06:39:30.419140Z","shell.execute_reply":"2022-01-19T06:39:39.121138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Asset_IDと暗号通貨の対応<br>\nWeightは、目的変数となっているLogReturnを算出する際に使用します。","metadata":{}},{"cell_type":"code","source":"asset_tmp = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nasset = asset_tmp.set_index('Asset_ID').sort_index()\nasset","metadata":{"execution":{"iopub.status.busy":"2022-01-19T06:39:39.123576Z","iopub.execute_input":"2022-01-19T06:39:39.124344Z","iopub.status.idle":"2022-01-19T06:39:39.141519Z","shell.execute_reply.started":"2022-01-19T06:39:39.124301Z","shell.execute_reply":"2022-01-19T06:39:39.140935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 各通貨毎に欠損値、欠損データを確認\nmissing_list = []\nfor asset_id, asset_name in zip(asset.index, asset.loc[:, 'Asset_Name']):\n    asset_df = data[data[\"Asset_ID\"]==asset_id].set_index(\"timestamp\")\n    gap_df = (asset_df.index[1:]-asset_df.index[:-1]).value_counts().reset_index()\n    drop_row = int(((gap_df['index']/60-1)*gap_df['timestamp']).sum())\n    missing_list.append([asset_name, asset_df.shape[0]] + asset_df.isna().sum().tolist() + [drop_row])\n\nmissing_df = pd.DataFrame(missing_list)\nmissing_df.columns = [\"Asset_Name\", \"TotalRows\", 'Missing_Asset_ID', 'Missing_Count', 'Missing_Open', \n                      'Missing_High', 'Missing_Low', 'Missing_Close', 'Missing_Volume', 'Missing_VWAP', 'Missing_Target', 'Missing_Row']\nmissing_df","metadata":{"execution":{"iopub.status.busy":"2022-01-19T06:39:39.142637Z","iopub.execute_input":"2022-01-19T06:39:39.142859Z","iopub.status.idle":"2022-01-19T06:39:44.393366Z","shell.execute_reply.started":"2022-01-19T06:39:39.142829Z","shell.execute_reply":"2022-01-19T06:39:44.392333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 学習\nチュートリアルで紹介されているupper_shadowとlower_shadowと特徴量に加えてみます。<br>\n通貨毎で特徴が異なるので、学習も通貨毎に分けて行います。","metadata":{}},{"cell_type":"code","source":"# チュートリアルより\n# 上ひげ、下ひげを計算\ndef upper_shadow(df):\n    return df['High'] - np.maximum(df['Close'], df['Open'])\n\ndef lower_shadow(df):\n    return np.minimum(df['Close'], df['Open']) - df['Low']\n\ndef data_split_by_asset(df_train, asset_id):\n    df_asset = data[data[\"Asset_ID\"] == asset_id]\n    \n    df_asset['Upper_Shadow'] = upper_shadow(df_asset)\n    df_asset['Lower_Shadow'] = lower_shadow(df_asset)\n    df_asset = df_asset.dropna(how='any')\n    \n    X = df_asset.drop(['timestamp', 'Asset_ID', 'Target'], axis=1)\n    y = df_asset['Target']\n\n    model = LGBMRegressor(n_estimators=10)\n    model.fit(X, y)\n    return X, y, model","metadata":{"execution":{"iopub.status.busy":"2022-01-19T06:39:44.394878Z","iopub.execute_input":"2022-01-19T06:39:44.395294Z","iopub.status.idle":"2022-01-19T06:39:44.403041Z","shell.execute_reply.started":"2022-01-19T06:39:44.395242Z","shell.execute_reply":"2022-01-19T06:39:44.402122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_dict = {}\ny_dict = {}\nmodel_dict = {}\n\nfor asset_id, asset_name in zip(asset.index, asset['Asset_Name']):\n    print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n    X, y, model = data_split_by_asset(data, asset_id)    \n    X_dict[asset_id] = X\n    y_dict[asset_id] = y\n    model_dict[asset_id] = model","metadata":{"execution":{"iopub.status.busy":"2022-01-19T06:39:44.404448Z","iopub.execute_input":"2022-01-19T06:39:44.404829Z","iopub.status.idle":"2022-01-19T06:40:07.282724Z","shell.execute_reply.started":"2022-01-19T06:39:44.404787Z","shell.execute_reply":"2022-01-19T06:40:07.281993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### テストデータの特徴量取得、予測\nCryptoコンペでは、データと一緒に提供されているgresearch_crypto Python moduleを使用する必要があります。<br>\nこれは、将来のデータを使用して予測を行わないようにするためであるようです。<br>\n[API Introduction](https://www.kaggle.com/sohier/detailed-api-introduction)に記載の通り、predictionを行います。<br>","metadata":{}},{"cell_type":"code","source":"import gresearch_crypto\n\nenv = gresearch_crypto.make_env()\niter_test = env.iter_test()","metadata":{"execution":{"iopub.status.busy":"2022-01-19T06:40:07.286025Z","iopub.execute_input":"2022-01-19T06:40:07.288145Z","iopub.status.idle":"2022-01-19T06:40:07.308499Z","shell.execute_reply.started":"2022-01-19T06:40:07.288099Z","shell.execute_reply":"2022-01-19T06:40:07.307504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test_entire = []\n\nfor i, (df_test, sample_prediction_df) in enumerate(iter_test):\n    # Asset_IDでモデルを分割しているため、1データ毎で予測\n    for j, test_row in df_test.iterrows():\n\n        model = model_dict[test_row['Asset_ID']]\n        test_row['Upper_Shadow'] = upper_shadow(test_row)\n        test_row['Lower_Shadow'] = lower_shadow(test_row)\n\n        X_test = test_row.drop(['timestamp', 'Asset_ID', 'row_id']).values.reshape(1, -1)\n        y_pred = model.predict(X_test)\n\n        sample_prediction_df.loc[sample_prediction_df['row_id'] == test_row['row_id'], 'Target'] = y_pred\n\n    df_test_entire.append(df_test)\n\n    # submissions提出\n    env.predict(sample_prediction_df)","metadata":{"execution":{"iopub.status.busy":"2022-01-19T06:40:07.310106Z","iopub.execute_input":"2022-01-19T06:40:07.310648Z","iopub.status.idle":"2022-01-19T06:40:07.501555Z","shell.execute_reply.started":"2022-01-19T06:40:07.310590Z","shell.execute_reply":"2022-01-19T06:40:07.500824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}