{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## **0. はじめに**\n\nコンペの概要と注意点を自分用にまとめます\n\n## **1. reference**\n* [Detailed API Introduction](https://www.kaggle.com/sohier/detailed-api-introduction)\n* [pandas, Matplotlib（mplfinance）でローソク足チャートを作成](https://note.nkmk.me/python-pandas-matplotlib-candlestick-chart/)\n* [OHLC Charts in Python](https://plotly.com/python/ohlc-charts/)\n\n## **2. 注意点**\n\n普段のコンペと違い、APIからtest用のデータを取得し、APIを使ってmodelの評価結果を提出する。<br>\nざっくりとしたコードの全体像は、上記のリンクに記載あり。簡単にまとめると、\n```\nimport gresearch_crypto   　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　##<= 今回のコンペ用python library\nenv = gresearch_crypto.make_env()   　　　　　　　　　　##<= 初期化。\n\n# Training data is in the competition dataset as usual\ntrain_df = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv', low_memory=False)\ntgt_1_model.fit(train_df)　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　#<= いつもどおりmodelを作成。\ntgt_2_model.fit(train_df)\niter_test = env.iter_test()　　　　　　　　　　　　　　　　　　　　　　　　　　　##<= テスト用データをapi経由で取得。\nfor (test_df, sample_prediction_df) in iter_test:\n    sample_prediction_df['Target'] = tgt_1_model.predict(test_df) ##<= モデルを通して提出用データ作成\n    env.predict(sample_prediction_df)   ##<= 提出！！\n```\n\n大体上記の流れになりそう。<br>\n外部マシンを使ったモデルを読み込んだり、いろんなモデルを試せるように、<br>\nEDAで一つのnotebook、model作成で一つのnotebook、他のnotebookのモデルを読み込んで提出するだけのnotebookの構成が良さそう。[A.I](↑一つのnotebookじゃなくてもよいか要確認。)<br>\n\n","metadata":{}},{"cell_type":"markdown","source":"## **3. Libraryインストールと読み込み**\n\nmplfinance is usefull for creating candle chart.","metadata":{}},{"cell_type":"code","source":"! pip install mplfinance\nimport gresearch_crypto\nimport pandas as pd\nimport numpy as np\nimport tqdm\nimport mplfinance as mpf","metadata":{"execution":{"iopub.status.busy":"2021-11-03T11:35:40.471743Z","iopub.execute_input":"2021-11-03T11:35:40.472194Z","iopub.status.idle":"2021-11-03T11:35:51.392628Z","shell.execute_reply.started":"2021-11-03T11:35:40.472142Z","shell.execute_reply":"2021-11-03T11:35:51.391648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **4.1 train data overview**","metadata":{}},{"cell_type":"code","source":"import gresearch_crypto\n#%%time\n## 読み込みに割と時間がかかるので、時間を確認。\n\n## 専用library(API)を使って初期化。一度初期化。\n#env = gresearch_crypto.make_env()\n\n## 読み込み\ntrain_df = pd.read_csv('../input/g-research-crypto-forecasting/train.csv', low_memory=False, \n                       dtype={'Asset_ID': 'int8', 'Count': 'int32', 'row_id': 'int32', 'Count': 'int32', \n                              'Open': 'float64', 'High': 'float64', 'Low': 'float64', 'Close': 'float64', \n                              'Volume': 'float64', 'VWAP': 'float64'\n                             }\n                      )\ntrain_df[\"timestamp\"] = pd.to_datetime(train_df['timestamp'].astype(int), unit='s')\ntrain_df.set_index(\"timestamp\",inplace=True)\ntrain_df.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T11:35:51.394927Z","iopub.execute_input":"2021-11-03T11:35:51.395244Z","iopub.status.idle":"2021-11-03T11:37:06.325352Z","shell.execute_reply.started":"2021-11-03T11:35:51.395198Z","shell.execute_reply":"2021-11-03T11:37:06.324505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.tail(3)","metadata":{"execution":{"iopub.status.busy":"2021-11-03T11:37:06.327372Z","iopub.execute_input":"2021-11-03T11:37:06.327876Z","iopub.status.idle":"2021-11-03T11:37:06.343573Z","shell.execute_reply.started":"2021-11-03T11:37:06.327831Z","shell.execute_reply":"2021-11-03T11:37:06.342768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-03T11:37:06.34593Z","iopub.execute_input":"2021-11-03T11:37:06.346221Z","iopub.status.idle":"2021-11-03T11:37:06.357368Z","shell.execute_reply.started":"2021-11-03T11:37:06.346187Z","shell.execute_reply":"2021-11-03T11:37:06.356537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 通貨は１４種類？","metadata":{}},{"cell_type":"code","source":"train_df[\"Asset_ID\"].unique()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T11:37:06.358771Z","iopub.execute_input":"2021-11-03T11:37:06.359363Z","iopub.status.idle":"2021-11-03T11:37:06.528552Z","shell.execute_reply.started":"2021-11-03T11:37:06.359309Z","shell.execute_reply":"2021-11-03T11:37:06.527964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **4.1.1 Is timestep 60[s]?**\n\nThe diff() method does not work for index..","metadata":{}},{"cell_type":"code","source":"train_df[\"tmp_dt\"] = train_df.index\ntrain_df[train_df[\"Asset_ID\"] == 0][\"tmp_dt\"].diff().value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T11:44:03.587077Z","iopub.execute_input":"2021-11-03T11:44:03.587777Z","iopub.status.idle":"2021-11-03T11:44:04.039106Z","shell.execute_reply.started":"2021-11-03T11:44:03.587736Z","shell.execute_reply":"2021-11-03T11:44:04.038306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **4.2 asset details over view**\n\nweightってなんやねんっ。後で調べる。","metadata":{}},{"cell_type":"code","source":"asset_df = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\nasset_df","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:31:28.981561Z","iopub.execute_input":"2021-11-03T09:31:28.982376Z","iopub.status.idle":"2021-11-03T09:31:29.016506Z","shell.execute_reply.started":"2021-11-03T09:31:28.982324Z","shell.execute_reply":"2021-11-03T09:31:29.015667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **4.3 supplemental train over view**","metadata":{}},{"cell_type":"code","source":"supplemental_df = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv')\nsupplemental_df[\"timestamp\"] = pd.to_datetime(supplemental_df['timestamp'].astype(int), unit='s')\nsupplemental_df.set_index(\"timestamp\",inplace=True)\nsupplemental_df","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:33:52.120647Z","iopub.execute_input":"2021-11-03T09:33:52.121003Z","iopub.status.idle":"2021-11-03T09:33:55.530275Z","shell.execute_reply.started":"2021-11-03T09:33:52.120964Z","shell.execute_reply":"2021-11-03T09:33:55.529386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **5.1 Visualize by plotly**\n普段plotlyをよく使うので。<br>\n表示範囲を絞れるので、便利。<br>\ndfのindexは、datetimeオブジェクトにしておく必要がある。<br>","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/g-research-crypto-forecasting/train.csv', low_memory=False, \n                       dtype={'Asset_ID': 'int8', 'Count': 'int32', 'row_id': 'int32', 'Count': 'int32', \n                              'Open': 'float64', 'High': 'float64', 'Low': 'float64', 'Close': 'float64', \n                              'Volume': 'float64', 'VWAP': 'float64'\n                             }\n                      )\n\nasset_id = 1\ntgt_df = train_df[train_df[\"Asset_ID\"] == asset_id]\ntgt_df = tgt_df[['Open', 'High', 'Low', 'Close', 'Volume','timestamp']]  ## ohlcvと時間の情報のみ必要。\ntgt_df[\"timestamp\"] = pd.to_datetime(tgt_df['timestamp'].astype(int), unit='s')\ntgt_df.set_index(\"timestamp\",inplace=True)\ntgt_df","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:35:39.887032Z","iopub.execute_input":"2021-11-03T09:35:39.887864Z","iopub.status.idle":"2021-11-03T09:35:40.26649Z","shell.execute_reply.started":"2021-11-03T09:35:39.887813Z","shell.execute_reply":"2021-11-03T09:35:40.26543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go\n\ntgt_df = tgt_df[0:1000]\nfig = go.Figure(data=[go.Candlestick(x=tgt_df.index, open=tgt_df['Open'], high=tgt_df['High'], low=tgt_df['Low'], close=tgt_df['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:17:29.923418Z","iopub.execute_input":"2021-11-03T09:17:29.923735Z","iopub.status.idle":"2021-11-03T09:17:30.118681Z","shell.execute_reply.started":"2021-11-03T09:17:29.923694Z","shell.execute_reply":"2021-11-03T09:17:30.117739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **5.2.1 Resampling \"change period\" to daily OHLC**\n\nresumple train_df to daily ohlc data","metadata":{}},{"cell_type":"code","source":"## Re read original data\nasset_id = 1\ntgt_df = train_df[train_df[\"Asset_ID\"] == asset_id]\ntgt_df = tgt_df[['Open', 'High', 'Low', 'Close', 'Volume','timestamp']]  ## ohlcvと時間の情報のみ必要。\ntgt_df[\"timestamp\"] = pd.to_datetime(tgt_df['timestamp'].astype(int), unit='s')\ntgt_df.set_index(\"timestamp\",inplace=True)\n\n","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:17:30.120453Z","iopub.execute_input":"2021-11-03T09:17:30.121017Z","iopub.status.idle":"2021-11-03T09:17:30.442996Z","shell.execute_reply.started":"2021-11-03T09:17:30.12097Z","shell.execute_reply":"2021-11-03T09:17:30.441985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## resample\nd_ohlcv = {'Open': 'first',\n           'High': 'max',\n           'Low': 'min',\n           'Close': 'last',\n           'Volume': 'sum'}\ntgt_df_daily = tgt_df.resample('D').agg(d_ohlcv)  #< \"D\" mean daily. \ntgt_df_daily = tgt_df_daily[0:1000]\nfig = go.Figure(data=[go.Candlestick(x=tgt_df_daily.index, open=tgt_df_daily['Open'], high=tgt_df_daily['High'], low=tgt_df_daily['Low'], close=tgt_df_daily['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:17:30.444306Z","iopub.execute_input":"2021-11-03T09:17:30.444565Z","iopub.status.idle":"2021-11-03T09:17:30.568405Z","shell.execute_reply.started":"2021-11-03T09:17:30.444535Z","shell.execute_reply":"2021-11-03T09:17:30.567725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **5.2.1 Resampling to 15 minutes OHLC**","metadata":{}},{"cell_type":"code","source":"d_ohlcv = {'Open': 'first',\n           'High': 'max',\n           'Low': 'min',\n           'Close': 'last',\n           'Volume': 'sum'}\ntgt_df_daily = tgt_df.resample('15min').agg(d_ohlcv)  #< 15 minutes. \ntgt_df_daily = tgt_df_daily[0:1000]\nfig = go.Figure(data=[go.Candlestick(x=tgt_df_daily.index, open=tgt_df_daily['Open'], high=tgt_df_daily['High'], low=tgt_df_daily['Low'], close=tgt_df_daily['Close'])])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:17:30.569729Z","iopub.execute_input":"2021-11-03T09:17:30.570186Z","iopub.status.idle":"2021-11-03T09:17:30.693011Z","shell.execute_reply.started":"2021-11-03T09:17:30.570144Z","shell.execute_reply":"2021-11-03T09:17:30.692332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **5.3 mplfinanceを使ってみる**\n\n**　注意 ** <br>\nmpfに渡すDFには、ohlcvの情報と、indexは、datetimeオブジェクトにしておく必要がある。<br>\nmpf require \"OHLCV\" columns and index is datetime object.\n\nUnn, no good.","metadata":{}},{"cell_type":"code","source":"mpf.plot(tgt_df[:100],type='candle')","metadata":{"execution":{"iopub.status.busy":"2021-11-03T09:17:30.695058Z","iopub.execute_input":"2021-11-03T09:17:30.695487Z","iopub.status.idle":"2021-11-03T09:17:31.007915Z","shell.execute_reply.started":"2021-11-03T09:17:30.695454Z","shell.execute_reply":"2021-11-03T09:17:31.007233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **6. look into test data**\n\ntestデータはapi経由で取得する。未来のデータを使って評価するため、将来データを受け取れる仕組みが必要なため。<br>","metadata":{}},{"cell_type":"code","source":"env = gresearch_crypto.make_env()\niter_test = env.iter_test()\n(test_df, sample_prediction_df) = next(iter_test)\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-03T12:22:44.493278Z","iopub.execute_input":"2021-11-03T12:22:44.493862Z","iopub.status.idle":"2021-11-03T12:22:44.628644Z","shell.execute_reply.started":"2021-11-03T12:22:44.493614Z","shell.execute_reply":"2021-11-03T12:22:44.627337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"asset_id = 1\ntgt_df = test_df[test_df[\"Asset_ID\"] == asset_id]\ntgt_df[\"timestamp\"] = pd.to_datetime(tgt_df['timestamp'].astype(int), unit='s')\ntgt_df.set_index(\"timestamp\",inplace=True)\ntgt_df","metadata":{},"execution_count":null,"outputs":[]}]}