{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Crypto forecasting tutorial / 暗号通貨予測のチュートリアル**","metadata":{"id":"8PNZS8gF3CVN"}},{"cell_type":"markdown","source":"参考:https://www.kaggle.com/junichiromorita/tutorial-to-the-g-research-crypto-competition-jpn","metadata":{}},{"cell_type":"markdown","source":"# G-Research Crypto forecasting competition / G-Research 暗号通貨予測コンペティション\n\nIn the [G-Research Crypto forecasting competition](link), participants have the challenge to predict  price returns across a bundle of major cryptocurrencies. To facilitate your participation, we have created this tutorial notebook covering some relevant concepts for the crypto forecasting challenge.\n\nG-Research 暗号通貨予測コンペティションでは、参加者は主要な暗号通貨群の価格リターンを予測するという課題が与えれれます。参加しやすいように、このチュートリアルノートでは、暗号通貨予測コンペティションに関連するコンセプトを説明しています。\n\nThe notebook presents an introduction to crypto forecasting, describing the structure and elements of the dataset, some relevant statistical properties, as well as building a couple of ML baseline models and providing an example code submission.\n\nこのノートブックでは、暗号通貨予測について紹介し、データセットの構造と要素、関連する統計的特性を説明するとともに、いくつかの機械学習ベースラインモデルを構築し、コード投稿の例を示しています。","metadata":{"id":"DO0FyA3q6rTl"}},{"cell_type":"markdown","source":"## The Cryptocurrency Market / 暗号通貨市場について\n\nFirst, a quick introduction to the crypto world. Cryptocurrencies have become an extremely popular and volatile market, delivering massive returns (as well as losses) to investors. Thousands of cryptocurrencies have been created with a few major ones that many of you will have heard of including Bitcoin (BTC), Ether (ETH) or Dogecoin (DOGE).\n\nまず、暗号通貨の世界について簡単に紹介します。暗号通貨は非常に人気が高く、変動の激しい市場となっており、投資家に多額のリターン（損失も含む）をもたらしています。何千もの暗号通貨が作られていますが、主なものとしては、ビットコイン（BTC）、イーサ（ETH）、ドージコイン（DOGE）などがあり、多くの人が聞いたことがあるでしょう。\n\nCryptocurrencies are traded extensively across crypto-exchanges, with an average volume of $41 billion traded daily over the last year, according to CryptoCompare (as of 25th July 2021). \n\n暗号通貨は暗号通貨取引所で広く取引されており、CryptoCompareによると、昨年1年間で毎日平均410億ドルの取引が行われています（2021年7月25日現在）。\n\nChanges in prices between different cryptocurrencies are highly interconnected. For example, Bitcoin has historically been a major driver of price changes across cryptocurrencies but other coins also impact the market.  \n\n異なる暗号通貨間の価格の変化は、高度に相互に関連しています。例えば、ビットコインは歴史的に暗号通貨間の価格変動の主な要因となってきましたが、他のコインも市場に影響を与えます。","metadata":{"id":"x3HpXt5v-ko4"}},{"cell_type":"markdown","source":"\n## Forecasting returns / リターンの予測\n\nA fundamental task in financial modeling is predicting how prices will behave in the near future. Using the time-series of historical prices as training data, we want to predict if prices will go up or down, and by how much, namely the asset *returns*.\n\n金融モデリングの基本的な課題は、近い将来の価格がどのように推移するかを予測することです。過去の価格の時系列を学習データとして、価格が上がるのか下がるのか、どれくらい上がるのかを予測したい、つまり資産のリターンを予測したいのです。\n\nIn this competition, Kagglers are challenged to build machine learning models to predict the returns of 14 popular cryptocurrencies, in the time scale of minutes to hours. You will have access to millions of rows of minute-by-minute cryptocurrency trading data, with which you'll design your forecasting models for all 14 assets simultaneously. Your predictions will be evaluated by how much they correlate with real market data collected during the three-month evaluation period after the competition has closed. \n\n本コンペティションでは、人気のある14種類の暗号通貨のリターンを予測する機械学習モデルを、数分から数時間の時間スケールで構築することがKagglersの課題となっています。数百万行の分刻みの暗号通貨取引データにアクセスし、それをもとに14の資産すべてを同時に対象とした予測モデルを設計することになります。あなたの予測は、コンテスト終了後の3ヶ月間の評価期間中に収集された実際の市場データとどれだけ相関しているかによって評価されます。\n\nCryptocurrency returns prediction remains an open and extremely challenging forecasting task. This is a fascinating problem domain for the ML community given the extreme volatility of the assets, the non-stationary nature of the data, the market and meme manipulation, the correlation between assets and the very fast changing market conditions. We hope you find it as fascinating as we do! \n\n暗号通貨のリターン予測は、依然として未解決で、非常に困難な予測課題です。これは、資産の極端な変動性、データの非定常性、市場や操作、資産間の相関性、非常に速い市場状況の変化を考えると、機械学習コミュニティにとって魅力的な問題領域です。私たちと同じように、皆さんにも魅力的な問題を見つけていただきたいと思います。","metadata":{"id":"fDAWC9JNZITK"}},{"cell_type":"markdown","source":"# Dataset description / データセットの説明\n\nNow, let's dive into the data! We start by loading the competition's dataset and inspecting its basic properties. \n\nそれでは、データを見てみましょう。まずは、コンペティションのデータセットを読み込んで、その基本的なプロパティを確認します。","metadata":{"id":"4e6OA0qc6x9h"}},{"cell_type":"markdown","source":"## Load the training set / トレーニングセットの読み込み","metadata":{"id":"OevdFjmtar0j"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom datetime import datetime","metadata":{"id":"90fqnpNQ2wd8","execution":{"iopub.status.busy":"2021-12-12T11:19:45.893203Z","iopub.execute_input":"2021-12-12T11:19:45.893609Z","iopub.status.idle":"2021-12-12T11:19:45.907846Z","shell.execute_reply.started":"2021-12-12T11:19:45.893501Z","shell.execute_reply":"2021-12-12T11:19:45.906774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import lightgbm as lgbm","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:19:45.909683Z","iopub.execute_input":"2021-12-12T11:19:45.909934Z","iopub.status.idle":"2021-12-12T11:19:48.175314Z","shell.execute_reply.started":"2021-12-12T11:19:45.909904Z","shell.execute_reply":"2021-12-12T11:19:48.174478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_folder = \"../input/g-research-crypto-forecasting/\"\n!ls $data_folder","metadata":{"id":"Zw_oi4NETkKv","outputId":"e6fe292d-721e-4c6a-bc70-9026cfd5b915","execution":{"iopub.status.busy":"2021-12-12T11:19:48.176668Z","iopub.execute_input":"2021-12-12T11:19:48.176992Z","iopub.status.idle":"2021-12-12T11:19:48.974638Z","shell.execute_reply.started":"2021-12-12T11:19:48.176960Z","shell.execute_reply":"2021-12-12T11:19:48.973578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df = pd.read_csv(data_folder + 'train.csv')","metadata":{"id":"La0-ivRAUopA","scrolled":true,"execution":{"iopub.status.busy":"2021-12-12T11:19:48.975950Z","iopub.execute_input":"2021-12-12T11:19:48.976230Z","iopub.status.idle":"2021-12-12T11:20:52.496029Z","shell.execute_reply.started":"2021-12-12T11:19:48.976196Z","shell.execute_reply":"2021-12-12T11:20:52.495188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df.head(10)","metadata":{"id":"14ZSPE6daadW","outputId":"b6a4f262-4071-4c2f-dd42-12a253452297","execution":{"iopub.status.busy":"2021-12-12T11:20:52.498334Z","iopub.execute_input":"2021-12-12T11:20:52.498617Z","iopub.status.idle":"2021-12-12T11:20:52.528470Z","shell.execute_reply.started":"2021-12-12T11:20:52.498585Z","shell.execute_reply":"2021-12-12T11:20:52.527615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that each row of the data set has the trading data for an asset, at a given minute timestamp, described in detail below. \n\nデータセットの各行には、特定の分のタイムスタンプにおける資産の取引データがあることがわかります。","metadata":{"id":"ZUPxDsW8bCh1"}},{"cell_type":"markdown","source":"## Data features / データの特徴\n\nWe can see the different features included in the dataset. Specifically, the features included per asset are the following:\n\nデータセットに含まれる様々な特徴を見ることができます。具体的には、アセットごとに含まれている特徴は以下の通りです。\n\n*   **timestamp**: All timestamps are returned as second Unix timestamps (the number of seconds elapsed since 1970-01-01 00:00:00.000 UTC). Timestamps in this dataset are multiple of 60, indicating minute-by-minute data. / すべてのタイムスタンプは、秒単位のUnixタイムスタンプ（1970-01-01 00:00:00.000 UTCからの経過秒数）で返されます。このデータセットのタイムスタンプは60の倍数で、1分ごとのデータを示しています。\n*   **Asset_ID**: The asset ID corresponding to one of the crytocurrencies (e.g. `Asset_ID = 1` for Bitcoin). The mapping from `Asset_ID` to crypto asset is contained in `asset_details.csv`. / いずれかの暗号通貨に対応するアセットID（例：Bitcoinの場合は`Asset_ID = 1`）。`Asset_ID`から暗号資産へのマッピングは`asset_details.csv`に含まれる。\n*   **Count**: Total number of trades in the time interval (last minute). / 時間間隔（最後の1分）での取引の総数。\n*   **Open**:\tOpening price of the time interval (in USD). / 時間間隔内の始値（単位：米ドル）。\n*   **High**:\tHighest price reached during time interval (in USD). / 時間間隔内の高値（単位：米ドル）。\n*   **Low**: Lowest price reached during time interval (in USD). / 時間間隔内の安値（単位：米ドル）。\n*   **Close**:\tClosing price of the time interval (in USD). / 時間間隔内の終値（米ドル）。\n*   **Volume**:\tQuantity of asset bought or sold, displayed in base currency USD. / 買ったまたは売った資産の量で、基本通貨の米ドルで表示されます。\n*   **VWAP**: The average price of the asset over the time interval, weighted by volume. VWAP is an aggregated form of trade data. / 取引時間中の平均価格を取引量で加重平均したもの。VWAPは取引データを集計したものです。\n*   **Target**: Residual log-returns for the asset over a 15 minute horizon. / 15分間の暗号資産の残余対数リターン。\n\nThe first two columns define the time and asset indexes for this data row. The 6 middle columns are feature columns with the trading data for this asset and minute in time. The last column is prediction target, which we will get to later in more detail.\n\n最初の2列は、このデータ行の時間と資産のインデックスを定義しています。中央の6列は、この資産と時間の分の取引データを含む特徴的な列です。最後の列は予測ターゲットですが、これについては後で詳しく説明します。\n\nWe also view the asset information, including the list of all assets, the `Asset_ID` to asset mapping, and the weight of each asset used to weigh their relative importance in the evaluation metric.\n\nまた、すべてのアセットのリスト、アセット_IDとアセットのマッピング、評価指標における相対的な重要性を評価するために使用される各アセットの重みなど、アセット情報も表示されます。","metadata":{"id":"YjN-TfBUx2ml"}},{"cell_type":"code","source":"asset_details = pd.read_csv(data_folder + 'asset_details.csv')\nasset_details","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:20:52.529644Z","iopub.execute_input":"2021-12-12T11:20:52.529892Z","iopub.status.idle":"2021-12-12T11:20:52.548696Z","shell.execute_reply.started":"2021-12-12T11:20:52.529859Z","shell.execute_reply":"2021-12-12T11:20:52.547732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing / 前処理","metadata":{"id":"S-oceNYiivUi"}},{"cell_type":"markdown","source":"## Dealing with missing data / 欠損データの処理\n","metadata":{"id":"K9ce7eWddHfM"}},{"cell_type":"markdown","source":"Let us inspect the data for another important asset, Ethereum.\n\nもう一つの重要な暗号資産であるEthereumのデータを調べてみましょう。","metadata":{"id":"AOpyJzKgFuYb"}},{"cell_type":"code","source":"crypto_df['date'] = crypto_df['timestamp'].apply(lambda x: datetime.fromtimestamp(x))","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:20:52.550220Z","iopub.execute_input":"2021-12-12T11:20:52.550928Z","iopub.status.idle":"2021-12-12T11:21:30.292399Z","shell.execute_reply.started":"2021-12-12T11:20:52.550893Z","shell.execute_reply":"2021-12-12T11:21:30.291509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# NaNとなっているTargetを学習データから除外した\ncrypto_df = crypto_df[(crypto_df['Target']>0)|(crypto_df['Target']<=0)]","metadata":{"id":"cuWT1Suxzjvb","outputId":"04ff726c-8858-4d87-b592-ee1b036f0996","execution":{"iopub.status.busy":"2021-12-12T11:21:30.293835Z","iopub.execute_input":"2021-12-12T11:21:30.294185Z","iopub.status.idle":"2021-12-12T11:21:31.649319Z","shell.execute_reply.started":"2021-12-12T11:21:30.294139Z","shell.execute_reply":"2021-12-12T11:21:31.648226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data visualisation / データの可視化\n\nWe  will start by visualising the Close prices for the two assets we have selected.\n\n選択した2つの暗号資産の終値を可視化してみます。","metadata":{"id":"_TcwTqisePHB"}},{"cell_type":"code","source":"# import matplotlib.pyplot as plt\n\n# # plot vwap time series for both chosen assets\n# f = plt.figure(figsize=(15,4))\n\n# # fill missing values for BTC\n# btc = btc.reindex(range(btc.index[0],btc.index[-1]+60,60),method='pad')\n\n# ax = f.add_subplot(121)\n# plt.plot(btc['Close'], label='BTC')\n# plt.legend()\n# plt.xlabel('Time')\n# plt.ylabel('Bitcoin')\n\n# ax2 = f.add_subplot(122)\n# ax2.plot(eth['Close'], color='red', label='ETH')\n# plt.legend()\n# plt.xlabel('Time')\n# plt.ylabel('Ethereum')\n\n# plt.tight_layout()\n# plt.show()","metadata":{"id":"PtVHC0YTxq-l","outputId":"525bc0d9-9673-42d0-ab73-aadc5e6afdad","execution":{"iopub.status.busy":"2021-12-12T11:21:31.650951Z","iopub.execute_input":"2021-12-12T11:21:31.652201Z","iopub.status.idle":"2021-12-12T11:21:31.656656Z","shell.execute_reply.started":"2021-12-12T11:21:31.652153Z","shell.execute_reply":"2021-12-12T11:21:31.655792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The assets have quite different history, but we could check if they correlate in recent times.\n\n両暗号資産の歴史は全く異なりますが、最近では相関関係があることを確認できました。","metadata":{"id":"ECfuzVHyzEF2"}},{"cell_type":"code","source":"# import time\n\n# # auxiliary function, from datetime to timestamp\n# totimestamp = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))\n\n# # create intervals\n# btc_mini_2021 = btc.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]\n# eth_mini_2021 = eth.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]","metadata":{"id":"mcfpDT6vNzpT","execution":{"iopub.status.busy":"2021-12-12T11:21:31.658074Z","iopub.execute_input":"2021-12-12T11:21:31.659203Z","iopub.status.idle":"2021-12-12T11:21:31.672131Z","shell.execute_reply.started":"2021-12-12T11:21:31.659152Z","shell.execute_reply":"2021-12-12T11:21:31.671414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # plot time series for both chosen assets\n# f = plt.figure(figsize=(7,8))\n\n# ax = f.add_subplot(211)\n# plt.plot(btc_mini_2021['Close'], label='btc')\n# plt.legend()\n# plt.xlabel('Time')\n# plt.ylabel('Bitcoin Close')\n\n# ax2 = f.add_subplot(212)\n# ax2.plot(eth_mini_2021['Close'], color='red', label='eth')\n# plt.legend()\n# plt.xlabel('Time')\n# plt.ylabel('Ethereum Close')\n\n# plt.tight_layout()\n# plt.show()","metadata":{"id":"EfZiXb74zQsK","outputId":"5039e885-cb14-487f-fe06-3a30e4cf7270","execution":{"iopub.status.busy":"2021-12-12T11:21:31.673440Z","iopub.execute_input":"2021-12-12T11:21:31.674232Z","iopub.status.idle":"2021-12-12T11:21:31.689956Z","shell.execute_reply.started":"2021-12-12T11:21:31.674184Z","shell.execute_reply":"2021-12-12T11:21:31.688879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On shorter intervals we can visually see some potential correlation between both assets, with some simultaneous ups and downs. A better format for analyzing such movements is by calculating asset returns. \n\nもっと短い間隔で見ると、両暗号資産の間には潜在的な相関関係があり、同時にいくつかの上昇と下降があることが視覚的にわかります。このような動きを分析するには、資産のリターンを計算するのが良いでしょう。","metadata":{"id":"O2-jEU3wPnNV"}},{"cell_type":"markdown","source":"# Building your prediction model / 予測モデルの構築","metadata":{"id":"EaIGlwUB9kPy"}},{"cell_type":"markdown","source":"## Prediction targets and evaluation / 予測対象と評価","metadata":{"id":"9Akm2RTz-rOH"}},{"cell_type":"markdown","source":"This forecasting competition aims to predict returns in the near future for prices $P^a$, for each asset $a$. For each row in the dataset, we include the target for prediction, `Target`. `Target` is derived from log returns ($R^a$) over 15 minutes.\n\nこの予測コンペティションは、各資産$a$について、価格$P^a$に対する近未来のリターンを予測することを目的としています。データセットの各行には、予測のターゲットである`Target`が含まれる。`Target`は15分間の対数収益率($R^a$)から得られる。\n\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$\n\nCrypto asset returns are highly correlated, following to a large extend the overall crypto market. As we want to test your ability to predict returns for individual assets, we perform a linear residualization, removing the market signal from individual asset returns when creating the two targets. In more detail, if $M(t)$ is the weighted average market returns, the target is:\n\n暗号資産のリターンは非常に相関性が高く、暗号市場全体を大きくフォローしています。個々の資産のリターンを予測する能力をテストしたいため、2つのターゲットを作成する際には、個々の資産のリターンから市場のシグナルを除去し、線形残差を行います。より詳細には、$M(t)$が加重平均市場リターンである場合、ターゲットは\n\n$$M(t) = \\frac{\\sum_a w^a R^a(t)}{\\sum_a w^a}  \\\\\n\\beta^a = \\frac{\\langle M \\cdot R^a \\rangle}{\\langle M^2 \\rangle} \\\\\n\\text{Target}^a(t) = R^a(t) - \\beta^a M(t)$$\n\nwhere the bracket $\\langle .\\rangle$ represent the rolling average over time (3750 minute windows). \n\nここで、括弧内の$\\langle .\\rangle$は、時間（3750分窓）でのローリング平均を表しています。\n\nSome rows have null values for targets due to missing values in future prices. Rows with nulls in the test set ground truth are ignored for scoring purposes.\n\n一部の行では、将来の価格の値が欠落しているため、ターゲットの値が欠損値になっています。テストセットのに欠損値がある行は、スコアリングのために無視されます。\n\nIn the competition, your predictions will be evaluated on a weighted version of the Pearson correlation coefficient, with weights given by the `Weight` column in the Asset Details file.\n\nコンペティションでは、あなたの予測は、ピアソン相関係数の加重バージョンで評価されます。加重は、資産詳細ファイルの Weight 列で与えられます。\n\nIn this tutorial, we will simplify things and use correlation (without weights) for evaluation, and consider only two assets, BTC and ETH.\n\nこのチュートリアルでは、物事を単純化して、相関関係（重みなし）を評価に使用し、BTC と ETH の 2 つの資産のみを検討します。","metadata":{"id":"hl29F0dC9mWc"}},{"cell_type":"markdown","source":"## Feature design / 特徴量設計\n\nWe first design a few relevant features to input to our model.\n\nまず、モデルに入力するための関連する特徴量をいくつか設計します。","metadata":{"id":"Motsk80VF7kj"}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Select some input features from the trading data: \n# # 5 min log return, abs(5 min log return), upper shadow, and lower shadow.\n# upper_shadow = lambda asset: asset.High - np.maximum(asset.Close,asset.Open)\n# lower_shadow = lambda asset: np.minimum(asset.Close,asset.Open)- asset.Low\n\n# X_btc = pd.concat([log_return(btc.VWAP,periods=5), log_return(btc.VWAP,periods=1).abs(), \n#                upper_shadow(btc), lower_shadow(btc)], axis=1)\n# y_btc = btc.Target\n\n# X_eth = pd.concat([log_return(eth.VWAP,periods=5), log_return(eth.VWAP,periods=1).abs(), \n#                upper_shadow(eth), lower_shadow(eth)], axis=1)\n# y_eth = eth.Target","metadata":{"id":"2kQz1YDeqsch","execution":{"iopub.status.busy":"2021-12-12T11:21:31.691538Z","iopub.execute_input":"2021-12-12T11:21:31.691948Z","iopub.status.idle":"2021-12-12T11:21:31.701421Z","shell.execute_reply.started":"2021-12-12T11:21:31.691906Z","shell.execute_reply":"2021-12-12T11:21:31.700369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preparing the data for building predictive models / 予測モデル構築のためのデータの準備\n\nAs we will train linear regression parameters, we need to separate training and test sets. To do so, we will compute X and y and split this data into train and test splits. Note that the test split represents a later part of the data, as it is commonly done in time series. \n\n線形回帰パラメータをトレーニングするので、トレーニングセットとテストセットを分ける必要があります。そのために、XとYを計算し、このデータを訓練用とテスト用に分割します。時系列でよく行われるように、テスト分割はデータの後の部分を使用していることに注意してください。","metadata":{"id":"26locv57nOiP"}},{"cell_type":"code","source":"crypto_df.tail()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:21:31.702677Z","iopub.execute_input":"2021-12-12T11:21:31.703029Z","iopub.status.idle":"2021-12-12T11:21:31.731448Z","shell.execute_reply.started":"2021-12-12T11:21:31.702996Z","shell.execute_reply":"2021-12-12T11:21:31.730375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df['Target'].max()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:21:31.734681Z","iopub.execute_input":"2021-12-12T11:21:31.734937Z","iopub.status.idle":"2021-12-12T11:21:31.777486Z","shell.execute_reply.started":"2021-12-12T11:21:31.734905Z","shell.execute_reply":"2021-12-12T11:21:31.776483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df['Target'].min()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:21:31.778609Z","iopub.execute_input":"2021-12-12T11:21:31.778847Z","iopub.status.idle":"2021-12-12T11:21:31.819341Z","shell.execute_reply.started":"2021-12-12T11:21:31.778819Z","shell.execute_reply":"2021-12-12T11:21:31.818432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# select training and test periods\n# 時系列データなので、testデータは未来になるように調整\ntrain_term_end = '2021-03-01'","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:21:31.820626Z","iopub.execute_input":"2021-12-12T11:21:31.820880Z","iopub.status.idle":"2021-12-12T11:21:31.825057Z","shell.execute_reply.started":"2021-12-12T11:21:31.820849Z","shell.execute_reply":"2021-12-12T11:21:31.823997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"asset_details","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:21:31.826630Z","iopub.execute_input":"2021-12-12T11:21:31.827679Z","iopub.status.idle":"2021-12-12T11:21:31.845206Z","shell.execute_reply.started":"2021-12-12T11:21:31.827626Z","shell.execute_reply":"2021-12-12T11:21:31.844359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:21:31.846891Z","iopub.execute_input":"2021-12-12T11:21:31.847853Z","iopub.status.idle":"2021-12-12T11:21:31.870511Z","shell.execute_reply.started":"2021-12-12T11:21:31.847810Z","shell.execute_reply":"2021-12-12T11:21:31.869406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"datamart= pd.merge(crypto_df,asset_details[['Asset_ID','Weight']],on='Asset_ID',how='left')","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:21:31.871982Z","iopub.execute_input":"2021-12-12T11:21:31.872746Z","iopub.status.idle":"2021-12-12T11:21:34.868370Z","shell.execute_reply.started":"2021-12-12T11:21:31.872699Z","shell.execute_reply":"2021-12-12T11:21:34.867362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# datamart.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:21:34.869596Z","iopub.execute_input":"2021-12-12T11:21:34.869835Z","iopub.status.idle":"2021-12-12T11:21:34.874152Z","shell.execute_reply.started":"2021-12-12T11:21:34.869807Z","shell.execute_reply":"2021-12-12T11:21:34.873059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# divide data into train and test, compute X and y\n# 時系列データなので、testデータは未来になるように調整\nlgbm_train = datamart[datamart['date']< train_term_end].fillna(0)\nlgbm_eval = datamart[datamart['date']>= train_term_end].fillna(0)","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:21:34.875998Z","iopub.execute_input":"2021-12-12T11:21:34.876644Z","iopub.status.idle":"2021-12-12T11:24:28.592638Z","shell.execute_reply.started":"2021-12-12T11:21:34.876597Z","shell.execute_reply":"2021-12-12T11:24:28.591789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lgbm_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:24:28.594178Z","iopub.execute_input":"2021-12-12T11:24:28.594550Z","iopub.status.idle":"2021-12-12T11:24:28.598011Z","shell.execute_reply.started":"2021-12-12T11:24:28.594520Z","shell.execute_reply":"2021-12-12T11:24:28.597436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"drop_list = ['Target','date']","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:24:28.599328Z","iopub.execute_input":"2021-12-12T11:24:28.599542Z","iopub.status.idle":"2021-12-12T11:24:28.613192Z","shell.execute_reply.started":"2021-12-12T11:24:28.599515Z","shell.execute_reply":"2021-12-12T11:24:28.612283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train = lgbm_train['Target']\nX_train = lgbm_train.drop(drop_list , axis=1)\ny_test = lgbm_eval['Target']\nX_test = lgbm_eval.drop(drop_list , axis=1)\n\n\ndataset_train=lgbm.Dataset(X_train, y_train)\ndataset_eval=lgbm.Dataset(X_test, y_test, reference=dataset_train)","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:24:28.614625Z","iopub.execute_input":"2021-12-12T11:24:28.615031Z","iopub.status.idle":"2021-12-12T11:24:29.293079Z","shell.execute_reply.started":"2021-12-12T11:24:28.614999Z","shell.execute_reply":"2021-12-12T11:24:29.292395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:24:29.294234Z","iopub.execute_input":"2021-12-12T11:24:29.294576Z","iopub.status.idle":"2021-12-12T11:24:29.297617Z","shell.execute_reply.started":"2021-12-12T11:24:29.294544Z","shell.execute_reply":"2021-12-12T11:24:29.296973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"params = {'task': 'train',\n          'boosting_type': 'gbdt',      # GBDTを指定\n          'objective': 'regression', # 目的 : 回帰  \n          'metric': {'rmse'}, # 評価指標 : rsme(平均二乗誤差の平方根) \n          'learning_rate': 0.1,         # 学習率\n          'num_leaves': 23,             # ノードの数\n          'min_data_in_leaf': 3,        # 決定木ノードの最小データ数\n          'num_iteration': 1000         # 予測器(決定木)の数:イタレーション\n         }\n ","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:24:29.298800Z","iopub.execute_input":"2021-12-12T11:24:29.299037Z","iopub.status.idle":"2021-12-12T11:24:29.311324Z","shell.execute_reply.started":"2021-12-12T11:24:29.299009Z","shell.execute_reply":"2021-12-12T11:24:29.310352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# モデルの学習\nlgbm_model = lgbm.train(params,\n                        train_set=dataset_train, # トレーニングデータの指定\n                        valid_sets=dataset_eval, # 検証データの指定\n                        early_stopping_rounds=1000\n                  )\n","metadata":{"id":"Ud_lz31L7S1x","execution":{"iopub.status.busy":"2021-12-12T11:24:29.312722Z","iopub.execute_input":"2021-12-12T11:24:29.312984Z","iopub.status.idle":"2021-12-12T11:30:03.307652Z","shell.execute_reply.started":"2021-12-12T11:24:29.312946Z","shell.execute_reply":"2021-12-12T11:30:03.306789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = lgbm_model.predict(X_test)","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:34:50.072186Z","iopub.execute_input":"2021-12-12T11:34:50.072907Z","iopub.status.idle":"2021-12-12T11:34:50.796172Z","shell.execute_reply.started":"2021-12-12T11:34:50.072864Z","shell.execute_reply":"2021-12-12T11:34:50.795284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgbm.plot_importance(lgbm_model, figsize=(12, 8))\n","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:32:07.081028Z","iopub.execute_input":"2021-12-12T11:32:07.081426Z","iopub.status.idle":"2021-12-12T11:32:07.362368Z","shell.execute_reply.started":"2021-12-12T11:32:07.081383Z","shell.execute_reply":"2021-12-12T11:32:07.361436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluate baselines / ベースラインの評価\n\nThe competition performance metric is weighted correlation. However, for now we will use simple correlation to evaluate the two baseline models built.\n\nコンペティションの評価指標は加重相関です。しかし、ここでは単純相関を用いて、構築した2つのベースラインモデルを評価します。","metadata":{"id":"4_LbXT3KEdv5"}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import mean_squared_error # モデル評価用(平均二乗誤差)\nfrom sklearn.metrics import r2_score # モデル評価用(決定係数)\n# モデル評価\n# rmse : 平均二乗誤差の平方根\nmse = mean_squared_error(y_test, y_pred) # MSE(平均二乗誤差)の算出\nrmse = np.sqrt(mse) # RSME = √MSEの算出\nprint('RMSE :',rmse)\n\n#r2 : 決定係数\nr2 = r2_score(y_test,y_pred)\nprint('R2 :',r2)","metadata":{"execution":{"iopub.status.busy":"2021-12-12T11:34:53.622211Z","iopub.execute_input":"2021-12-12T11:34:53.622690Z","iopub.status.idle":"2021-12-12T11:34:53.704615Z","shell.execute_reply.started":"2021-12-12T11:34:53.622656Z","shell.execute_reply":"2021-12-12T11:34:53.703580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission / 提出方法\n\nNote that this is a Code Competition, in which you must submit your notebook to be run against the hidden private data. Your notebook should use the provided python time-series API, which ensures that models do not peek forward in time. To use the API, follow the instructions and template in [Code Competition Detailed API instructions](https://www.kaggle.com/sohier/detailed-api-introduction) and [Basic Submission Template](https://www.kaggle.com/sohier/basic-submission-template).\n\nこれはコードコンペティションであり、隠されたプライベートデータに対して実行されるノートブックを提出する必要があります。ノートブックは、提供されているpython時系列APIを使用する必要があります。これにより、モデルが時間的に先に進むことがないようになっています。APIを使用するには、[Code Competition Detailed API instructions](https://www.kaggle.com/sohier/detailed-api-introduction)と[Basic Submission Template](https://www.kaggle.com/sohier/basic-submission-template)の指示とテンプレートに従ってください。","metadata":{"id":"alEWivyFSY68"}}]}