{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Introduction \n\nOver $40 billion worth of cryptocurrencies are traded every day. They are among the most popular assets for speculation and investment, yet have proven wildly volatile. Fast-fluctuating prices have made millionaires of a lucky few, and delivered crushing losses to others. Could some of these price movements have been predicted in advance?\n\nIn this competition, you'll use your machine learning expertise to forecast short term returns in 14 popular cryptocurrencies. We have amassed a dataset of millions of rows of high-frequency market data dating back to 2018 which you can use to build your model. Once the submission deadline has passed, your final score will be calculated over the following 3 months using live crypto data as it is collected.\n\nThe simultaneous activity of thousands of traders ensures that most signals will be transitory, persistent alpha will be exceptionally difficult to find, and the danger of overfitting will be considerable. In addition, since 2018, interest in the cryptomarket has exploded, so the volatility and correlation structure in our data are likely to be highly non-stationary. The successful contestant will pay careful attention to these considerations, and in the process gain valuable insight into the art and science of financial forecasting.\n\nG-Research is Europe’s leading quantitative finance research firm. We have long explored the extent of market prediction possibilities, making use of machine learning, big data, and some of the most advanced technology available. Specializing in data science and AI education for workforces, Cambridge Spark is partnering with G-Research for this competition.","metadata":{}},{"cell_type":"markdown","source":"### Index\n\n1. About the Data\n2. EDA\n3. Feature Engineering\n4. Training\n5. HyperP Tuning\n6. Submission","metadata":{}},{"cell_type":"code","source":"# DF\nimport pandas as pd\nimport numpy as np\nfrom datetime import datetime\n# Model\nfrom lightgbm import LGBMRegressor\n#Data\nimport gresearch_crypto\nimport traceback\nimport time\nfrom datetime import datetime\n# Plotters\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nfrom sklearn.model_selection import GridSearchCV\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-06T05:06:40.305816Z","iopub.execute_input":"2022-01-06T05:06:40.306736Z","iopub.status.idle":"2022-01-06T05:06:40.31434Z","shell.execute_reply.started":"2022-01-06T05:06:40.306606Z","shell.execute_reply":"2022-01-06T05:06:40.313222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 1. About the Data\n\n**train.csv**\n\n- timestamp: All timestamps are returned as second Unix timestamps (the number of seconds elapsed since 1970-01-01 00:00:00.000 UTC). Timestamps in this dataset are multiple of 60, indicating minute-by-minute data.\n- Asset_ID: The asset ID corresponding to one of the crytocurrencies (e.g. Asset_ID = 1 for Bitcoin). The mapping from Asset_ID to crypto asset is contained in asset_details.csv.\n- Count: Total number of trades in the time interval (last minute).\n- Open: Opening price of the time interval (in USD).\n- High: Highest price reached during time interval (in USD).\n- Low: Lowest price reached during time interval (in USD).\n- Close: Closing price of the time interval (in USD).\n- Volume: Quantity of asset bought or sold, displayed in base currency USD.\n- VWAP: The average price of the asset over the time interval, weighted by volume. VWAP is an aggregated form of trade data.\n- Target: Residual log-returns for the asset over a 15 minute horizon.\n\n**supplemental_train.csv** \n\nAfter the submission period is over this file's data will be replaced with cryptoasset prices from the submission period. In the Evaluation phase, the train, train supplement, and test set will be contiguous in time, apart from any missing data. The current copy, which is just filled approximately the right amount of data from train.csv is provided as a placeholder.\n\n**asset_details.csv** \n\nProvides the real name and of the cryptoasset for each Asset_ID and the weight each cryptoasset receives in the metric. Weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact the models.\n\n**example_sample_submission.csv**\n\nAn example of the data that will be delivered by the time series API. The data is just copied from train.csv.\n\n**example_test.csv** \n\nAn example of the data that will be delivered by the time series API.","metadata":{}},{"cell_type":"code","source":"# Load Datasets\n\npath = \"/kaggle/input/g-research-crypto-forecasting/\"\ndf_train = pd.read_csv(path + \"train.csv\")\ndf_test = pd.read_csv(path + \"example_test.csv\")\ndf_asset_details = pd.read_csv(path + \"asset_details.csv\")\ndf_supp_train = pd.read_csv(path + \"supplemental_train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:06:40.316283Z","iopub.execute_input":"2022-01-06T05:06:40.317327Z","iopub.status.idle":"2022-01-06T05:07:16.381389Z","shell.execute_reply.started":"2022-01-06T05:06:40.317283Z","shell.execute_reply":"2022-01-06T05:07:16.380381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:16.38276Z","iopub.execute_input":"2022-01-06T05:07:16.383046Z","iopub.status.idle":"2022-01-06T05:07:16.407657Z","shell.execute_reply.started":"2022-01-06T05:07:16.383013Z","shell.execute_reply":"2022-01-06T05:07:16.406556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:16.409609Z","iopub.execute_input":"2022-01-06T05:07:16.411127Z","iopub.status.idle":"2022-01-06T05:07:16.446695Z","shell.execute_reply.started":"2022-01-06T05:07:16.411078Z","shell.execute_reply":"2022-01-06T05:07:16.445325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_asset_details","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:16.449278Z","iopub.execute_input":"2022-01-06T05:07:16.450196Z","iopub.status.idle":"2022-01-06T05:07:16.462695Z","shell.execute_reply.started":"2022-01-06T05:07:16.45015Z","shell.execute_reply":"2022-01-06T05:07:16.462027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_supp_train","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:16.463951Z","iopub.execute_input":"2022-01-06T05:07:16.464599Z","iopub.status.idle":"2022-01-06T05:07:16.488779Z","shell.execute_reply.started":"2022-01-06T05:07:16.46456Z","shell.execute_reply":"2022-01-06T05:07:16.48808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2. EDA","metadata":{}},{"cell_type":"code","source":"df_train.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:16.490093Z","iopub.execute_input":"2022-01-06T05:07:16.490368Z","iopub.status.idle":"2022-01-06T05:07:25.261682Z","shell.execute_reply.started":"2022-01-06T05:07:16.490337Z","shell.execute_reply":"2022-01-06T05:07:25.260671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:25.262916Z","iopub.execute_input":"2022-01-06T05:07:25.263144Z","iopub.status.idle":"2022-01-06T05:07:25.754856Z","shell.execute_reply.started":"2022-01-06T05:07:25.263117Z","shell.execute_reply":"2022-01-06T05:07:25.753942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- VWAP has 9 Nans","metadata":{}},{"cell_type":"code","source":"df_train.timestamp\n# Time stamps are int, lets convert these to timestamps type for better indexing","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:25.756234Z","iopub.execute_input":"2022-01-06T05:07:25.756797Z","iopub.status.idle":"2022-01-06T05:07:25.766931Z","shell.execute_reply.started":"2022-01-06T05:07:25.75675Z","shell.execute_reply":"2022-01-06T05:07:25.765693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"timestamper = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:25.768481Z","iopub.execute_input":"2022-01-06T05:07:25.768766Z","iopub.status.idle":"2022-01-06T05:07:25.779162Z","shell.execute_reply.started":"2022-01-06T05:07:25.768727Z","shell.execute_reply":"2022-01-06T05:07:25.777883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc = df_train[df_train[\"Asset_ID\"]==1].set_index(\"timestamp\") # Asset_ID = 1 for Bitcoin\neth = df_train[df_train[\"Asset_ID\"]==6].set_index(\"timestamp\") # Asset_ID = 6 for Ethereum\nbnb = df_train[df_train[\"Asset_ID\"]==0].set_index(\"timestamp\") # Asset_ID = 0 for Binance Coin\nada = df_train[df_train[\"Asset_ID\"]==3].set_index(\"timestamp\") # Asset_ID = 3 for Cardano\n\nbeg_btc = datetime.fromtimestamp(btc.index[0]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nend_btc = datetime.fromtimestamp(btc.index[-1]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nbeg_eth = datetime.fromtimestamp(eth.index[0]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nend_eth = datetime.fromtimestamp(eth.index[-1]).strftime(\"%A, %B %d, %Y %I:%M:%S\")\nbeg_bnb = datetime.fromtimestamp(eth.index[0]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nend_bnb = datetime.fromtimestamp(eth.index[-1]).strftime(\"%A, %B %d, %Y %I:%M:%S\")\nbeg_ada = datetime.fromtimestamp(eth.index[0]).strftime(\"%A, %B %d, %Y %I:%M:%S\") \nend_ada = datetime.fromtimestamp(eth.index[-1]).strftime(\"%A, %B %d, %Y %I:%M:%S\")\n\nprint('Bitcoin from ', beg_btc, ' to ', end_btc) \nprint('Ethereum from ', beg_eth, ' to ', end_eth)\nprint('Binance from ', beg_bnb, ' to ', end_bnb) \nprint('Cardano from ', beg_ada, ' to ', end_ada)","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:25.780784Z","iopub.execute_input":"2022-01-06T05:07:25.781128Z","iopub.status.idle":"2022-01-06T05:07:27.248965Z","shell.execute_reply.started":"2022-01-06T05:07:25.781085Z","shell.execute_reply":"2022-01-06T05:07:27.247955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Data ranges from 2018 to 2021","metadata":{}},{"cell_type":"markdown","source":"Heat maps of 5 coins","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.heatmap(btc[['Count','Open','High','Low','Close','Volume','VWAP','Target']].corr(), \n            vmin=-1.0, vmax=1.0, annot=True, cmap='coolwarm', linewidths=0.1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:27.250575Z","iopub.execute_input":"2022-01-06T05:07:27.250826Z","iopub.status.idle":"2022-01-06T05:07:28.28555Z","shell.execute_reply.started":"2022-01-06T05:07:27.250796Z","shell.execute_reply":"2022-01-06T05:07:28.284516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.heatmap(eth[['Count','Open','High','Low','Close','Volume','VWAP','Target']].corr(), \n            vmin=-1.0, vmax=1.0, annot=True, cmap='coolwarm', linewidths=0.1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:28.287195Z","iopub.execute_input":"2022-01-06T05:07:28.287551Z","iopub.status.idle":"2022-01-06T05:07:29.290579Z","shell.execute_reply.started":"2022-01-06T05:07:28.287508Z","shell.execute_reply":"2022-01-06T05:07:29.289673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,6))\nsns.heatmap(bnb[['Count','Open','High','Low','Close','Volume','VWAP','Target']].corr(), \n            vmin=-1.0, vmax=1.0, annot=True, cmap='coolwarm', linewidths=0.1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:29.292974Z","iopub.execute_input":"2022-01-06T05:07:29.293229Z","iopub.status.idle":"2022-01-06T05:07:30.311454Z","shell.execute_reply.started":"2022-01-06T05:07:29.2932Z","shell.execute_reply":"2022-01-06T05:07:30.310154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 3. Feature Engineering","metadata":{}},{"cell_type":"code","source":"def hlco_ratio(df): \n    return (df['High'] - df['Low'])/(df['Close']-df['Open'])\ndef upper_shadow(df):\n    return df['High'] - np.maximum(df['Close'], df['Open'])\ndef lower_shadow(df):\n    return np.minimum(df['Close'], df['Open']) - df['Low']\n\ndef get_features(df):\n    df_feat = df[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].copy()\n    df_feat['Upper_Shadow'] = upper_shadow(df_feat)\n    df_feat['hlco_ratio'] = hlco_ratio(df_feat)\n    df_feat['Lower_Shadow'] = lower_shadow(df_feat)\n    return df_feat","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:30.31277Z","iopub.execute_input":"2022-01-06T05:07:30.313344Z","iopub.status.idle":"2022-01-06T05:07:30.321085Z","shell.execute_reply.started":"2022-01-06T05:07:30.313305Z","shell.execute_reply":"2022-01-06T05:07:30.319906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 4. Training","metadata":{}},{"cell_type":"code","source":"# train test split df_train into 70% train rows and 30% valid rows\ntrain_data = df_train\n\ndef get_Xy_and_model_for_asset(df_train, asset_id):\n    df = df_train[df_train[\"Asset_ID\"] == asset_id]\n    \n    df = df.sample(frac=0.3)\n    df_proc = get_features(df)\n    df_proc['y'] = df['Target']\n    df_proc.replace([np.inf, -np.inf], np.nan, inplace=True)\n    df_proc = df_proc.dropna(how=\"any\")\n    \n    \n    X = df_proc.drop(\"y\", axis=1)\n    y = df_proc[\"y\"]   \n    model = LGBMRegressor()\n    model.fit(X, y)\n    return X, y, model\n\nXs = {}\nys = {}\nmodels = {}\n\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n    X, y, model = get_Xy_and_model_for_asset(train_data, asset_id)       \n    try:\n        Xs[asset_id], ys[asset_id], models[asset_id] = X, y, model\n    except: \n        Xs[asset_id], ys[asset_id], models[asset_id] = None, None, None ","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:30.322388Z","iopub.execute_input":"2022-01-06T05:07:30.323228Z","iopub.status.idle":"2022-01-06T05:07:56.573285Z","shell.execute_reply.started":"2022-01-06T05:07:30.32318Z","shell.execute_reply":"2022-01-06T05:07:56.572514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:56.574837Z","iopub.execute_input":"2022-01-06T05:07:56.575511Z","iopub.status.idle":"2022-01-06T05:07:56.588055Z","shell.execute_reply.started":"2022-01-06T05:07:56.575461Z","shell.execute_reply":"2022-01-06T05:07:56.586535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 5. HyperP Tuning","metadata":{}},{"cell_type":"code","source":"parameters = {\n    'num_leaves': range(21, 161, 10),\n    'learning_rate': [0.1, 0.01, 0.05]\n}\n\nnew_models = {}\nfor asset_id, asset_name in zip(df_asset_details['Asset_ID'], df_asset_details['Asset_Name']):\n    print(\"GridSearchCV for: \" + asset_name)\n    grid_search = GridSearchCV(\n        estimator=get_Xy_and_model_for_asset(df_train, asset_id)[2],\n        param_grid=parameters,\n        n_jobs = -1,\n        cv = 5,\n        verbose=True\n    )\n    grid_search.fit(Xs[asset_id], ys[asset_id])\n    new_models[asset_id] = grid_search.best_estimator_\n    grid_search.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2022-01-06T05:07:56.589769Z","iopub.execute_input":"2022-01-06T05:07:56.590411Z","iopub.status.idle":"2022-01-06T06:08:38.058856Z","shell.execute_reply.started":"2022-01-06T05:07:56.590371Z","shell.execute_reply":"2022-01-06T06:08:38.057592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 6. Submission","metadata":{}},{"cell_type":"code","source":"env = gresearch_crypto.make_env()\niter_test = env.iter_test()\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    for j , row in df_test.iterrows():        \n        if new_models[row['Asset_ID']] is not None:\n            try:\n                model = new_models[row['Asset_ID']]\n                x_test = get_features(row)\n                y_pred = model.predict(pd.DataFrame([x_test]))[0]\n                df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n            except:\n                df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n                traceback.print_exc()\n        else: \n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0  \n    \n    env.predict(df_pred)","metadata":{"execution":{"iopub.status.busy":"2022-01-06T06:08:38.062955Z","iopub.execute_input":"2022-01-06T06:08:38.063259Z","iopub.status.idle":"2022-01-06T06:08:38.466723Z","shell.execute_reply.started":"2022-01-06T06:08:38.063226Z","shell.execute_reply":"2022-01-06T06:08:38.465718Z"},"trusted":true},"execution_count":null,"outputs":[]}]}