{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G-Research Crypto Forecasting\n\n## To Do List\n* What is time series API.\n* Find better way to deal with missing values. Currently We are just removing the rows with missing values.\n* Make a quick solution and submit it to understand the submission process.\n* How to handle missing values in timeseries data.\n\n## Problem Statement\n\n*  Number of Cryptocurrencies: 14\n* Data from 2018.\n* Submissions are evaluated on a weighted version of the Pearson correlation coefficient.\n* forecasting competition\n\n## Dataset Description\n\n## Solution","metadata":{}},{"cell_type":"markdown","source":"## Import Required Packages","metadata":{}},{"cell_type":"code","source":"# Import required modules.\n\nimport tensorflow as tf\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nimport os\n\nprint('TensorFlow: {}'.format(tf.__version__))","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:26:50.560353Z","iopub.execute_input":"2021-12-30T06:26:50.560633Z","iopub.status.idle":"2021-12-30T06:26:55.536267Z","shell.execute_reply.started":"2021-12-30T06:26:50.560605Z","shell.execute_reply":"2021-12-30T06:26:55.535319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore the Dataset","metadata":{}},{"cell_type":"code","source":"df_asset_details = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\ndf = pd.read_csv('../input/g-research-crypto-forecasting/train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:26:55.537915Z","iopub.execute_input":"2021-12-30T06:26:55.53813Z","iopub.status.idle":"2021-12-30T06:27:58.13038Z","shell.execute_reply.started":"2021-12-30T06:26:55.538104Z","shell.execute_reply":"2021-12-30T06:27:58.129555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Assest Details Dataframe shape: ', df_asset_details.shape)\nprint('Train Datafrme shape: ', df.shape)","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:27:58.131615Z","iopub.execute_input":"2021-12-30T06:27:58.131936Z","iopub.status.idle":"2021-12-30T06:27:58.137607Z","shell.execute_reply.started":"2021-12-30T06:27:58.131883Z","shell.execute_reply":"2021-12-30T06:27:58.136534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_asset_details","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:27:58.140348Z","iopub.execute_input":"2021-12-30T06:27:58.140835Z","iopub.status.idle":"2021-12-30T06:27:58.167332Z","shell.execute_reply.started":"2021-12-30T06:27:58.14079Z","shell.execute_reply":"2021-12-30T06:27:58.166427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(20)","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:27:58.169121Z","iopub.execute_input":"2021-12-30T06:27:58.169457Z","iopub.status.idle":"2021-12-30T06:27:58.199712Z","shell.execute_reply.started":"2021-12-30T06:27:58.169412Z","shell.execute_reply":"2021-12-30T06:27:58.198671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Split the data into training, validation and test datasets.\n\nAs you can see that data in the dataframe is organized as ascending order of timestamp. And for each timestemp, the assest details for each assest are randomly organize. So, we can separate the data now without any issue. We'll split the data into 70, 20 and 10 percent for training, validation and test, respectively.","metadata":{}},{"cell_type":"code","source":"ds_size = len(df)\n\ndf_train = df[:int(ds_size*0.7)]\ndf_val = df[int(ds_size*0.7):int(ds_size*0.9)]\ndf_test = df[int(ds_size*0.9):]\n\nprint('Shape of training dataset: ', df_train.shape)\nprint('Shape of validation dataset: ', df_val.shape)\nprint('Shape of test dataset: ', df_test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:27:58.200804Z","iopub.execute_input":"2021-12-30T06:27:58.201029Z","iopub.status.idle":"2021-12-30T06:27:58.209093Z","shell.execute_reply.started":"2021-12-30T06:27:58.201004Z","shell.execute_reply":"2021-12-30T06:27:58.208257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Separate the data for each assest","metadata":{}},{"cell_type":"code","source":"# For asset_ID = 0\ndf_train_0 = df_train[df_train['Asset_ID'] == 0][0:50000].copy()\ndf_val_0 = df_val[df_val['Asset_ID'] == 0][0:20000].copy()\ndf_test_0 = df_test[df_test['Asset_ID'] == 0][0:10000].copy()","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:27:58.210265Z","iopub.execute_input":"2021-12-30T06:27:58.211088Z","iopub.status.idle":"2021-12-30T06:27:58.569599Z","shell.execute_reply.started":"2021-12-30T06:27:58.211051Z","shell.execute_reply":"2021-12-30T06:27:58.568654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Similarly, you can select data from any assest and from any dataset whether. Here we will experiment with small part of the data for a single assest.","metadata":{}},{"cell_type":"markdown","source":"## Clean the data\n* Look for missing values.\n* Look for outliers.\n* Look for correlation.\n* Plot the data to find any kind of periodicity.\n* Normalize the data appropriately.","metadata":{}},{"cell_type":"markdown","source":"Let's first plot the data to see if there is some periodicty to the data","metadata":{}},{"cell_type":"code","source":"columns = ['Count', 'Open', 'High', 'Low', 'Close',\n       'Volume', 'VWAP', 'Target']\n\nplt.figure(figsize=(12, 12))\n\nfor i in range(8):\n    i += 1\n    plt.subplot(3, 3, i)\n    plt.plot(df_train_0[columns[(i-1)]])\n    plt.title(columns[(i-1)])\n    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:27:58.570894Z","iopub.execute_input":"2021-12-30T06:27:58.571146Z","iopub.status.idle":"2021-12-30T06:27:59.696927Z","shell.execute_reply.started":"2021-12-30T06:27:58.571115Z","shell.execute_reply":"2021-12-30T06:27:59.695843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note:\n* No seasonality in data.\n* Plots of High, Low, Close, Open, and VWAP are exactly same.\n* Replot everything which full data.\n\nLet's handle the missing values.","metadata":{}},{"cell_type":"code","source":"df_train_0.info()        # You can see that target have missing values.","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:27:59.698103Z","iopub.execute_input":"2021-12-30T06:27:59.698321Z","iopub.status.idle":"2021-12-30T06:27:59.718872Z","shell.execute_reply.started":"2021-12-30T06:27:59.698295Z","shell.execute_reply":"2021-12-30T06:27:59.718086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:33:17.535704Z","iopub.execute_input":"2021-12-30T06:33:17.536018Z","iopub.status.idle":"2021-12-30T06:33:18.02503Z","shell.execute_reply.started":"2021-12-30T06:33:17.535984Z","shell.execute_reply":"2021-12-30T06:33:18.024193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missed_target = 750338\ntotal_target = len(df)\nprint('{:.2f}% values of target are missing in the data.'.format((missed_target*100)/(total_target)))","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:38:50.924027Z","iopub.execute_input":"2021-12-30T06:38:50.924504Z","iopub.status.idle":"2021-12-30T06:38:50.929544Z","shell.execute_reply.started":"2021-12-30T06:38:50.92447Z","shell.execute_reply":"2021-12-30T06:38:50.92861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Look for outliers.\n\ndf_train_0.describe().transpose()","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:27:59.732438Z","iopub.execute_input":"2021-12-30T06:27:59.732678Z","iopub.status.idle":"2021-12-30T06:27:59.793562Z","shell.execute_reply.started":"2021-12-30T06:27:59.732649Z","shell.execute_reply":"2021-12-30T06:27:59.792702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Correlation matrix\n\ndf_train_0.corr()","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:27:59.794778Z","iopub.execute_input":"2021-12-30T06:27:59.795078Z","iopub.status.idle":"2021-12-30T06:27:59.831747Z","shell.execute_reply.started":"2021-12-30T06:27:59.795051Z","shell.execute_reply":"2021-12-30T06:27:59.830863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## References:\n1. Handling missing values. [https://www.kaggle.com/juejuewang/handle-missing-values-in-time-series-for-beginners]\n2. https://otexts.com/fpp2/missing-outliers.html\n","metadata":{}}]}