{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G-Research Time Series-Specific Exploratory Data Analysis","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom datetime import datetime\nimport time","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:16:54.799573Z","iopub.execute_input":"2022-01-13T19:16:54.800005Z","iopub.status.idle":"2022-01-13T19:16:54.827949Z","shell.execute_reply.started":"2022-01-13T19:16:54.799878Z","shell.execute_reply":"2022-01-13T19:16:54.827281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Wrangling","metadata":{}},{"cell_type":"markdown","source":"### Get the Data\n\nAvailable train data inclues test data, this leading to astounding scores on the public leaderboard. In time series, such a phenomenon when info about the future is leaked to our analysis or training process is called a lookahead. A lookahead is a way, through data, to find out something about the future earlier thant you ought to know it. Information about what will happen in the future propagates back in time in our modelling and affects how our model behaves earlier in time (generally tends to faking model performance improvement). Here, data posterior to `2021-06-13` is leaky, i.e. it is data bout the future that should not be used for anything else than testing models or assumptions. The `read_csv_strict()` function designed by [dataista0](https://www.kaggle.com/julian3833) helps in avoiding the caveat of using future data for training models.","metadata":{}},{"cell_type":"code","source":"path = '../input/g-research-crypto-forecasting/'\n\ndef read_csv_strict(file_name='train.csv'): \n    df = pd.read_csv(path+file_name)\n    for asset in df['Asset_ID'].unique():\n        df.loc[df['Asset_ID']==asset, 'datetime'] = pd.to_datetime(df.loc[df['Asset_ID']==asset,'timestamp'], unit='s')\n    df = df[df['datetime']<'2021-06-13 00:00:00']\n    df.drop('timestamp', axis=1, inplace=True)\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:16:54.829253Z","iopub.execute_input":"2022-01-13T19:16:54.829800Z","iopub.status.idle":"2022-01-13T19:16:54.835564Z","shell.execute_reply.started":"2022-01-13T19:16:54.829769Z","shell.execute_reply":"2022-01-13T19:16:54.835010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#  Load train data and asset details\ntrain_data = read_csv_strict()\nasset_details = pd.read_csv(path+'asset_details.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:16:54.836497Z","iopub.execute_input":"2022-01-13T19:16:54.837033Z","iopub.status.idle":"2022-01-13T19:18:06.012173Z","shell.execute_reply.started":"2022-01-13T19:16:54.836993Z","shell.execute_reply":"2022-01-13T19:18:06.010820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:18:06.014361Z","iopub.execute_input":"2022-01-13T19:18:06.014716Z","iopub.status.idle":"2022-01-13T19:18:06.044697Z","shell.execute_reply.started":"2022-01-13T19:18:06.014668Z","shell.execute_reply":"2022-01-13T19:18:06.043989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"asset_details.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:18:06.046967Z","iopub.execute_input":"2022-01-13T19:18:06.047265Z","iopub.status.idle":"2022-01-13T19:18:06.058192Z","shell.execute_reply.started":"2022-01-13T19:18:06.047231Z","shell.execute_reply":"2022-01-13T19:18:06.057525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Join train and asset details to get asset names and weight alongside their time series\ntrain_data = pd.merge(train_data, asset_details, on='Asset_ID')\ntrain_data.set_index('datetime', inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:18:06.059429Z","iopub.execute_input":"2022-01-13T19:18:06.060171Z","iopub.status.idle":"2022-01-13T19:18:10.328145Z","shell.execute_reply.started":"2022-01-13T19:18:06.060127Z","shell.execute_reply":"2022-01-13T19:18:10.326920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory Data Analysis","metadata":{}},{"cell_type":"markdown","source":"### Time Series-Specific Exploratory Methods\n\nTime series can be approached in a more specific manner by looking at values at different times in a given series. More precisely, relationships between values at different times in the same series constitute the basis of what we're going to study in the following part of the notebook.","metadata":{}},{"cell_type":"markdown","source":"#### Plots, plots, plots","metadata":{}},{"cell_type":"code","source":"# Global plot\n# logarithm is for visual considerations\n\nf = plt.figure(figsize=(20, 8))\nfor asset in train_data['Asset_Name'].unique():\n    plt.plot(np.log(train_data[train_data['Asset_Name']==asset]['Close']), label=asset);\nplt.title('Assets close value evolution over time');\nplt.xlabel('Time');\nplt.ylabel('Close value');\nplt.legend();","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:18:10.362120Z","iopub.execute_input":"2022-01-13T19:18:10.362625Z","iopub.status.idle":"2022-01-13T19:20:26.398068Z","shell.execute_reply.started":"2022-01-13T19:18:10.362588Z","shell.execute_reply":"2022-01-13T19:20:26.397047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Seasonal plots\n# Daily average Close\nfor asset in train_data['Asset_Name'].unique():\n    f, ax = plt.subplots(figsize=(10,7))\n    t = train_data[train_data['Asset_Name']==asset]\n    for year in t.index.year.unique():\n        y = t[t.index.year==year]\n        y.groupby([y.index.month, y.index.day])['Close'].mean().plot(label=str(year))\n        \n        plt.title(f'{asset} daily average close value yearly plot')\n        plt.xlabel('Time');\n        plt.ylabel(f'{asset} daily average close value');\n        plt.legend();","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:51:19.581573Z","iopub.execute_input":"2022-01-13T19:51:19.581876Z","iopub.status.idle":"2022-01-13T19:52:28.163692Z","shell.execute_reply.started":"2022-01-13T19:51:19.581844Z","shell.execute_reply":"2022-01-13T19:52:28.162996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Seasonal plots\n# Monthly average Close\nfor asset in train_data['Asset_Name'].unique():\n    f, ax = plt.subplots(figsize=(10,7))\n    t = train_data[train_data['Asset_Name']==asset]\n    for year in t.index.year.unique():\n        y = t[t.index.year==year]\n        y.groupby([y.index.month])['Close'].mean().plot(label=str(year))\n        \n        plt.title(f'{asset} monthly average close value yearly plot')\n        plt.xlabel('Time');\n        plt.ylabel(f'{asset} monthly average close value');\n        plt.legend();","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:52:28.164915Z","iopub.execute_input":"2022-01-13T19:52:28.165149Z","iopub.status.idle":"2022-01-13T19:53:33.527136Z","shell.execute_reply.started":"2022-01-13T19:52:28.165121Z","shell.execute_reply":"2022-01-13T19:53:33.526035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Log returns","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:35:42.993652Z","iopub.execute_input":"2022-01-13T19:35:42.995039Z","iopub.status.idle":"2022-01-13T19:35:42.999488Z","shell.execute_reply.started":"2022-01-13T19:35:42.994977Z","shell.execute_reply":"2022-01-13T19:35:42.998713Z"}}},{"cell_type":"code","source":"# Compute log of return = percentage change\ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:48:27.079898Z","iopub.execute_input":"2022-01-13T19:48:27.080838Z","iopub.status.idle":"2022-01-13T19:48:33.815613Z","shell.execute_reply.started":"2022-01-13T19:48:27.080781Z","shell.execute_reply":"2022-01-13T19:48:33.814527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for asset in train_data['Asset_Name'].unique():\n    f, ax = plt.subplots(figsize=(10,7))\n    log_return(train_data[train_data['Asset_Name']==asset]['Close'], periods=1)[1:].plot() # First row has an empty return as previous value is unknown\n    plt.title(f'{asset} monthly average close value daily log return')\n    plt.xlabel('Time');\n    plt.ylabel('log(pct_change)');\n    plt.legend();","metadata":{"execution":{"iopub.status.busy":"2022-01-13T19:53:33.529340Z","iopub.execute_input":"2022-01-13T19:53:33.530117Z","iopub.status.idle":"2022-01-13T19:56:59.050618Z","shell.execute_reply.started":"2022-01-13T19:53:33.530061Z","shell.execute_reply":"2022-01-13T19:56:59.049698Z"},"trusted":true},"execution_count":null,"outputs":[]}]}