{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"edaedd5b-f90e-466d-80fa-12cdcc6ad02c","_cell_guid":"ee1cf711-3578-44dc-b410-cdfaed2abc8d","jupyter":{"outputs_hidden":false},"collapsed":false,"execution":{"iopub.status.busy":"2021-12-01T07:07:17.503083Z","iopub.execute_input":"2021-12-01T07:07:17.50336Z","iopub.status.idle":"2021-12-01T07:07:17.52174Z","shell.execute_reply.started":"2021-12-01T07:07:17.503331Z","shell.execute_reply":"2021-12-01T07:07:17.520965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us start by taking a look, how the target values evolve over time per currency.","metadata":{"_uuid":"2f8cc1e8-315a-4586-b444-0e35bda48e9e","_cell_guid":"ce02b459-717a-445c-9de1-febbc028d76f","trusted":true}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport gc\n\ndtypes = {\n    'timestamp': np.int64,\n    'Asset_ID': np.int8,\n     'Count': np.int32,\n     'Open': np.float64,\n     'High': np.float64,\n     'Low': np.float64,\n    'Close': np.float64,\n     'Volume': np.float64,\n     'VWAP': np.float64,\n    'Target': np.float64,\n}\ndata = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv', dtype=dtypes, usecols=list(dtypes.keys()))\ndata['Time'] = pd.to_datetime(data['timestamp'], unit='s')\n\ndetails = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/asset_details.csv')\n\ndata = pd.merge(data, \n                details, \n                on ='Asset_ID', \n                how ='left')\n\nprint(data.head())\nprint(details)","metadata":{"_uuid":"b71cee9a-58e3-4753-a5ee-94fbeba04e64","_cell_guid":"8b961e6e-c9aa-49cb-b799-1363aa14bc34","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-01T07:07:17.561775Z","iopub.execute_input":"2021-12-01T07:07:17.562578Z","iopub.status.idle":"2021-12-01T07:08:17.201662Z","shell.execute_reply.started":"2021-12-01T07:07:17.562507Z","shell.execute_reply":"2021-12-01T07:08:17.200506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Subplots are organized in a Rows x Cols Grid\n# Tot and Cols are known\nTot = len(details.Asset_ID)\nCols = 4\n\n\n# Compute Rows required\nRows = Tot // Cols \nRows += Tot % Cols\n\n# Create a Position index\nPosition = range(1,Tot + 1)\n\n# Create main figure\nfig = plt.figure(1)\nfig.set_figheight(20)\nfig.set_figwidth(20)\n\nfor k in range(Tot):\n  # add every single subplot to the figure with a for loop\n\n    tmp_df = data[data.Asset_ID == details.Asset_ID[k]]\n    ax = fig.add_subplot(Rows,Cols,Position[k])\n    ax.plot(tmp_df.Time, tmp_df.Target)\n    ax.set_title(details.Asset_Name[k])\n\nplt.show()\n\ndel tmp_df","metadata":{"_uuid":"e06c0261-7efe-4674-91e1-e11367285e21","_cell_guid":"3f2571c9-e772-467a-b08d-705e0095f8a8","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-01T07:08:17.204635Z","iopub.execute_input":"2021-12-01T07:08:17.204979Z","iopub.status.idle":"2021-12-01T07:08:29.013335Z","shell.execute_reply.started":"2021-12-01T07:08:17.204923Z","shell.execute_reply":"2021-12-01T07:08:29.01271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In my opinion, we see very typical things here with respect to financial data. Time series plots of log-returns for assets typically look similar, i.e., the arithmetic mean is close to zero, while we observe volatility clustering which means in certain time periods volatility is higher than in others. Let us take a look at the distributions for the target variable.","metadata":{"_uuid":"dd7fa327-5755-4bf1-bba8-b6bf6b9730b3","_cell_guid":"1f32e3d6-e66a-453f-906d-a25e7d239984","trusted":true}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Subplots are organized in a Rows x Cols Grid\n# Tot and Cols are known\nTot = len(details.Asset_ID)\nCols = 4\n\n\n# Compute Rows required\nRows = Tot // Cols \nRows += Tot % Cols\n\n# Create a Position index\nPosition = range(1,Tot + 1)\n\n# Create main figure\nfig = plt.figure(1)\nfig.set_figheight(20)\nfig.set_figwidth(20)\n\nfor k in range(Tot):\n  # add every single subplot to the figure with a for loop\n\n    tmp_df = data[data.Asset_ID == details.Asset_ID[k]]\n    ax = fig.add_subplot(Rows,Cols,Position[k])\n    ax.hist(tmp_df.Target, bins = 50)\n    ax.set_xlim(-0.1, 0.1)\n    ax.set_title(details.Asset_Name[k])\n\nplt.show()\n\ndel tmp_df","metadata":{"_uuid":"733935f3-c311-4eab-8a82-8a28e8cbbabb","_cell_guid":"eff89103-4564-4aa9-909c-d0a6ee93a201","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-01T07:08:29.014544Z","iopub.execute_input":"2021-12-01T07:08:29.014779Z","iopub.status.idle":"2021-12-01T07:08:36.315643Z","shell.execute_reply.started":"2021-12-01T07:08:29.014748Z","shell.execute_reply":"2021-12-01T07:08:36.314743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the currencies exhibit rather symmetric distributions, yet, it looks like all currencies exhibit excess curtosis which means extreme events are more likely than under the assumption of a normal distribution. Now let us take a look if target values are correlated at each point in time.","metadata":{"_uuid":"d226ea7c-7e13-4137-a46d-1aa0bd5ab58e","_cell_guid":"e1c40663-57b8-4315-96f2-9aab062b3976","trusted":true}},{"cell_type":"code","source":"all_timestamps = np.sort(data['timestamp'].unique())\ntargets = pd.DataFrame(index=all_timestamps)\n\nfor i, id_ in enumerate(details.Asset_ID):\n    asset = data[data.Asset_ID == id_].set_index(keys='timestamp')\n    price = pd.Series(index=all_timestamps, data=asset['Close'])\n    targets[details.Asset_Name[i]] = (\n        price.shift(periods=-16) /\n        price.shift(periods=-1)\n    ) - 1\n    \nprint(targets.head())\n\nimport seaborn as sns\n\nsns.heatmap(targets.corr())\nplt.show()","metadata":{"_uuid":"3e5602a7-f9c1-4b09-81c4-f3dcc8d55221","_cell_guid":"020b7bb8-c4d7-490c-aae0-bff04547f992","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-01T07:08:36.317126Z","iopub.execute_input":"2021-12-01T07:08:36.317367Z","iopub.status.idle":"2021-12-01T07:08:47.346483Z","shell.execute_reply.started":"2021-12-01T07:08:36.317338Z","shell.execute_reply":"2021-12-01T07:08:47.345691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that target values for each currency exhibit high correlations in the cross-section. Unfortunately, this does not support us when making future predictions, as information in the cross-section is contemporaneous and not predictive. Now, let us try to find out, how systematic market movements of currencies behave. To do so, I simply normalize \"Close\"-values for each currency and calculate the naive (equally-weighted) mean for all currencies which are available over a point in time. Finally, I visualize this. Obviously, this is rather a rough quick and dirty method for checking systematic market behavior.","metadata":{"_uuid":"30210526-b72a-4c3c-b2a8-353833729586","_cell_guid":"851555a7-1d56-470b-9b90-16b0c9c8a0f7","trusted":true}},{"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\n\nclosing_prices = pd.DataFrame(index=all_timestamps)\n\nfor i, id_ in enumerate(details.Asset_ID):\n    asset = data[data.Asset_ID == id_].set_index(keys='timestamp')\n    price = pd.Series(index=all_timestamps, data=asset['Close'])\n    closing_prices[details.Asset_Name[i]] = price\n    \nmin_max_scaler = MinMaxScaler()\nclosing_prices_ = min_max_scaler.fit_transform(closing_prices)\nplt.plot(closing_prices.index, np.mean(closing_prices_, axis = 1))","metadata":{"_uuid":"1869f51d-6ea0-44e2-83cf-1dc564ce567b","_cell_guid":"9af1ca16-b21d-49ed-8df0-aca6cc490462","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-01T07:08:47.350097Z","iopub.execute_input":"2021-12-01T07:08:47.350326Z","iopub.status.idle":"2021-12-01T07:08:57.002685Z","shell.execute_reply.started":"2021-12-01T07:08:47.3503Z","shell.execute_reply":"2021-12-01T07:08:57.001611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I would say that we may observe a little systematic behavior which looks similar to economic cycles. This information may be further refined and used when splitting the data. Next, let us use Bitcoin as an example to check the raw data for possible predictive power.","metadata":{"_uuid":"b1ef2d28-a31a-43ec-9824-262fa3f2444c","_cell_guid":"c643660b-bc50-4fde-b9fb-5cb4090e8142","trusted":true}},{"cell_type":"code","source":"btc = data[data.Asset_ID == 1]\nbtc.set_index('timestamp', inplace = True)\nbtc.head()","metadata":{"_uuid":"e13df492-cafe-4128-8b73-715d8ae9c3dc","_cell_guid":"36edbb90-dd9d-42ea-9ed0-79de9c19a96f","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-01T07:08:57.004121Z","iopub.execute_input":"2021-12-01T07:08:57.004365Z","iopub.status.idle":"2021-12-01T07:08:57.307372Z","shell.execute_reply.started":"2021-12-01T07:08:57.004336Z","shell.execute_reply":"2021-12-01T07:08:57.306471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Taking a look at the subplots below indicate that alle variables except the target variable are non-stationary which is bad for making models that generalize over time.","metadata":{"_uuid":"f936d428-b463-4c6a-8880-a57ed0c6214a","_cell_guid":"42d90826-c571-4127-8f0d-343853a97db2","trusted":true}},{"cell_type":"code","source":"btc[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP', 'Target']].plot(subplots = True)","metadata":{"_uuid":"3b236357-880e-4161-8594-0cdf6e4740aa","_cell_guid":"6dfe16c2-d3d3-4ad2-910b-eaf1b70e0ef9","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-01T07:08:57.308571Z","iopub.execute_input":"2021-12-01T07:08:57.309408Z","iopub.status.idle":"2021-12-01T07:09:46.753327Z","shell.execute_reply.started":"2021-12-01T07:08:57.309369Z","shell.execute_reply":"2021-12-01T07:09:46.75233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, I tried first differences, but even those values do not seem to be stationary, so I directly tried the first log-differences below. Strict stationarity means the (unconditional) distribution stays the same over time for a given variable. Weak stationarity means at least the mean, variance and autocovariance are the same over time.","metadata":{"_uuid":"f0f41487-0c1b-4dcf-be9b-ef7d9ec42789","_cell_guid":"f25e0e8c-2369-466f-a1cd-52c8cd465744","trusted":true}},{"cell_type":"code","source":"first_log_differences = btc[['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']].apply(np.log).diff()\nfirst_log_differences.plot(subplots = True)","metadata":{"_uuid":"5bc0de40-0cae-44e7-98ca-05cc814a7eb2","_cell_guid":"3e43974b-bb39-42ea-9aa3-56d725c4fe17","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-01T07:09:46.75495Z","iopub.execute_input":"2021-12-01T07:09:46.755274Z","iopub.status.idle":"2021-12-01T07:10:39.681324Z","shell.execute_reply.started":"2021-12-01T07:09:46.755231Z","shell.execute_reply":"2021-12-01T07:10:39.680172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the plots below, I want to take a look if approximately the mean and the standard deviation are constant over time. For most of the time this seems to be the case, but there are some breaktroughs at some points in time which can really have a negative effect if we would use these variables for prediction.","metadata":{"_uuid":"19123fd0-3a37-4937-8e58-4c570b34efd2","_cell_guid":"5f342d24-e97a-4ea1-8a4a-9d827c52ab32","trusted":true}},{"cell_type":"code","source":"first_log_differences.rolling(7200).mean().plot(subplots = True)","metadata":{"_uuid":"06cfc75c-2109-42e5-87cb-91f5ae2a5dd0","_cell_guid":"2eeb24a1-3634-42c1-b302-e10fe0c349c6","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-01T07:10:39.682895Z","iopub.execute_input":"2021-12-01T07:10:39.683218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"first_log_differences.rolling(7200).std().plot(subplots = True)","metadata":{"_uuid":"30ca0d36-5555-44fd-82bc-b52ccb4c3044","_cell_guid":"2519fc81-e463-472a-b640-e51988ef1cc7","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us check if the feature variables (here the log-differences) exhibit any correlation to the target values.","metadata":{"_uuid":"b713bb19-9502-4bc3-9a09-5b04fcb5b8f9","_cell_guid":"5e87be38-2b69-4b7b-b4e7-a5124b0a7369","trusted":true}},{"cell_type":"code","source":"new_data = btc[['Target']].merge(first_log_differences, left_index = True, right_index = True)\nsns.heatmap(new_data.corr())\nplt.show()","metadata":{"_uuid":"0969cd4b-e319-46b6-bd5f-d2777257daa4","_cell_guid":"732e4692-1c67-4998-8d62-e7d6386659cc","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unfortunately, this does not seem to be the case. So, I further check if there is some correlation between the past of feature variables and the current target variable. To do so, I write a function which approximately calculates auto-cross-correlation. To be honest, I do not know if this is actually defined somewhere in statistics, however, what I want to find out if there is any (linear) relationship between past observations of feature variables and the current target obsevation.","metadata":{"_uuid":"ed255f65-5e40-42bc-bde5-e0fd0b474f69","_cell_guid":"23fa6236-559c-46ac-90ce-7d9255c68e01","trusted":true}},{"cell_type":"code","source":"def cross_autocorr(df, targetname, varname, lag = 1):\n    df.dropna(inplace = True)\n    x_m = df[targetname].mean()\n    y_m = df[varname].mean()\n\n    x_s = df[targetname].std()\n    y_s = df[varname].std()\n\n    return (x_s * y_s)**(-1) * np.mean((df[targetname].values[lag:] - x_m) * (df[varname].values[:-lag] - y_m))\n\ncross_autocorrs = {}\nfor var in ['Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP']:\n    autocorrs_tmp = []\n    for l in range(1, 30, 1):\n        autocorrs_tmp.append(cross_autocorr(new_data, 'Target', var, lag = l))\n        \n    cross_autocorrs[var] = autocorrs_tmp\n    \nsns.heatmap(pd.DataFrame(cross_autocorrs, index = range(1, 30, 1)))","metadata":{"_uuid":"40a623e0-5e12-4ddc-9169-4ba6a462fd00","_cell_guid":"53b3bb75-acff-42e8-996b-36c29574f97c","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we all may have guessed, this isn't the case! So, we are forced to create some features which may have at least a little correlation to the target variable. Otherwise, the best and most sophisticated model will fail to learn about making good predictions. So, good luck everyone in finding those features and if you find them let me know;)","metadata":{"_uuid":"336b4dd9-3c2a-4fae-a8f5-59d052e8d11e","_cell_guid":"18fbf315-0084-4e17-83ad-d815ce207135","trusted":true}},{"cell_type":"markdown","source":"","metadata":{"_uuid":"ce934c6a-ec96-444f-b477-4f6cf5804005","_cell_guid":"fc37aeb3-48ca-4cb5-8a3d-7829e7e83ad1","trusted":true}}]}