{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\ndata = pd.read_csv(\"../input/g-research-crypto-forecasting/train.csv\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-11-07T23:39:56.610548Z","iopub.execute_input":"2021-11-07T23:39:56.611116Z","iopub.status.idle":"2021-11-07T23:40:52.89596Z","shell.execute_reply.started":"2021-11-07T23:39:56.611029Z","shell.execute_reply":"2021-11-07T23:40:52.895095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Convert the timestamp to a more readable format.","metadata":{}},{"cell_type":"code","source":"data['Time'] = pd.to_datetime(data['timestamp'], unit='s')","metadata":{"execution":{"iopub.status.busy":"2021-11-07T23:41:01.570107Z","iopub.execute_input":"2021-11-07T23:41:01.570402Z","iopub.status.idle":"2021-11-07T23:41:02.927853Z","shell.execute_reply.started":"2021-11-07T23:41:01.570373Z","shell.execute_reply":"2021-11-07T23:41:02.927113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"raw","source":"Let's check the timestamps in the data from some random days in the middle of the month. For each of these periods we can see that we have data for the times 23:59, 00:00, and 00:01. ","metadata":{}},{"cell_type":"code","source":"print(data[(data.Time>='2018-01-01 23:59:00') & (data.Time<='2018-01-02 00:01:00')]['Time'].dt.time.unique())\nprint(data[(data.Time>='2019-03-15 23:59:00') & (data.Time<='2019-03-16 00:01:00')]['Time'].dt.time.unique())\nprint(data[(data.Time>='2020-09-23 23:59:00') & (data.Time<='2020-09-24 00:01:00')]['Time'].dt.time.unique())\nprint(data[(data.Time>='2021-07-08 23:59:00') & (data.Time<='2021-07-09 00:01:00')]['Time'].dt.time.unique())","metadata":{"execution":{"iopub.status.busy":"2021-11-07T23:43:26.732661Z","iopub.execute_input":"2021-11-07T23:43:26.732929Z","iopub.status.idle":"2021-11-07T23:43:27.358694Z","shell.execute_reply.started":"2021-11-07T23:43:26.732902Z","shell.execute_reply":"2021-11-07T23:43:27.357824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"However, as we transition from one month to the next, we no longer have the first minute of the month -- 00:00.","metadata":{}},{"cell_type":"code","source":"print(data[(data.Time>='2018-01-31 23:59:00') & (data.Time<='2018-02-01 00:01:00')]['Time'].dt.time.unique())\nprint(data[(data.Time>='2019-03-31 23:59:00') & (data.Time<='2019-04-01 00:01:00')]['Time'].dt.time.unique())\nprint(data[(data.Time>='2020-09-30 23:59:00') & (data.Time<='2020-10-01 00:01:00')]['Time'].dt.time.unique())\nprint(data[(data.Time>='2021-07-31 23:59:00') & (data.Time<='2021-08-01 00:01:00')]['Time'].dt.time.unique())","metadata":{"execution":{"iopub.status.busy":"2021-11-07T23:45:06.069972Z","iopub.execute_input":"2021-11-07T23:45:06.070272Z","iopub.status.idle":"2021-11-07T23:45:06.692443Z","shell.execute_reply.started":"2021-11-07T23:45:06.07024Z","shell.execute_reply":"2021-11-07T23:45:06.691394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Normally if the data for (t+1) or (t+16) is missing, then Target will be NaN. For example, for Asset_ID 11 there is no Target for 2018-01-01 00:01:00 because data is missing for 2018-01-01 00:17:00.\n\nAlso note that at 00:14:00 the Target is NaN because there is no data for the period after it, 00:15:00.","metadata":{}},{"cell_type":"code","source":"data[(data.Asset_ID==11)][:40]","metadata":{"execution":{"iopub.status.busy":"2021-11-08T00:13:48.340013Z","iopub.execute_input":"2021-11-08T00:13:48.340337Z","iopub.status.idle":"2021-11-08T00:13:48.658147Z","shell.execute_reply.started":"2021-11-08T00:13:48.340302Z","shell.execute_reply":"2021-11-08T00:13:48.657345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When the first minute of the month is missing we should expect there to be a missing Target 1 minute and 16 minutes prior, however there are no missing targets at all!","metadata":{}},{"cell_type":"code","source":"data[(data.Time>='2018-01-31 23:40:00') & (data.Time<='2018-02-01 00:01:00') & (data.Asset_ID==1)][-30:]","metadata":{"execution":{"iopub.status.busy":"2021-11-08T00:12:12.485034Z","iopub.execute_input":"2021-11-08T00:12:12.485341Z","iopub.status.idle":"2021-11-08T00:12:12.708315Z","shell.execute_reply.started":"2021-11-08T00:12:12.485309Z","shell.execute_reply":"2021-11-08T00:12:12.707704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So how is the target being calculated when the first minute of the month is missing? Is it using the Close price from 00:01:00? Or is is being calculated based on the Close at 00:00:00 -- data which we cannot see?","metadata":{}}]}