{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# On-line Feature Engineering\n\nThe goal of this notebook is to provide a framework for online feature engineering that seems to be needed for this competition.\n\nNote that:\n\n\n- The notebook mainly rely on my previous work in the JaneStreet competition. (See: https://www.kaggle.com/lucasmorin/running-algos-fe-for-fast-inference)\n- The notebook implement an online version of my previous exploratory notebook (See: https://www.kaggle.com/lucasmorin/crypto-forecasting-common-factors) and associated discussion (https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/288555).\n- The data is then used to calibrate a lgbm with custom feval and an importance framework (see: https://www.kaggle.com/lucasmorin/online-fe-lgbm-feval-importances). \n- The online FE framework is then used for submission (See: https://www.kaggle.com/lucasmorin/ensemble-submission/edit/run/81219507)","metadata":{}},{"cell_type":"markdown","source":"## Features engineering techniques :\n\n- [Start with the end](#Start) \n- [Get Data](#Get_Data)\n- [Reorder Data](#Reorder_Data)\n- [Missing Values](#Missing_Values)\n- [Base Feature Engineering](#Base_FE)\n- [Market Features](#Market_Features)\n- [Time Features](#Time_Features)\n- [Running Moving Average](#RMA) (<- Magic)\n- [Moving Average Features](#MA_FE)\n- [Betas](#Betas)\n- [Putting it all together](#All) (<- All the features)\n- [Building Folds](#Folds)\n- [Running Variance](#Variance)\n- [Complete Feature Exploration](#FE_exploration) (New)","metadata":{}},{"cell_type":"code","source":"import gresearch_crypto\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tqdm import tqdm\nfrom datetime import datetime\n\ndef timestamp_to_date(timestamp):\n    return(datetime.fromtimestamp(timestamp))\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\nenv = gresearch_crypto.make_env()\n\niter_test = env.iter_test()\n\n(test_df, sample_prediction_df) = next(iter_test)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-12-27T22:46:09.170041Z","iopub.execute_input":"2021-12-27T22:46:09.170736Z","iopub.status.idle":"2021-12-27T22:46:09.268921Z","shell.execute_reply.started":"2021-12-27T22:46:09.170594Z","shell.execute_reply":"2021-12-27T22:46:09.267933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Start'></a>\n# Start with the end\nLooking at iterator submission data.","metadata":{}},{"cell_type":"code","source":"test_df","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.270976Z","iopub.execute_input":"2021-12-27T22:46:09.271309Z","iopub.status.idle":"2021-12-27T22:46:09.297189Z","shell.execute_reply.started":"2021-12-27T22:46:09.271264Z","shell.execute_reply":"2021-12-27T22:46:09.296312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Get_Data'></a>\n# Get Data\nChange data from pandas to numpy.","metadata":{}},{"cell_type":"code","source":"timestamp,Asset_ID,Count,O,H,L,C,Volume,VWAP,row_id = (test_df[col].values for col in ['timestamp','Asset_ID','Count','Open','High','Low','Close','Volume','VWAP','row_id'])","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.298157Z","iopub.execute_input":"2021-12-27T22:46:09.298898Z","iopub.status.idle":"2021-12-27T22:46:09.305668Z","shell.execute_reply.started":"2021-12-27T22:46:09.298860Z","shell.execute_reply":"2021-12-27T22:46:09.304576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Reorder_Data'></a>\n# Reorder data\n\nNot sure this is entirely necessary depending on your model.\nI do that so that I can handle data of constant size.","metadata":{}},{"cell_type":"code","source":"order = np.argsort(Asset_ID)\norder","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.308229Z","iopub.execute_input":"2021-12-27T22:46:09.308849Z","iopub.status.idle":"2021-12-27T22:46:09.322330Z","shell.execute_reply.started":"2021-12-27T22:46:09.308798Z","shell.execute_reply":"2021-12-27T22:46:09.321691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Missing_Values'></a>\n# Missing value ?\n\nHandling missing assets: adding rows with nan. ","metadata":{}},{"cell_type":"code","source":"test_df_missing = test_df[test_df.Asset_ID.isin([1,2,3])]\n\nmissing_ID = [i for i in range(14) if i not in [1,2,3]]\n\nval = test_df_missing.values\n\nfor i in missing_ID:\n    val = np.append(val,np.expand_dims(np.array((timestamp[0],i,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan)),axis=0),axis=0)\n    \nval","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.323297Z","iopub.execute_input":"2021-12-27T22:46:09.323962Z","iopub.status.idle":"2021-12-27T22:46:09.348565Z","shell.execute_reply.started":"2021-12-27T22:46:09.323926Z","shell.execute_reply":"2021-12-27T22:46:09.347684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Base_FE'></a>\n# Base Feature Enginerring","metadata":{}},{"cell_type":"code","source":"asset_details = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\n\n#create dictionnary of weights\ndict_weights = {}\nfor i in range(asset_details.shape[0]):\n    dict_weights[asset_details.iloc[i,0]] = asset_details.iloc[i,1]\n    \nweigths = np.array([dict_weights[i] for i in range(14)])","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.350529Z","iopub.execute_input":"2021-12-27T22:46:09.352001Z","iopub.status.idle":"2021-12-27T22:46:09.369295Z","shell.execute_reply.started":"2021-12-27T22:46:09.351955Z","shell.execute_reply":"2021-12-27T22:46:09.368380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"VWAP = np.where(np.isinf(VWAP),(C+O)/2,VWAP)","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.370398Z","iopub.execute_input":"2021-12-27T22:46:09.370711Z","iopub.status.idle":"2021-12-27T22:46:09.375644Z","shell.execute_reply.started":"2021-12-27T22:46:09.370671Z","shell.execute_reply":"2021-12-27T22:46:09.374827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dtype={'Asset_ID': 'int8', 'Count': 'int32', 'row_id': 'int32', 'Count': 'int32',\n       'Open': 'float32', 'High': 'float32', 'Low': 'float32', 'Close': 'float32',\n       'Volume': 'float32', 'VWAP': 'float32'}\n\ntest_df = test_df.astype(dtype)","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.376946Z","iopub.execute_input":"2021-12-27T22:46:09.377683Z","iopub.status.idle":"2021-12-27T22:46:09.391741Z","shell.execute_reply.started":"2021-12-27T22:46:09.377606Z","shell.execute_reply":"2021-12-27T22:46:09.390890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Standardising Features\nbase = C\nO = O/base\nH = H/base\nL = L/base\nC = C/base\nVWAP = VWAP/base\nPrice = base\n\n# Using dollars \nDollars = Volume * Price\nVolume_per_trade = Volume/Count\nDollars_per_trade = Dollars/Count\n\n# log returns and volatility estimators\nlog_ret = np.log(C/O)\nGK_vol = (1 / 2 * np.log(H/L) ** 2 - (2 * np.log(2) - 1) * np.log(C / O) ** 2)\nRS_vol = np.log(H/C)*np.log(H/O) + np.log(L/C)*np.log(L/O)","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.392975Z","iopub.execute_input":"2021-12-27T22:46:09.393281Z","iopub.status.idle":"2021-12-27T22:46:09.405718Z","shell.execute_reply.started":"2021-12-27T22:46:09.393242Z","shell.execute_reply":"2021-12-27T22:46:09.404914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Market_Features'></a>\n# Market Features","metadata":{}},{"cell_type":"code","source":"#get back missing values in weights \nweigths = np.where(np.isnan(O),O,weigths)\nMarket_Features = np.nansum(np.array([Count,O,H,L,C,Price,Volume,VWAP,Dollars,Volume_per_trade,Dollars_per_trade,log_ret,GK_vol,RS_vol])*weigths/np.nansum(weigths),axis=1)\n","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.407926Z","iopub.execute_input":"2021-12-27T22:46:09.408455Z","iopub.status.idle":"2021-12-27T22:46:09.419753Z","shell.execute_reply.started":"2021-12-27T22:46:09.408421Z","shell.execute_reply":"2021-12-27T22:46:09.418926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Time_Features'></a>\n# Time Features","metadata":{}},{"cell_type":"code","source":"timestamp = timestamp_to_date(timestamp[0])\n\nsin_month = (np.sin(2 * np.pi * timestamp.month/12))\ncos_month = (np.cos(2 * np.pi * timestamp.month/12))\nsin_day = (np.sin(2 * np.pi * timestamp.day/31))\ncos_day = (np.cos(2 * np.pi * timestamp.day/31))\nsin_hour = (np.sin(2 * np.pi * timestamp.hour/24))\ncos_hour = (np.cos(2 * np.pi * timestamp.hour/24))\nsin_minute = (np.sin(2 * np.pi * timestamp.minute/60))\ncos_minute = (np.cos(2 * np.pi * timestamp.minute/60))\n\ntime_features = np.array((sin_month,cos_month,sin_day,cos_day,sin_hour,cos_hour,sin_minute,cos_minute))\ntime_features","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.420929Z","iopub.execute_input":"2021-12-27T22:46:09.421274Z","iopub.status.idle":"2021-12-27T22:46:09.437364Z","shell.execute_reply.started":"2021-12-27T22:46:09.421245Z","shell.execute_reply":"2021-12-27T22:46:09.436387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='RMA'></a>\n# Running Moving Average\n\nStandard pandas moving average implementation would look like this:","metadata":{}},{"cell_type":"code","source":"#rw = 10000\n#train_data_rolled = train_data.rolling(window=rw).mean()","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.438688Z","iopub.execute_input":"2021-12-27T22:46:09.439237Z","iopub.status.idle":"2021-12-27T22:46:09.447847Z","shell.execute_reply.started":"2021-12-27T22:46:09.439202Z","shell.execute_reply":"2021-12-27T22:46:09.446953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"But that wouldn't be practical. One idea is to get values in memory, then perform the mean. This would be rather inefficient too. \nA better approach is to keep track of the cumulated sum. Only adding the last instance / removing the further one in time at each time step.","metadata":{}},{"cell_type":"code","source":"import collections\nfrom collections import deque\n\nclass RunningMean:\n    def __init__(self, WIN_SIZE=20, n_size = 1):\n        self.n = 0\n        self.mean = np.zeros(n_size)\n        self.cum_sum = 0\n        self.past_value = 0\n        self.WIN_SIZE = WIN_SIZE\n        self.windows = collections.deque(maxlen=WIN_SIZE+1)\n        \n    def clear(self):\n        self.n = 0\n        self.windows.clear()\n\n    def push(self, x):\n        #currently fillna with past value, might want to change that\n        x = fillna_npwhere(x, self.past_value)\n        self.past_value = x\n        \n        self.windows.append(x)\n        self.cum_sum += x\n        \n        if self.n < self.WIN_SIZE:\n            self.n += 1\n            self.mean = self.cum_sum / float(self.n)\n            \n        else:\n            self.cum_sum -= self.windows.popleft()\n            self.mean = self.cum_sum / float(self.WIN_SIZE)\n\n    def get_mean(self):\n        return self.mean if self.n else np.zeros(n_size)\n\n    def __str__(self):\n        return \"Current window values: {}\".format(list(self.windows))\n\n# Temporary removing njit as it cause many bugs down the line\n# Problems mainly due to data types, I have to find where I need to constraint types so as not to make njit angry\n#@njit\ndef fillna_npwhere(array, values):\n    if np.isnan(array.sum()):\n        array = np.where(np.isnan(array), values, array)\n    return array","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.449910Z","iopub.execute_input":"2021-12-27T22:46:09.450852Z","iopub.status.idle":"2021-12-27T22:46:09.462970Z","shell.execute_reply.started":"2021-12-27T22:46:09.450815Z","shell.execute_reply":"2021-12-27T22:46:09.462102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='MA_FE'></a>\n# Moving Average Features","metadata":{}},{"cell_type":"code","source":"%%time \n\nMA_lags = [2,5,15,30,60,120,300,1800,3750,10*24*60,30*24*60]\n\n\nFeatures = np.transpose(np.array([Count,O,H,L,C,Price,Volume,VWAP,Dollars,Volume_per_trade,Dollars_per_trade,log_ret,GK_vol,RS_vol]))\n\n#instantiation Moving average features dict\ndict_RM = {}\ndict_RM_M = {}\nfor lag in MA_lags:\n    dict_RM[lag] = RunningMean(lag)\n    dict_RM_M[lag] = RunningMean(lag)\n\nfor i in tqdm(range(10000)):\n    \n    MA_Features = []\n    MA_Features_M  = [] \n    \n    for lag in MA_lags:\n        dict_RM[lag].push(Features)\n        dict_RM_M[lag].push(Market_Features)\n        \n        MA_Features.append(dict_RM[lag].get_mean())\n        MA_Features_M.append(dict_RM_M[lag].get_mean())\n        \n    MA_Features = np.concatenate(MA_Features,axis=1)\n    MA_Features_M = np.concatenate(MA_Features_M)","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:09.465222Z","iopub.execute_input":"2021-12-27T22:46:09.465650Z","iopub.status.idle":"2021-12-27T22:46:12.792664Z","shell.execute_reply.started":"2021-12-27T22:46:09.465585Z","shell.execute_reply":"2021-12-27T22:46:12.791768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Betas'></a>\n# Betas\n\nFor a lack of a better implementation I start with just two memories. ","metadata":{}},{"cell_type":"code","source":"%%time \n\nbeta_lags = [30,60,120,300,600,1800,3750,10*24*60,30*24*60]\n\n#instantiation dict betas\ndict_MM = {}\ndict_Mr = {}\nfor lag in beta_lags:\n    dict_MM[lag] = RunningMean(lag)\n    dict_Mr[lag] = RunningMean(lag)\n\nfor i in tqdm(range(10000)):\n    \n    betas = []\n    \n    for lag in beta_lags:\n        dict_MM[lag].push(Market_Features[11]**2)\n        dict_Mr[lag].push(Market_Features[11]*Features[11])\n        betas.append(np.expand_dims(dict_Mr[lag].get_mean()/dict_MM[lag].get_mean(),axis=1))\n        \n    betas = np.concatenate(betas,axis=1)\n","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:12.794973Z","iopub.execute_input":"2021-12-27T22:46:12.795270Z","iopub.status.idle":"2021-12-27T22:46:16.706953Z","shell.execute_reply.started":"2021-12-27T22:46:12.795216Z","shell.execute_reply":"2021-12-27T22:46:16.706077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='All'></a>\n# Putting it all together - cleaning and testing","metadata":{}},{"cell_type":"code","source":"%%time\n\n#not building the weights each loops\nasset_details = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\ndict_weights = {}\nfor i in range(asset_details.shape[0]):\n    dict_weights[asset_details.iloc[i,0]] = asset_details.iloc[i,1]\nweigths = np.array([dict_weights[i] for i in range(14)])\n\n# only needed when saving ?\ndtype={'Asset_ID': 'int8', 'Count': 'int32', 'row_id': 'int32', 'Count': 'int32',\n       'Open': 'float32', 'High': 'float32', 'Low': 'float32', 'Close': 'float32',\n       'Volume': 'float32', 'VWAP': 'float32'}\n#test_df = test_df.astype(dtype)\n\n#refactoring functions:\n\ndef Clean_df(x):\n    Asset_ID = x[:,1]\n    timestamp = x[0,0]\n    if len(Asset_ID)<14:\n        missing_ID = [i for i in range(14) if i not in Asset_ID]\n        for i in missing_ID:\n            row = np.array((timestamp,i,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan))\n            x = np.concatenate((x,np.expand_dims(row,axis=0)))\n    x = x[np.argsort(x[:,1])]\n    return (x[:,i] for i in range(x.shape[1]))\n\ndef Base_Feature_fn(timestamp,Asset_ID,Count,O,H,L,C,Volume,VWAP):\n    VWAP = np.where(np.isinf(VWAP),(C+O)/2,VWAP)\n    base = C\n    O = O/base\n    H = H/base\n    L = L/base\n    C = C/base\n    VWAP = VWAP/base\n    Price = base\n\n    Dollars = Volume * Price\n    Volume_per_trade = Volume/Count\n    Dollars_per_trade = Dollars/Count\n\n    log_ret = np.log(C/O)\n    log_ret_H = np.log(H/C)\n    log_ret_L = np.log(C/L)\n    log_ret_VWAP = np.log(C/VWAP)\n    \n    GK_vol = (1 / 2 * np.log(H/L) ** 2 - (2 * np.log(2) - 1) * np.log(C/O) ** 2)\n    RS_vol = np.log(H/C)*np.log(H/O) + np.log(L/C)*np.log(L/O)\n\n    #return(np.transpose(np.array([Count,O,H,L,C,Price,Volume,VWAP,Dollars,Volume_per_trade,Dollars_per_trade,log_ret,GK_vol,RS_vol])))\n    \n    log_Count,log_Volume,log_Dollars,log_Volume_per_trade,log_Dollars_per_trade = np.log([Count,Volume,Dollars,Volume_per_trade,Dollars_per_trade])\n\n    return(np.transpose(np.array([log_ret,log_ret_H,log_ret_L,log_ret_VWAP,GK_vol,RS_vol,log_Count,log_Volume,log_Dollars,log_Volume_per_trade,log_Dollars_per_trade])))\n\ndef Time_Feature_fn(timestamp):\n    \n    sin_month = (np.sin(2 * np.pi * timestamp.month/12))\n    cos_month = (np.cos(2 * np.pi * timestamp.month/12))\n    sin_day = (np.sin(2 * np.pi * timestamp.day/31))\n    cos_day = (np.cos(2 * np.pi * timestamp.day/31))\n    sin_hour = (np.sin(2 * np.pi * timestamp.hour/24))\n    cos_hour = (np.cos(2 * np.pi * timestamp.hour/24))\n    sin_minute = (np.sin(2 * np.pi * timestamp.minute/60))\n    cos_minute = (np.cos(2 * np.pi * timestamp.minute/60))\n\n    return(np.array((sin_month,cos_month,sin_day,cos_day,sin_hour,cos_hour,sin_minute,cos_minute)))\n\nMA_lags = [2,5,15,30,60,120,300,1800,3750,10*24*60,30*24*60]\n\n#instantiation Moving average features dict\ndict_RM = {}\ndict_RM_M = {}\n\nfor lag in MA_lags:\n    dict_RM[lag] = RunningMean(lag)\n    dict_RM_M[lag] = RunningMean(lag)\n    \nbeta_lags = [60,300,1800,3750,10*24*60,30*24*60]\n\n#instantiation dict betas\ndict_MM = {}\ndict_Mr = {}\nfor lag in beta_lags:\n    dict_MM[lag] = RunningMean(lag)\n    dict_Mr[lag] = RunningMean(lag)\n\nfor i in tqdm(range(10000)):\n    \n    timestamp,Asset_ID,Count,O,H,L,C,Volume,VWAP,row_id = Clean_df(test_df.values)\n    \n    # np.array([Count,O,H,L,C,Price,Volume,VWAP,Dollars,Volume_per_trade,Dollars_per_trade,log_ret,GK_vol,RS_vol])\n    Features = Base_Feature_fn(timestamp,Asset_ID,Count,O,H,L,C,Volume,VWAP)\n    \n    #removing wieghts when data is missing so that they don't appears in market\n    weigths = np.where(np.isnan(O),O,weigths)\n    Market_Features = np.nansum(Features*np.expand_dims(weigths,axis=1)/np.nansum(weigths),axis=0)\n    #Market_Features = np.tile(Market_Features,(14,1))\n    \n    #np.array((sin_month,cos_month,sin_day,cos_day,sin_hour,cos_hour,sin_minute,cos_minute))\n    timestamp = timestamp_to_date(timestamp[0])\n    Time_Features = Time_Feature_fn(timestamp)\n    #Time_Features = np.tile(Time_Features,(14,1))\n    \n    MA_Features = []\n    MA_Features_M  = [] \n    \n    for lag in MA_lags:\n        dict_RM[lag].push(Features)\n        dict_RM_M[lag].push(Market_Features)\n        \n        MA_Features.append(dict_RM[lag].get_mean())\n        MA_Features_M.append(dict_RM_M[lag].get_mean())\n        \n    MA_Features = np.concatenate(MA_Features,axis=1)\n    MA_Features_M = np.concatenate(MA_Features_M)\n    #MA_Features_M = np.tile(MA_Features_M,(14,1))\n    \n    betas = []\n    \n    for lag in beta_lags:\n        dict_MM[lag].push(Market_Features[0]**2)\n        dict_Mr[lag].push(Market_Features[0]*Features[:,0])\n        betas.append(np.expand_dims(dict_Mr[lag].get_mean()/dict_MM[lag].get_mean(),axis=1))\n        \n    betas = np.concatenate(betas,axis=1)\n    \n    #print(values)\n    #for data in [Features,np.tile(Market_Features,(14,1)),np.tile(Time_Features,(14,1)),MA_Features,np.tile(MA_Features_M,(14,1)),betas]:\n        #print(data.shape)\n    \n    values = np.concatenate((Features,np.tile(Market_Features,(14,1)),np.tile(Time_Features,(14,1)),MA_Features,np.tile(MA_Features_M,(14,1)),betas),axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:16.708520Z","iopub.execute_input":"2021-12-27T22:46:16.709034Z","iopub.status.idle":"2021-12-27T22:46:27.131324Z","shell.execute_reply.started":"2021-12-27T22:46:16.708987Z","shell.execute_reply":"2021-12-27T22:46:27.130487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Features_names = ['log_ret','log_ret_H','log_ret_L','log_ret_VWAP','GK_vol','RS_vol','log_Count','log_Volume','log_Dollars','log_Volume_per_trade','log_Dollars_per_trade']\nMarket_Features_names = [s+'_M' for s in Features_names]\nTime_Features_names = ['sin_month','cos_month','sin_day','cos_day','sin_hour','cos_hour','sin_minute','cos_minute']\nMA_Features_names = [s+'_'+str(lag) for lag in MA_lags for s in Features_names ]\nMA_Features_M_names = [s+'_'+str(lag) for lag in MA_lags for s in Market_Features_names]\nbetas_names = ['betas_'+str(lag) for lag in beta_lags]\n\nAll_names = Features_names + Market_Features_names + Time_Features_names + MA_Features_names + MA_Features_M_names + betas_names\ndf_values = pd.DataFrame(values, columns = All_names)","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:46:27.133196Z","iopub.execute_input":"2021-12-27T22:46:27.133442Z","iopub.status.idle":"2021-12-27T22:46:27.141973Z","shell.execute_reply.started":"2021-12-27T22:46:27.133410Z","shell.execute_reply":"2021-12-27T22:46:27.141071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Folds'></a>\n# Creating Training Folds\n\nFor the design of the folds, see discussion here: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/288555","metadata":{}},{"cell_type":"code","source":"DEBUG  = False\nnrows = 100000 if DEBUG else None\n\ndtype={'Asset_ID': 'int8', 'Count': 'int32', 'row_id': 'int32', 'Count': 'int32',\n       'Open': 'float32', 'High': 'float32', 'Low': 'float32', 'Close': 'float32',\n       'Volume': 'float32', 'VWAP': 'float32'}\n\ntrain_df = pd.read_csv('../input/g-research-crypto-forecasting/train.csv', low_memory=False, dtype=dtype, nrows=nrows)\nasset_details = pd.read_csv('../input/g-research-crypto-forecasting/asset_details.csv')\n\n#filter to avoid time leakage with the data \nfilter_leakage = pd.to_datetime(train_df['timestamp'], unit='s') < '2021-06-13 00:00:00'\ntrain_df = train_df[filter_leakage]","metadata":{"execution":{"iopub.status.busy":"2021-12-27T23:01:04.076333Z","iopub.execute_input":"2021-12-27T23:01:04.076694Z","iopub.status.idle":"2021-12-27T23:01:04.506866Z","shell.execute_reply.started":"2021-12-27T23:01:04.076653Z","shell.execute_reply":"2021-12-27T23:01:04.505751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate the class/group data\n\nimport os\ntime_ids = train_df.timestamp.unique()\n\nn_fold = 5\nsplits = 0.6\nntimes = len(time_ids)\n\nembargo_train_test = 100 if DEBUG else 60*24*30\nembargo_fold = 100 if DEBUG else 60*24*30\n\ntime_per_fold = (ntimes - 5*embargo_train_test - 5*embargo_fold)/5\ntrain_len = splits*time_per_fold \ntest_len = (1-splits)*time_per_fold\n\nfold_start = [np.int(i*(len(time_ids)+1)/5) for i in range(6)]\n\nfor i in range(n_fold):\n    time_folds = time_ids[fold_start[i]:fold_start[i+1]-1]\n    df_fold = train_df[train_df.timestamp.isin(time_folds)]\n    df_fold.to_parquet('df_fold_'+str(i)+'.parquet')\n    \ndel train_df\n\ndict_fold = {}\n\nfor fold in range(n_fold):\n    print('fold:'+str(fold))\n    \n    df_fold = pd.read_parquet('df_fold_'+str(fold)+'.parquet')\n    time_ids = df_fold.timestamp.unique()\n    \n    test_train_len = len(time_ids) - embargo_train_test - embargo_fold\n    \n    train_start = embargo_fold + 1\n    train_end = embargo_fold + np.int(test_train_len*0.6) + 1\n    test_start = embargo_fold + np.int(test_train_len*0.6) + embargo_train_test + 1\n    test_end = len(df_fold.timestamp.unique())\n    \n    dict_fold['train_fold_'+str(fold)] = time_ids[train_start:train_end]\n    dict_fold['test_fold_'+str(fold)] = time_ids[test_start:test_end]\n\ndel df_fold","metadata":{"execution":{"iopub.status.busy":"2021-12-27T23:01:05.345370Z","iopub.execute_input":"2021-12-27T23:01:05.345711Z","iopub.status.idle":"2021-12-27T23:01:05.765952Z","shell.execute_reply.started":"2021-12-27T23:01:05.345676Z","shell.execute_reply":"2021-12-27T23:01:05.764988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nimport os\nfrom random import random\n\nsampling = 0.05\n\nMA_lags = [2,5,15,30,60,120,300,1800,3750,2*3750,7*24*60]\nbeta_lags = [15,30,60,120,300,600,1800,3750,2*3750,7*24*60]\n\nFeatures_names = ['log_ret','log_ret_H','log_ret_L','log_ret_VWAP','GK_vol','RS_vol','log_Count','log_Volume','log_Dollars','log_Volume_per_trade','log_Dollars_per_trade']\nMarket_Features_names = [s+'_M' for s in Features_names]\nTime_Features_names = ['sin_month','cos_month','sin_day','cos_day','sin_hour','cos_hour','sin_minute','cos_minute']\nMA_Features_names = [s+'_'+str(lag) for lag in MA_lags for s in Features_names ]\nMA_Features_M_names = [s+'_'+str(lag) for lag in MA_lags for s in Market_Features_names]\nbetas_names = ['betas_'+str(lag) for lag in beta_lags]\n\nAll_names = Features_names + Market_Features_names + Time_Features_names + MA_Features_names + MA_Features_M_names + betas_names\n#df_values = pd.DataFrame(values, columns = All_names)\n\nfor fold in range(n_fold):\n    \n    df_train_fold = pd.DataFrame()\n    df_test_fold = pd.DataFrame()\n    \n    df_read = pd.read_parquet(\"df_fold_\"+str(fold)+'.parquet')\n    \n    #instantiation Moving average features dict\n    dict_RM = {}\n    dict_RM_M = {}\n\n    for lag in MA_lags:\n        dict_RM[lag] = RunningMean(lag)\n        dict_RM_M[lag] = RunningMean(lag)\n\n    #instantiation dict betas\n    dict_MM = {}\n    dict_Mr = {}\n    for lag in beta_lags:\n        dict_MM[lag] = RunningMean(lag)\n        dict_Mr[lag] = RunningMean(lag)\n\n    f = ['timestamp','Asset_ID','Count','Open','High','Low','Close','Volume','VWAP','Target']\n    t = df_read['timestamp'].values\n    ids, index = np.unique(t, return_index=True)\n\n    Values = df_read[f].values\n    splits = np.split(Values, index[1:])\n    out = []\n\n    for time_id, x in tqdm(zip(ids.tolist(), splits)):\n        #df = Clean_df(pd.DataFrame(x,columns=f))\n\n        #timestamp,Asset_ID,Count,O,H,L,C,Volume,VWAP,row_id = (test_df[col].values for col in ['timestamp','Asset_ID','Count','Open','High','Low','Close','Volume','VWAP','row_id'])\n        timestamp,Asset_ID,Count,O,H,L,C,Volume, VWAP,Target = Clean_df(x)\n\n        # np.array([Count,O,H,L,C,Price,Volume,VWAP,Dollars,Volume_per_trade,Dollars_per_trade,log_ret,GK_vol,RS_vol])\n        Features = Base_Feature_fn(timestamp,Asset_ID,Count,O,H,L,C,Volume,VWAP)\n\n        #removing wieghts when data is missing so that they don't appears in market\n        weigths_curr = np.where(np.isnan(O),O,weigths)\n        Market_Features = np.nansum(Features*np.expand_dims(weigths_curr,axis=1)/np.nansum(weigths_curr),axis=0)\n        #Market_Features = np.tile(Market_Features,(14,1))\n\n        #np.array((sin_month,cos_month,sin_day,cos_day,sin_hour,cos_hour,sin_minute,cos_minute))\n        time = timestamp_to_date(timestamp[0])\n        Time_Features = Time_Feature_fn(time)\n        #Time_Features = np.tile(Time_Features,(14,1))\n\n        MA_Features = []\n        MA_Features_M  = [] \n\n        for lag in MA_lags:\n            dict_RM[lag].push(Features.copy())\n            dict_RM_M[lag].push(Market_Features.copy())\n\n            MA_Features.append(dict_RM[lag].get_mean())\n            MA_Features_M.append(dict_RM_M[lag].get_mean())\n        \n        #standardise w/ 3750 lag\n        ref = 3750\n        \n        for i in range(len(MA_lags)):\n            if MA_lags[i] == ref:\n                MA_ref = dict_RM[MA_lags[i]].get_mean().copy()\n                MA_M_ref = dict_RM_M[MA_lags[i]].get_mean().copy()\n        \n        \n                \n                \n        Features[:,-6:] = (Features[:,-6:] - MA_ref[:,-6:]).copy()\n        Market_Features[-6:] = (Market_Features[-6:] - MA_M_ref[-6:]).copy()\n                \n        for i in range(len(MA_lags)):\n            MA_Features[i][:,-6:] = (MA_Features[i][:,-6:] - MA_ref[:,-6:]).copy()\n            MA_Features_M[i][-6:] = (MA_Features_M[i][-6:] - MA_M_ref[-6:]).copy()\n\n        MA_Features_agg = np.concatenate(MA_Features,axis=1)\n        MA_Features_M_agg = np.concatenate(MA_Features_M)\n\n        betas = []\n\n        for lag in beta_lags:\n            dict_MM[lag].push(Market_Features[0]**2)\n            dict_Mr[lag].push(Market_Features[0]*Features[:,0])\n            betas.append(np.expand_dims(dict_Mr[lag].get_mean()/dict_MM[lag].get_mean(),axis=1))\n\n        betas = np.concatenate(betas,axis=1)\n        betas = np.nan_to_num(betas, nan=0., posinf=0., neginf=0.) \n\n        values = np.concatenate((Features,np.tile(Market_Features,(14,1)),np.tile(Time_Features,(14,1)),MA_Features_agg,np.tile(MA_Features_M_agg,(14,1)),betas, np.expand_dims(Target,axis=1)),axis=1)\n        \n        if random() < sampling:\n            out.append(np.concatenate((np.expand_dims(timestamp,axis=1),np.expand_dims(Asset_ID,axis=1),np.float32(values)),axis=1))\n    \n    df_out = pd.DataFrame(np.concatenate(out), columns = ['timestamp','Asset_ID'] + All_names + ['Target']).astype({'timestamp': 'int64','Asset_ID': 'int64'})\n    \n    df_out = df_out[~np.isnan(df_out.Target)]\n\n    ind_train = df_out.timestamp.isin(dict_fold['train_fold_'+str(fold)])\n    ind_test = df_out.timestamp.isin(dict_fold['test_fold_'+str(fold)])\n    \n    df_train_fold = df_out[ind_train]\n    df_test_fold = df_out[ind_test]\n    \n    df_train_fold.to_parquet('train_fold_'+str(fold)+'.parquet')\n    df_test_fold.to_parquet('test_fold_'+str(fold)+'.parquet')\n    \n    pd.DataFrame(df_train_fold.mean(),columns=['mean']).to_parquet('mean_fold_'+str(fold)+'.parquet')\n    pd.DataFrame(df_train_fold.std(),columns=['std']).to_parquet('std_fold_'+str(fold)+'.parquet')\n    ","metadata":{"execution":{"iopub.status.busy":"2021-12-27T23:06:27.285878Z","iopub.execute_input":"2021-12-27T23:06:27.286155Z","iopub.status.idle":"2021-12-27T23:06:49.243915Z","shell.execute_reply.started":"2021-12-27T23:06:27.286126Z","shell.execute_reply":"2021-12-27T23:06:49.242825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"execution":{"iopub.status.busy":"2021-12-27T23:04:20.961372Z","iopub.execute_input":"2021-12-27T23:04:20.961720Z","iopub.status.idle":"2021-12-27T23:04:20.973045Z","shell.execute_reply.started":"2021-12-27T23:04:20.961659Z","shell.execute_reply":"2021-12-27T23:04:20.972080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"execution":{"iopub.status.busy":"2021-12-27T23:02:20.669772Z","iopub.execute_input":"2021-12-27T23:02:20.670398Z","iopub.status.idle":"2021-12-27T23:02:20.682949Z","shell.execute_reply.started":"2021-12-27T23:02:20.670367Z","shell.execute_reply":"2021-12-27T23:02:20.682320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\n\npickle.dump(dict_RM, open('dict_RM_4.pkl', 'wb'))\npickle.dump(dict_RM_M, open('dict_RM_M_4.pkl', 'wb'))\npickle.dump(dict_MM, open('dict_MM_4.pkl', 'wb'))\npickle.dump(dict_Mr, open('dict_MR_4.pkl', 'wb'))","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:47:52.079048Z","iopub.execute_input":"2021-12-27T22:47:52.079288Z","iopub.status.idle":"2021-12-27T22:47:52.527999Z","shell.execute_reply.started":"2021-12-27T22:47:52.079259Z","shell.execute_reply":"2021-12-27T22:47:52.527234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='Variance'></a>\n# Variance\n\nFor the moment I get volatility estimators from Garman-Klass estimation on OLHC data. In the future a better estimation might be needed. I share some code below for the second moment (adapted from same stack overflow post). Some code were added to deal with bug but it might not be entirely clean.","metadata":{}},{"cell_type":"code","source":"from __future__ import division\nimport collections\nimport math\n\nclass RunningStats:\n    def __init__(self, WIN_SIZE=20, n_size = 1):\n        self.n = 0\n        self.mean = 0\n        self.run_var = 0\n        self.n_size = n_size\n        self.WIN_SIZE = WIN_SIZE\n        self.past_value = 0\n        self.windows = collections.deque(maxlen=WIN_SIZE+1)\n\n    def clear(self):\n        self.n = 0\n        self.windows.clear()\n\n    def push(self, x):\n        \n        x = fillna_npwhere(x, self.past_value)\n        self.past_value = x\n\n        self.windows.append(x)\n\n        if self.n < self.WIN_SIZE:\n            # Calculating first variance\n            self.n += 1\n            delta = x - self.mean\n            self.mean += delta / self.n\n            self.run_var += delta * (x - self.mean)\n        else:\n            # Adjusting variance\n            x_removed = self.windows.popleft()\n            old_m = self.mean\n            self.mean += (x - x_removed) / self.WIN_SIZE\n            self.run_var += (x + x_removed - old_m - self.mean) * (x - x_removed)\n\n    def get_mean(self):\n        return self.mean if self.n else np.zeros(n_size)\n\n    def get_var(self):\n        return self.run_var / (self.n) if self.n > 1 else np.zeros(self.n_size)\n\n    def get_std(self):\n        return np.sqrt(self.get_var())\n\n    def get_all(self):\n        return list(self.windows)\n\n    def __str__(self):\n        return \"Current window values: {}\".format(list(self.windows))","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:47:52.529087Z","iopub.execute_input":"2021-12-27T22:47:52.529679Z","iopub.status.idle":"2021-12-27T22:47:52.543020Z","shell.execute_reply.started":"2021-12-27T22:47:52.529640Z","shell.execute_reply":"2021-12-27T22:47:52.542110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \n\nfrom tqdm import tqdm\nimport random\n\nFeatures = np.array([Count,O,H,L,C,Price,Volume,VWAP,Dollars,Volume_per_trade,Dollars_per_trade,log_ret,GK_vol,RS_vol])\n\nMarket_Features\nlags = [2,5,15,30,60,120,300,1800,3750,10*24*60,30*24*60]\n\ndict_vol = {}\ndict_vol_M = {}\n\n#instantiation\nfor lag in lags:\n    dict_vol[lag] = RunningStats(lag)\n    dict_vol_M[lag] = RunningStats(lag)\n\nfor i in tqdm(range(10000)):\n    \n    vol = []\n    vol_M = []\n    \n    for lag in lags:\n        dict_vol[lag].push(Features[0]+0.001*np.array([random.random() for i in range(14)]))\n        dict_vol_M[lag].push(Market_Features[0]+0.005*np.array([random.random() for i in range(14)]))\n        \n        vol.append(dict_vol[lag].get_var())\n        vol_M.append(dict_vol_M[lag].get_var())","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:47:52.545975Z","iopub.execute_input":"2021-12-27T22:47:52.546253Z","iopub.status.idle":"2021-12-27T22:48:00.100728Z","shell.execute_reply.started":"2021-12-27T22:47:52.546224Z","shell.execute_reply":"2021-12-27T22:48:00.099747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dict_vol_M[lag].get_std()","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:48:00.123600Z","iopub.execute_input":"2021-12-27T22:48:00.123846Z","iopub.status.idle":"2021-12-27T22:48:00.132487Z","shell.execute_reply.started":"2021-12-27T22:48:00.123816Z","shell.execute_reply":"2021-12-27T22:48:00.131792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Seems to work (better use a non constant input)","metadata":{}},{"cell_type":"markdown","source":"<a id='FE_exploration'></a>\n# Complete Feature Exploration","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nfor c in df_train_fold.columns:\n    if c == 'Asset_ID':\n        continue\n    print(c)\n    print(df_train_fold[c].describe())\n    df_plot = df_train_fold[[c,'Asset_ID']].pivot(columns='Asset_ID')\n    df_plot[c].plot(kind = 'hist', stacked=True, bins=100).set_xlim((np.min(df_plot[c].quantile(0.025)),np.max(df_plot[c].quantile(0.975))))\n    #plt.hist(df_train_fold[c],bins=100)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-27T22:48:00.133916Z","iopub.execute_input":"2021-12-27T22:48:00.134172Z","iopub.status.idle":"2021-12-27T22:48:00.163051Z","shell.execute_reply.started":"2021-12-27T22:48:00.134131Z","shell.execute_reply":"2021-12-27T22:48:00.161737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}