{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":30894,"databundleVersionId":3147400,"sourceType":"competition"},{"sourceId":3011023,"sourceType":"datasetVersion","datasetId":1844304}],"dockerImageVersionId":30153,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G-Research Crypto Forecasting\n\nCryptocurrency ('Crypto') is a digital currency based on blockchain technology. This currency can be traded like another assets of investment. This means, crypto can give profit but you must take on a fair amount of research to fully understand how each system works.\n\nG-Research is Europe’s leading quantitative finance research firm. G-Research have long explored the extent of market prediction possibilities, making use of machine learning, big data, and some of the most advanced technology available, including crypto market.\n\nUsing machine learning with this dataset, we can forecast short term returns in 14 popular cryptocurrencies. We have amassed a dataset of millions of rows of high-frequency market data dating back to 2018 which you can use to build your model. Once the submission deadline has passed, your final score will be calculated over the following 3 months using live crypto data as it is collected.\n\nIn addition, since 2018, interest in the cryptomarket has exploded, so the volatility and correlation structure in our data are likely to be highly non-stationary. The successful contestant will pay careful attention to these considerations, and in the process gain valuable insight into the art and science of financial forecasting.","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nimport os\n!ls ../input/imgeee","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-01T14:17:53.532624Z","iopub.execute_input":"2022-02-01T14:17:53.533134Z","iopub.status.idle":"2022-02-01T14:17:54.226145Z","shell.execute_reply.started":"2022-02-01T14:17:53.533043Z","shell.execute_reply":"2022-02-01T14:17:54.225328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Image(\"../input/imgeee/624b0264e17758f4a4394526096afc03.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-01T14:17:54.227764Z","iopub.execute_input":"2022-02-01T14:17:54.227971Z","iopub.status.idle":"2022-02-01T14:17:54.255335Z","shell.execute_reply.started":"2022-02-01T14:17:54.227945Z","shell.execute_reply":"2022-02-01T14:17:54.254636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This dataset have some columns that divided in some tables, there are:\n\nAsset_ID - An ID code for the cryptoasset.\n\nWeight  - Weight for the cryptoasset.\n\nAsset_Name - Name for the cryptoasset.\n\ngroup_num - when env.iter_test() is called to iterate through the test set, it yields one group per iteration.\n\nrow_id - id of row in test dataset\n\nTarget  - 15 minute residualized returns.\n\ntimestamp - A timestamp for the minute covered by the row.\n\nCount - The number of trades that took place this minute.\n\nOpen - The USD price at the beginning of the minute.\n\nHigh - The highest USD price during the minute.\n\nLow - The lowest USD price during the minute.\n\nClose - The USD price at the end of the minute.\n\nVolume - The number of cryptoasset units traded during the minute.\n\nVWAP - The volume weighted average price for the minute.\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# 1. Load datasets","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom datetime import datetime\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n\nimport lightgbm as ltb\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\n# simple preprocessing of the data \nscaler = StandardScaler()\nfrom sklearn.model_selection import TimeSeriesSplit, cross_val_score\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import Ridge, Lasso, ElasticNet, LinearRegression\nfrom sklearn.svm import SVR\nfrom sklearn.neighbors import KNeighborsRegressor\n\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-01T14:17:54.256323Z","iopub.execute_input":"2022-02-01T14:17:54.256626Z","iopub.status.idle":"2022-02-01T14:17:57.526298Z","shell.execute_reply.started":"2022-02-01T14:17:54.256591Z","shell.execute_reply":"2022-02-01T14:17:57.525449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission = '/kaggle/input/g-research-crypto-forecasting/example_sample_submission.csv'\nasset_detail = '/kaggle/input/g-research-crypto-forecasting/asset_details.csv'\nexample_test = '/kaggle/input/g-research-crypto-forecasting/example_test.csv'\ntrain = '/kaggle/input/g-research-crypto-forecasting/train.csv'\nsup_train = '/kaggle/input/g-research-crypto-forecasting/supplemental_train.csv'\n\nsample_submit = pd.read_csv(sample_submission)\nasset = pd.read_csv(asset_detail)\ntrain = pd.read_csv(train)\nsup_train = pd.read_csv(sup_train)\ntest = pd.read_csv(example_test)","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:17:57.528692Z","iopub.execute_input":"2022-02-01T14:17:57.528965Z","iopub.status.idle":"2022-02-01T14:18:56.72955Z","shell.execute_reply.started":"2022-02-01T14:17:57.528931Z","shell.execute_reply":"2022-02-01T14:18:56.728808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"row = sample_submit.merge(train, on = 'Target', how = 'left')\n","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:18:56.730629Z","iopub.execute_input":"2022-02-01T14:18:56.730868Z","iopub.status.idle":"2022-02-01T14:19:19.739958Z","shell.execute_reply.started":"2022-02-01T14:18:56.730835Z","shell.execute_reply":"2022-02-01T14:19:19.739226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"row.groupby('Asset_ID')['group_num'].count().reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:19.741356Z","iopub.execute_input":"2022-02-01T14:19:19.741628Z","iopub.status.idle":"2022-02-01T14:19:19.800299Z","shell.execute_reply.started":"2022-02-01T14:19:19.741595Z","shell.execute_reply":"2022-02-01T14:19:19.799451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Import Library to predict","metadata":{}},{"cell_type":"code","source":"print('sample_submit shape:',sample_submit.shape)\nprint('asse shape:',asset.shape)\nprint('train shape:',train.shape)\nprint('sup_train shape:',sup_train.shape)\nprint('test shape:',test.shape)\n\n4847361,2","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:19.801804Z","iopub.execute_input":"2022-02-01T14:19:19.802074Z","iopub.status.idle":"2022-02-01T14:19:19.811227Z","shell.execute_reply.started":"2022-02-01T14:19:19.802038Z","shell.execute_reply":"2022-02-01T14:19:19.81045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.concat([train, sup_train], levels = 'timestamp')","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:19.81264Z","iopub.execute_input":"2022-02-01T14:19:19.813153Z","iopub.status.idle":"2022-02-01T14:19:20.693306Z","shell.execute_reply.started":"2022-02-01T14:19:19.813113Z","shell.execute_reply":"2022-02-01T14:19:20.692579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:20.695422Z","iopub.execute_input":"2022-02-01T14:19:20.695845Z","iopub.status.idle":"2022-02-01T14:19:21.120949Z","shell.execute_reply.started":"2022-02-01T14:19:20.695809Z","shell.execute_reply":"2022-02-01T14:19:21.120236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:21.124236Z","iopub.execute_input":"2022-02-01T14:19:21.124439Z","iopub.status.idle":"2022-02-01T14:19:21.137838Z","shell.execute_reply.started":"2022-02-01T14:19:21.124414Z","shell.execute_reply":"2022-02-01T14:19:21.137198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{}},{"cell_type":"code","source":"def hlco_ratio(df): \n    return np.log(df['High']) - (np.log(df['Low'])/np.log(df['Close'])- np.log(df['Open']))\ndef upper_shadow(df):\n    return np.log(df['High']) - np.maximum(np.log(df['Close']), np.log(df['Open']))\ndef lower_shadow(df):\n    return np.minimum(np.log(df['Close']), (np.log(df['Open']) - np.log(df['Low'])))\n\ndef get_features(df):\n    df_feat = df[['year', 'month', 'day', 'weekday', 'hour', 'minute', 'Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP', 'Target']].copy()\n    df_feat['Upper_Shadow'] = upper_shadow(df_feat)\n    df_feat['hlco_ratio'] = hlco_ratio(df_feat)\n    df_feat['Lower_Shadow'] = lower_shadow(df_feat)\n    return df_feat","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:21.139128Z","iopub.execute_input":"2022-02-01T14:19:21.139603Z","iopub.status.idle":"2022-02-01T14:19:21.149362Z","shell.execute_reply.started":"2022-02-01T14:19:21.139567Z","shell.execute_reply":"2022-02-01T14:19:21.148643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" df_train = train.copy()\n","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:21.150775Z","iopub.execute_input":"2022-02-01T14:19:21.151023Z","iopub.status.idle":"2022-02-01T14:19:21.861586Z","shell.execute_reply.started":"2022-02-01T14:19:21.150989Z","shell.execute_reply":"2022-02-01T14:19:21.860842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Clean the null values","metadata":{}},{"cell_type":"code","source":"def ResidualizeMarket(df, mktColumn, window):\n  if mktColumn not in df.columns:\n    return df\n\n  mkt = df[mktColumn]\n\n  num = df.multiply(mkt.values, axis=0).rolling(window).mean().values  #numerator of linear regression coefficient\n  denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values  #denominator of linear regression coefficient\n  beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.)  #if regression fell over, use beta of 0\n\n  resultRet = df - (beta * mkt.values).T  #perform residualization\n  resultBeta = 0.*df + beta.T  #shape beta\n\n  return resultRet.drop(columns=[mktColumn]), resultBeta.drop(columns=[mktColumn])\n\n# # Or using KNN Imputer to handle missing values\n\n# from sklearn.experimental import enable_iterative_imputer\n# from sklearn.impute import IterativeImputer\n\n# imp_iter = IterativeImputer(max_iter=10, random_state=0)\n# df_train[['VWAP', 'Target']] = imp_iter.fit_transform(df_train[['VWAP', 'Target']])\n\n# Function log_return_ahead computes R_t = log(P_{t+16} / P_{t+1})\ndef log_return_ahead(series, periods=1): \n    return -np.log(series).diff(periods=-periods).shift(-1)","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:21.862985Z","iopub.execute_input":"2022-02-01T14:19:21.863238Z","iopub.status.idle":"2022-02-01T14:19:21.872846Z","shell.execute_reply.started":"2022-02-01T14:19:21.863193Z","shell.execute_reply":"2022-02-01T14:19:21.870616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prices = train.pivot(index=[\"timestamp\"], columns=[\"Asset_ID\"], values=[\"Close\"])","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:21.874878Z","iopub.execute_input":"2022-02-01T14:19:21.876232Z","iopub.status.idle":"2022-02-01T14:19:31.14048Z","shell.execute_reply.started":"2022-02-01T14:19:21.87618Z","shell.execute_reply":"2022-02-01T14:19:31.139725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# prices = prices.reindex(range(prices.index[0], prices.index[-1]+60,60), method='pad')","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:31.141836Z","iopub.execute_input":"2022-02-01T14:19:31.142079Z","iopub.status.idle":"2022-02-01T14:19:31.146635Z","shell.execute_reply.started":"2022-02-01T14:19:31.142049Z","shell.execute_reply":"2022-02-01T14:19:31.145224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prices.index = prices.index.map(lambda x: datetime.fromtimestamp(x))","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:31.147817Z","iopub.execute_input":"2022-02-01T14:19:31.148196Z","iopub.status.idle":"2022-02-01T14:19:35.157337Z","shell.execute_reply.started":"2022-02-01T14:19:31.148161Z","shell.execute_reply":"2022-02-01T14:19:35.156597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"log_returns_15min = log_return_ahead(prices, periods=15)","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:35.161707Z","iopub.execute_input":"2022-02-01T14:19:35.163845Z","iopub.status.idle":"2022-02-01T14:19:35.715772Z","shell.execute_reply.started":"2022-02-01T14:19:35.163803Z","shell.execute_reply":"2022-02-01T14:19:35.71502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"weights = asset.Weight.values","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:35.716879Z","iopub.execute_input":"2022-02-01T14:19:35.719334Z","iopub.status.idle":"2022-02-01T14:19:35.725397Z","shell.execute_reply.started":"2022-02-01T14:19:35.719303Z","shell.execute_reply":"2022-02-01T14:19:35.723163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"weighted_avg_market_log_returns = log_returns_15min.mul(weights, axis='columns').mean(axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:35.726403Z","iopub.execute_input":"2022-02-01T14:19:35.726645Z","iopub.status.idle":"2022-02-01T14:19:35.853077Z","shell.execute_reply.started":"2022-02-01T14:19:35.726611Z","shell.execute_reply":"2022-02-01T14:19:35.852366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"log_returns_15min[\"market\"] = weighted_avg_market_log_returns\nresidualized_market_returns, beta = ResidualizeMarket(log_returns_15min, \"market\", window=3750)","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:35.854224Z","iopub.execute_input":"2022-02-01T14:19:35.854495Z","iopub.status.idle":"2022-02-01T14:19:37.961365Z","shell.execute_reply.started":"2022-02-01T14:19:35.85446Z","shell.execute_reply":"2022-02-01T14:19:37.960551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"residualized_market_returns","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:37.965791Z","iopub.execute_input":"2022-02-01T14:19:37.967739Z","iopub.status.idle":"2022-02-01T14:19:38.005821Z","shell.execute_reply.started":"2022-02-01T14:19:37.967699Z","shell.execute_reply":"2022-02-01T14:19:38.005186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"residualized_market_returns.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:38.009374Z","iopub.execute_input":"2022-02-01T14:19:38.011245Z","iopub.status.idle":"2022-02-01T14:19:38.086142Z","shell.execute_reply.started":"2022-02-01T14:19:38.011207Z","shell.execute_reply":"2022-02-01T14:19:38.085532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since, there is lot of null value using this calc, it is mean return is null(0) so missing value in Target column changed with 0(null).","metadata":{}},{"cell_type":"markdown","source":"## Reduce Memory Usage","metadata":{}},{"cell_type":"code","source":"def reduce_mem_usage(df):\n  \n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n    \n    for col in df.columns:\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n#         else:\n#             df[col] = df[col].astype('category')\n\n    df[\"Count\"] = df[\"Count\"].astype(np.int16)\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    \n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n     \n    return df\n\ndf_train = reduce_mem_usage(df_train)","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:38.089747Z","iopub.execute_input":"2022-02-01T14:19:38.091612Z","iopub.status.idle":"2022-02-01T14:19:42.36502Z","shell.execute_reply.started":"2022-02-01T14:19:38.091573Z","shell.execute_reply":"2022-02-01T14:19:42.364256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above is the function to reduce the data memory. Data type Float64 is changed, and then Training data was reduced from 1.8GB to 0.5GB. Data type change works well like this.","metadata":{}},{"cell_type":"markdown","source":"## Training Model","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def timing(df_train):\n    df_train['Date'] = df_train[\"timestamp\"].map(lambda x: datetime.fromtimestamp(x))\n\n    df_train['year'] = df_train['Date'].dt.year\n    df_train['month'] = df_train['Date'].dt.month\n    df_train['day'] = df_train['Date'].dt.day\n    df_train['weekday'] = df_train['Date'].dt.weekday\n    df_train['hour'] = df_train['Date'].dt.hour\n    df_train['minute'] = df_train['Date'].dt.minute\n    \n    return df_train","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:42.366535Z","iopub.execute_input":"2022-02-01T14:19:42.3668Z","iopub.status.idle":"2022-02-01T14:19:42.372829Z","shell.execute_reply.started":"2022-02-01T14:19:42.366763Z","shell.execute_reply":"2022-02-01T14:19:42.372119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample = df_train[df_train['Asset_ID'] == 1]\nsample = timing(sample)\nsample = get_features(sample)\nsample = sample[['year', 'month', 'day', 'weekday', 'hour', 'minute', 'Count', 'Volume', 'VWAP', 'Upper_Shadow', 'Lower_Shadow', 'hlco_ratio', 'Target']]\nsample.replace([np.inf, -np.inf], np.nan, inplace=True)\nsample = sample.fillna(0)","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:42.374646Z","iopub.execute_input":"2022-02-01T14:19:42.37524Z","iopub.status.idle":"2022-02-01T14:19:46.261884Z","shell.execute_reply.started":"2022-02-01T14:19:42.375179Z","shell.execute_reply":"2022-02-01T14:19:46.261042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import xgboost as xgb ","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:46.263511Z","iopub.execute_input":"2022-02-01T14:19:46.263809Z","iopub.status.idle":"2022-02-01T14:19:46.337996Z","shell.execute_reply.started":"2022-02-01T14:19:46.26377Z","shell.execute_reply":"2022-02-01T14:19:46.337252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# #find the best model\n# X, y = sample.drop('Target',1), sample.Target\n# X = scaler.fit_transform(X)\n# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.22\n#                                                     , shuffle=True,random_state = 24)\n\n# models = []\n# models.append(('Ridge', Ridge())) \n# models.append(('Lasso', Lasso()))\n# models.append(('E-Net', ElasticNet()))\n# models.append(('SVR', SVR(gamma='auto')))\n# models.append(('KNN', KNeighborsRegressor())) \n# models.append(('LGBMRegressor', ltb.LGBMRegressor()))\n# models.append(('XGBRegressor',xgb.XGBRegressor()))\n\n\n# tscv = TimeSeriesSplit(n_splits=10)\n\n# # Evaluate each model in turn\n# results = []\n# names = []\n# for name, model in models:\n#   # TimeSeries Cross validation\n  \n#   cv_results = cross_val_score(model, X_train, y_train, cv=tscv, scoring='r2')\n#   results.append(cv_results)\n#   names.append(name)\n#   print ('%s: %f (%f)' % (name, cv_results.mean(), cv_results.std()))\n\n# # Compare Algorithms\n# # plt.boxplot(results, labels=names)\n# # plt.title('Algorithm Comparison')\n# # plt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:46.33953Z","iopub.execute_input":"2022-02-01T14:19:46.339805Z","iopub.status.idle":"2022-02-01T14:19:46.344648Z","shell.execute_reply.started":"2022-02-01T14:19:46.339766Z","shell.execute_reply":"2022-02-01T14:19:46.343735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train test split df_train into 80% train rows and 20% valid rows\n#train_data = train\n# train_data = df_train.sample(frac = 0.8)\n# valid_data = df_train.drop(train_data.index)\n\ndef get_Xy_and_model_for_asset(df_train, asset_id):\n    df = df_train[df_train[\"Asset_ID\"] == asset_id]\n    \n    df = df.sample(frac = 0.2)\n    df_proc = df\n    df_proc = timing(df_proc)\n    df_proc = get_features(df_proc)\n    df_proc[['year', 'month', 'day', 'weekday', 'hour', 'minute', 'Count', 'Volume', 'VWAP', 'Upper_Shadow', 'Lower_Shadow', 'hlco_ratio', 'Target']] = df_proc[['year', 'month', 'day', 'weekday',  'hour', 'minute', 'Count', 'Volume', 'VWAP', 'Upper_Shadow', 'Lower_Shadow', 'hlco_ratio', 'Target']]\n#     df_proc['y'] = df['Target']\n    df_proc.replace([np.inf, -np.inf], np.nan, inplace=True)\n    df_proc = df_proc.fillna(0)\n    X = df_proc.drop('Target', axis = 1)\n\n    X = scaler.fit_transform(X)\n    \n    y = df_proc[\"Target\"]   \n    model = ltb.LGBMRegressor()\n    \n#     KNeighborsRegressor()\n    \n    \n    model.fit(X, y)\n    return X, y, model\n\nXs = {}\nys = {}\nmodels = {}\n \n\n\n\nfor asset_id, asset_name in zip(asset['Asset_ID'], asset['Asset_Name']):\n    print(f\"Training model for {asset_name:<16} (ID={asset_id:<2})\")\n    X, y, model = get_Xy_and_model_for_asset(df_train, asset_id)       \n    try:\n        Xs[asset_id], ys[asset_id], models[asset_id] = X, y, model\n    except: \n        Xs[asset_id], ys[asset_id], models[asset_id] = None, None, None","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:19:46.349777Z","iopub.execute_input":"2022-02-01T14:19:46.350425Z","iopub.status.idle":"2022-02-01T14:20:32.447678Z","shell.execute_reply.started":"2022-02-01T14:19:46.350369Z","shell.execute_reply":"2022-02-01T14:20:32.447064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nparameters = {\n#     'max_depth': range (2, 10, 1),\n    'num_leaves': range(21, 161, 10),\n    'learning_rate': [0.1, 0.01, 0.05]\n}\n\n# {\"n_neighbors\": range(1, 20)}\n# \n\n\n# tscv = TimeSeriesSplit(n_splits=5)\nnew_models = {}\nfor asset_id, asset_name in zip(asset['Asset_ID'], asset['Asset_Name']):\n    print(\"GridSearchCV for: \" + asset_name)\n    grid_search = GridSearchCV(\n        estimator=get_Xy_and_model_for_asset(df_train, asset_id)[2], # bitcoin\n        param_grid=parameters,\n        n_jobs = -1,\n        cv = 20,\n        verbose=True,\n        scoring = 'r2'\n    )\n    grid_search.fit(Xs[asset_id], ys[asset_id])\n    new_models[asset_id] = grid_search.best_estimator_\n    grid_search.best_estimator_\n    print('best score',grid_search.best_score_)","metadata":{"execution":{"iopub.status.busy":"2022-02-01T14:20:32.451335Z","iopub.execute_input":"2022-02-01T14:20:32.452897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission ","metadata":{}},{"cell_type":"code","source":"!pip install env","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gresearch_crypto\n# env = gresearch_crypto.make_env()\niter_test = env.iter_test()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_pred","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\nfor i, (df_test, df_pred) in enumerate(iter_test):\n    df_test = reduce_mem_usage(df_test)\n    for j , row in df_test.iterrows():        \n        if new_models[row['Asset_ID']] is not None:\n            try:\n                model = new_models[row['Asset_ID']]\n                row = timing(row)\n                x_test = get_features(row)\n                y_pred = model.predict(pd.DataFrame([x_test]))[0]\n                df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = y_pred\n            except:\n                df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0\n                traceback.print_exc()\n        else: \n            df_pred.loc[df_pred['row_id'] == row['row_id'], 'Target'] = 0  \n    \n    env.predict(df_pred)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.read_csv('/kaggle/working/submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}