{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This is a short notebook to show a simple way to extract trade indicators from the given data for those, who like myself, aren't very knowledgable about this domain. This method is based on the methodology as described in [this paper by Ortu et al](https://arxiv.org/pdf/2102.08189.pdf)\n\nThe authors categorize the features to their model as:\n* Technical indicators: Such as those already provided to us in this competition\n* Trade indicators: Additional features calculated from the technical indicators such as Relative Strength Index(RSI), price momentum index etc. \n* Social indicators: These include features obtained through sentiment analysis of social media posts\n\nDue to the constraints of this competition, we cannot add social indicators, but we can compute the trade indicators from data already provided. This notebook will show you how. ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom datetime import datetime\nimport os\nimport matplotlib.pyplot as plt\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:55:29.980357Z","iopub.execute_input":"2021-11-17T09:55:29.981105Z","iopub.status.idle":"2021-11-17T09:55:33.404491Z","shell.execute_reply.started":"2021-11-17T09:55:29.981001Z","shell.execute_reply":"2021-11-17T09:55:33.403482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Import all the usual packages and read the data as you usually do.","metadata":{}},{"cell_type":"code","source":"data_path = '../input/g-research-crypto-forecasting'\nassets = pd.read_csv(os.path.join(data_path, 'asset_details.csv'))\ntrain_df = pd.read_csv(os.path.join(data_path, 'train.csv'))\ntrain_df['asset_name'] = train_df.Asset_ID.map(assets.set_index('Asset_ID').Asset_Name)\nprint(f'There are {len(train_df)} rows in the dataset')\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T11:45:46.524415Z","iopub.execute_input":"2021-11-17T11:45:46.524774Z","iopub.status.idle":"2021-11-17T11:45:46.595807Z","shell.execute_reply.started":"2021-11-17T11:45:46.524693Z","shell.execute_reply":"2021-11-17T11:45:46.594669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import time\ntotimestamp = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:32.647102Z","iopub.execute_input":"2021-11-17T09:56:32.647376Z","iopub.status.idle":"2021-11-17T09:56:32.654744Z","shell.execute_reply.started":"2021-11-17T09:56:32.647339Z","shell.execute_reply":"2021-11-17T09:56:32.653297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To demonstrate this method I'm going to choose only a small subset of the data. Here I'm extracting only data from bitcoin for a three month period. ","metadata":{}},{"cell_type":"code","source":"bit = train_df[train_df.Asset_ID == 1].set_index('timestamp')\nbit = bit.loc[totimestamp('01/01/2021'):totimestamp('01/04/2021')]\nbit.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:32.656860Z","iopub.execute_input":"2021-11-17T09:56:32.657192Z","iopub.status.idle":"2021-11-17T09:56:33.084060Z","shell.execute_reply.started":"2021-11-17T09:56:32.657148Z","shell.execute_reply":"2021-11-17T09:56:33.083234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, to extract these features we'll be using the stockstats package. You can find the documentation [here](https://pypi.org/project/stockstats/). It is essentially a wrapper to a pandas DataFrame that is preloaded with formulas for about 30 indicators. \n\nAll you have to do is use the *retype* function to cast your pandas DataFrame into a *StockDataFrame*.\n\nThis package assumes that your data is sorted by time and contains certain columns. You also have to make sure your column names match what it expects. Here are the columns it requires:\n\n* open: corresponding to Open in our dataset\n* close: corresponding to Close in our dataset\n* high: corresponding to High in our dataset\n* low: corresponding to Low in our dataset\n* volume: corresponding to Volume in our dataset\n* amount: corresponding to Count in our dataset\n","metadata":{}},{"cell_type":"code","source":"#Installing and importing the package\n!pip install stockstats\nfrom stockstats import StockDataFrame","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Preparing the columns \nbit.drop(['Asset_ID','asset_name', 'VWAP', 'Target'], axis=1, inplace=True)\nbit.rename(columns = {'Count':'amount','Open':'open','High':'high','Low':'low',\n                     'Close':'close','Volume':'volume'}, inplace=True)\nbit.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stock = StockDataFrame.retype(bit)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:33.189925Z","iopub.execute_input":"2021-11-17T09:56:33.190425Z","iopub.status.idle":"2021-11-17T09:56:33.197562Z","shell.execute_reply.started":"2021-11-17T09:56:33.190361Z","shell.execute_reply":"2021-11-17T09:56:33.196728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before you calculate the features you can tune the default values as I have done below. For complete list of options that are available for tuning and extraction refer the docs.","metadata":{}},{"cell_type":"code","source":"KDJ_WINDOW = 10\nBOLL_WINDOW = 10\nMACD_EMA_SHORT = 10\nPDI_SMMA = 10\nMDI_SMMA = 10\nDX_SMMA = 10\nADX_EMA = 5\nADXR_EMA = 10\nTRIX_EMA_WINDOW = 10\nTEMA_EMA_WINDOW = 10\n","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:33.199199Z","iopub.execute_input":"2021-11-17T09:56:33.199642Z","iopub.status.idle":"2021-11-17T09:56:33.206838Z","shell.execute_reply.started":"2021-11-17T09:56:33.199553Z","shell.execute_reply":"2021-11-17T09:56:33.205973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Creating the features\nbit['volume_delta'] = stock['volume_delta']\nbit['open_-2_r'] = stock['open_-2_r']\nbit['cr']= stock['cr']\nbit['kdjk'] = stock['kdjk']\nbit['open_2_sma'] = stock['open_2_sma']\nbit['macd'] = stock['macd']\nbit['boll'] = stock['boll']\nbit['boll_ub'] = stock['boll_ub']\nbit['boll_lb'] = stock['boll_lb']\nbit['rsi_12'] = stock['rsi_12']\nbit['wr_10'] = stock['wr_10']\nbit['cci'] = stock['cci']\nbit['tr'] = stock['tr']\nbit['atr'] = stock['atr']\nbit['dma'] = stock['dma']\nbit['pdi'] = stock['pdi']\nbit['dx'] = stock['dx']\nbit['adx'] = stock['adx']\nbit['adxr'] = stock['adxr']\nbit['trix'] = stock['trix']\nbit['tema'] = stock['tema']\nbit['vr'] = stock['vr']","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:33.208503Z","iopub.execute_input":"2021-11-17T09:56:33.208895Z","iopub.status.idle":"2021-11-17T09:56:33.216848Z","shell.execute_reply.started":"2021-11-17T09:56:33.208850Z","shell.execute_reply":"2021-11-17T09:56:33.215992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bit.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:33.219344Z","iopub.execute_input":"2021-11-17T09:56:33.219955Z","iopub.status.idle":"2021-11-17T09:56:33.238259Z","shell.execute_reply.started":"2021-11-17T09:56:33.219909Z","shell.execute_reply":"2021-11-17T09:56:33.237349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I wanted to see how these features would perform in a model, so I built a small model to see if there's any improvement while using these features. Instead of building a model to predict the target as we're doing in this competition, I created a new target for a classification problem as was done in the paper stated above. A '1' corresponds to a price increase and '0' to a decrease. So, essentially this model will predict if the price of bitcoin will increase or decrease in the next minute. ","metadata":{}},{"cell_type":"code","source":"#Creating new target\nbit['target'] = bit.open.shift(-1) - bit.close\nbit['target'] = bit.target.apply(lambda x:1 if x>0 else 0)\nbit.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:33.240040Z","iopub.execute_input":"2021-11-17T09:56:33.241154Z","iopub.status.idle":"2021-11-17T09:56:33.251396Z","shell.execute_reply.started":"2021-11-17T09:56:33.241110Z","shell.execute_reply":"2021-11-17T09:56:33.250636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bit.target.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:33.371873Z","iopub.execute_input":"2021-11-17T09:56:33.372503Z","iopub.status.idle":"2021-11-17T09:56:33.385049Z","shell.execute_reply.started":"2021-11-17T09:56:33.372458Z","shell.execute_reply":"2021-11-17T09:56:33.383844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = bit.drop(['target'],axis=1).values\ny = bit.target.values","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:33.396732Z","iopub.execute_input":"2021-11-17T09:56:33.397352Z","iopub.status.idle":"2021-11-17T09:56:33.405512Z","shell.execute_reply.started":"2021-11-17T09:56:33.397308Z","shell.execute_reply":"2021-11-17T09:56:33.404474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_size = int(len(X)*0.70)\ntrain, test = X[0:train_size], X[train_size:len(X)]\ny_train, y_test = y[0:train_size], y[train_size:len(X)]\nprint('Observations: %d' % (len(X)))\nprint('Training Observations: %d' % (len(train)))\nprint('Testing Observations: %d' % (len(test)))","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:33.418478Z","iopub.execute_input":"2021-11-17T09:56:33.419012Z","iopub.status.idle":"2021-11-17T09:56:33.427863Z","shell.execute_reply.started":"2021-11-17T09:56:33.418933Z","shell.execute_reply":"2021-11-17T09:56:33.426502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from xgboost import XGBClassifier\nmodel = XGBClassifier( learning_rate=0.05, n_estimators=500, \n                      subsample=1.0,alpha=0, tree_method = 'gpu_hist')\nmodel.fit(train, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:33.429987Z","iopub.execute_input":"2021-11-17T09:56:33.430711Z","iopub.status.idle":"2021-11-17T09:56:36.876909Z","shell.execute_reply.started":"2021-11-17T09:56:33.430667Z","shell.execute_reply":"2021-11-17T09:56:36.876202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = model.predict(test)","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:36.880648Z","iopub.execute_input":"2021-11-17T09:56:36.881317Z","iopub.status.idle":"2021-11-17T09:56:37.209505Z","shell.execute_reply.started":"2021-11-17T09:56:36.881264Z","shell.execute_reply":"2021-11-17T09:56:37.208843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score\naccuracy = accuracy_score(y_test, y_pred)\nprint(f'model accuracy is :{accuracy}')\n\nfrom sklearn.metrics import precision_score\nprecision = precision_score(y_test, y_pred, average='binary')\nprint(f'model precision is :{precision}')\n\nfrom sklearn.metrics import recall_score\nrecall = recall_score(y_test, y_pred, average='binary')\nprint(f'model recall is :{recall}')","metadata":{"execution":{"iopub.status.busy":"2021-11-17T09:56:37.213538Z","iopub.execute_input":"2021-11-17T09:56:37.215377Z","iopub.status.idle":"2021-11-17T09:56:37.226199Z","shell.execute_reply.started":"2021-11-17T09:56:37.215343Z","shell.execute_reply":"2021-11-17T09:56:37.225322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is a 3% increase in model accuracy while using these newly extracted features(Run this notebook while commenting out the extracted features to obtain the score without). \n\nNote: This is not a definitive evaluation, I have not stress tested this with bigger data, optimised the algorithm or even evaluated which of, or even all, these features affect the model score. The goal of this notebook was purely to show the feature extraction.\n\nI hope you have enjoyed this notebook and that this helps you in building better models for yourselves. ","metadata":{}}]}