{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"56e6aaf3-f600-48d8-9b40-68ca001cf084","_cell_guid":"d2c43fb6-edff-440a-82a2-869bb06f8215","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# G-Research 01: Data Reading, Formatting and Grouping by Time\n\nIn this notebook, I would like to share some useful functionality for G-Research project.\n- Data formatting and keeping the names of attributes as named tuple to be able to use them anywhere in the poject in straighforward way. This approach allows you to change the name of attribute on one place without affecting the code.\n- Grouping by time. This is very important feature. The basic goal is to get the trading data set with all its attributes, but on different scale.","metadata":{}},{"cell_type":"markdown","source":"## General Python Libraries","metadata":{}},{"cell_type":"code","source":"from datetime import datetime","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## My Code Exported from Scripts\n\nI always keep huge chunks of code outside of the notebook (aka client-server pattern). In that case, I can reuse the code anywhere and anytime and create shorter and better readable notebooks.","metadata":{}},{"cell_type":"code","source":"from g_research_data import A\nfrom g_research_transformations import DFAttributesTypeTransformer\nfrom g_research_transformations import ManyToDatetimeIndexTransformer\nfrom g_research_transformations import GroupDFBlocksByTimeTransformer","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Reading the Data\n\nFirst of all, I read the data from the file.\n\n> **NOTE:** In this notebook, I am using just a sample of the data, of course can be replaced by the whole data set. To take all the data, just delete the code in comment.","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv('../input/g-research-crypto-forecasting/train.csv')\nprint(data.shape)\ndata.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **TO USE THE WHOLE DATA SET JUST DELETE THE FOLLOWING CELL**","metadata":{}},{"cell_type":"code","source":"# TAKING JUST A SAMPLE ###########################################################\ndata = data.loc[0:10000,]\nprint(data.shape)\ndata.head()\n# END(TAKING JUST A SAMPLE) ######################################################","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Formatting the Data\n\nNext, I am doing following:\n- I created a NamedTuple if attribute names and types. That can be found in *g_research_data* file.\n- I am formatting the timestamp to match my future use.\n- I created a transformer for properly setting the attributes. At this point, I am using *DFAttributesTypeTransformer* from the file *g_research_transformations*.\n\n> NOTE: In the file *g_research_transformations*, I am using the same interface for all the transformation classes. That is why you can see some parent class and then children classes.","metadata":{}},{"cell_type":"code","source":"# Example usage of variable A\nprint(A.timestamp)\nprint(A.timestamp.name)\nprint(A.timestamp.type)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[A.timestamp.name] = [datetime.fromtimestamp(ts) for ts in data[A.timestamp.name]]\n\ntype_transformer = DFAttributesTypeTransformer()\ndf = type_transformer.fit_predict(data, dict(A))\n\ndf.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Taking Only One Asset\n\nI am just going to take one asset for showing the functionality.","metadata":{}},{"cell_type":"code","source":"take_asset = 0\ndf_asset = df.loc[df[A.asset_id.name] == take_asset,]\nprint(df_asset.shape)\ndf_asset.head(16)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Grouping Rows Based on Time\n\nPrediction time frame/window is very important in time series. I created a transformer class that can do the grouping based on time window and create attributes relevant to cryptocurrency trading.  \n\nThe transformer *GroupDFBlocksByTimeTransformer* from the file *g_research_transformations* is used as follows:\n- First parameter is a data frame with date time attribute in datetime index format and another column to be transformer. \n- Second parameter is the name of the date time attribute.\n- Third parameter is the window lenght as in [here](https://pandas.pydata.org/docs/reference/api/pandas.Series.dt.floor.html). Examples:\n    - \"15min\", \"75min\", ... for minutes,\n    - \"1H\", ... for hours,\n    - \"1D\", ... for days.\n- Fourth parameter is the name of grouping function. Currently available are: \"mean\", \"max\", \"min\", \"first\", \"last\".\n\nLet us have a look at examples useful for our usecase.","metadata":{}},{"cell_type":"code","source":"grouping_transformer = GroupDFBlocksByTimeTransformer()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"window_length = \"10min\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In following cell I am going to create a grouped *open* column. It is the first value for that interval. \n\nYou can check the values in the first cell of the chapter *Taking Only One Asset*.","metadata":{}},{"cell_type":"code","source":"grouping_function = \"first\"\nattribute_from = A.open.name\n\ndf_grouped = grouping_transformer.fit_predict(\n    df_asset[[A.timestamp.name, attribute_from]], \n    A.timestamp.name, \n    window_length, \n    grouping_function\n)\ndf_grouped.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In following cell I am going to create a grouped *high* column. It is the max value for that interval:\n\nYou can check the values in the first cell of the chapter *Taking Only One Asset*.","metadata":{}},{"cell_type":"code","source":"grouping_function = \"max\"\nattribute_from = A.high.name\n\ndf_grouped = grouping_transformer.fit_predict(\n    df_asset[[A.timestamp.name, attribute_from]], \n    A.timestamp.name, \n    window_length, \n    grouping_function\n)\ndf_grouped.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Recreating the Data Frame\n\nLet us use the former to recreate all the OHLC attributes (Open, High, Low, Close) for window length 10 minutes.","metadata":{}},{"cell_type":"code","source":"window_length = \"10min\"\n\n# Open\ngrouping_function = \"first\"\nattribute_from = A.open.name\ndf_final = grouping_transformer.fit_predict(\n    df_asset[[A.timestamp.name, attribute_from]], \n    A.timestamp.name, \n    window_length, \n    grouping_function\n)\n\ndf_final.rename(columns={\"FIRST\":attribute_from}, inplace=True)\n\n# High\ngrouping_function = \"max\"\nattribute_from = A.high.name\ndf_pom = grouping_transformer.fit_predict(\n    df_asset[[A.timestamp.name, attribute_from]], \n    A.timestamp.name, \n    window_length, \n    grouping_function\n)\ndf_final[attribute_from] = df_pom[grouping_function.upper()]\n\n# Low\ngrouping_function = \"min\"\nattribute_from = A.low.name\ndf_pom = grouping_transformer.fit_predict(\n    df_asset[[A.timestamp.name, attribute_from]], \n    A.timestamp.name, \n    window_length, \n    grouping_function\n)\ndf_final[attribute_from] = df_pom[grouping_function.upper()]\n\n# Close\ngrouping_function = \"last\"\nattribute_from = A.close.name\ndf_pom = grouping_transformer.fit_predict(\n    df_asset[[A.timestamp.name, attribute_from]], \n    A.timestamp.name, \n    window_length, \n    grouping_function\n)\ndf_final[attribute_from] = df_pom[grouping_function.upper()]\n\ndf_final.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I hope this short tutorial can show you how to effectivile change the window length of the data set!","metadata":{}}]}