{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-22T14:25:09.929675Z","iopub.execute_input":"2022-02-22T14:25:09.930005Z","iopub.status.idle":"2022-02-22T14:25:09.962736Z","shell.execute_reply.started":"2022-02-22T14:25:09.929915Z","shell.execute_reply":"2022-02-22T14:25:09.961900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this competition, we will be forecasting return on 14 popular crypto currencies. We have train.csv data as always which is the main data. Since it has around 3 GBs of data, it is always good to start with small data set. We can either use supplemental_train.csv or we can generate our own sample data also.","metadata":{}},{"cell_type":"code","source":"import subprocess\n\nproc       = subprocess.Popen([\"wc -l /kaggle/input/g-research-crypto-forecasting/train.csv\"], stdout=subprocess.PIPE, shell=True)\n(out, err) = proc.communicate()\nnRecords   = int(out.strip().split()[0])\nprint(\"Records in Train Data: {}\".format(nRecords))","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:25:09.965150Z","iopub.execute_input":"2022-02-22T14:25:09.966018Z","iopub.status.idle":"2022-02-22T14:25:35.099011Z","shell.execute_reply.started":"2022-02-22T14:25:09.965970Z","shell.execute_reply":"2022-02-22T14:25:35.097883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we know number of records in train data, we can also generate a random sample from the data instead of reading the whole file.","metadata":{}},{"cell_type":"code","source":"import random\nimport pandas as pd \n\nskiprows = random.sample(range(0, 24236807), 20000000)\nskiprows.remove(0)\ntrain_data_sub = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/train.csv', skiprows = skiprows)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:25:35.100911Z","iopub.execute_input":"2022-02-22T14:25:35.101260Z","iopub.status.idle":"2022-02-22T14:26:40.902032Z","shell.execute_reply.started":"2022-02-22T14:25:35.101211Z","shell.execute_reply":"2022-02-22T14:26:40.901114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Size of supplemental Data Created Manually : {}'.format(train_data_sub.shape[0]))\nprint('No of Features in Train Data : {}'.format(train_data_sub.shape[1]))","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:40.904082Z","iopub.execute_input":"2022-02-22T14:26:40.904359Z","iopub.status.idle":"2022-02-22T14:26:40.911467Z","shell.execute_reply.started":"2022-02-22T14:26:40.904322Z","shell.execute_reply":"2022-02-22T14:26:40.910443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have created a sunset of train data which has around 4.2 M records(almost of double of the sample data provided here). Let's use this and explore the data.","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.width', 130) # Helps us visualize more features in a singlew row\npd.set_option('display.float_format', lambda x: '%.3f' % x) # Print values only upto 3 Decimal Digits","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:40.912651Z","iopub.execute_input":"2022-02-22T14:26:40.912876Z","iopub.status.idle":"2022-02-22T14:26:40.921906Z","shell.execute_reply.started":"2022-02-22T14:26:40.912849Z","shell.execute_reply":"2022-02-22T14:26:40.921078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Basic Stats** : Focus on what is already available with pandas","metadata":{}},{"cell_type":"markdown","source":"> Top 5 Records","metadata":{}},{"cell_type":"code","source":"train_data_sub.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:40.923236Z","iopub.execute_input":"2022-02-22T14:26:40.923688Z","iopub.status.idle":"2022-02-22T14:26:40.952239Z","shell.execute_reply.started":"2022-02-22T14:26:40.923657Z","shell.execute_reply":"2022-02-22T14:26:40.951394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Features Definition:\n* timestamp - A timestamp for the minute covered by the row.\n* Asset_ID - An ID code for the cryptoasset.\n* Count - The number of trades that took place this minute.\n* Open - The USD price at the beginning of the minute.\n* High - The highest USD price during the minute.\n* Low - The lowest USD price during the minute.\n* Close - The USD price at the end of the minute.\n* Volume - The number of cryptoasset units traded during the minute.\n* VWAP - The volume weighted average price for the minute.\n* Target - 15 minute residualized returns.","metadata":{}},{"cell_type":"markdown","source":"> Bottom 5 Records","metadata":{}},{"cell_type":"code","source":"train_data_sub.tail()","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:40.953265Z","iopub.execute_input":"2022-02-22T14:26:40.953483Z","iopub.status.idle":"2022-02-22T14:26:40.965590Z","shell.execute_reply.started":"2022-02-22T14:26:40.953456Z","shell.execute_reply":"2022-02-22T14:26:40.964675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Use combination of head and tail to observe some in B/W observation","metadata":{}},{"cell_type":"code","source":"print('Check 5 records from index 1000 to 1004 \\n')\nprint(train_data_sub.head(1005).tail(5))","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:40.966787Z","iopub.execute_input":"2022-02-22T14:26:40.967014Z","iopub.status.idle":"2022-02-22T14:26:40.983402Z","shell.execute_reply.started":"2022-02-22T14:26:40.966987Z","shell.execute_reply":"2022-02-22T14:26:40.982522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We see that the view when using print is different from the default view.**","metadata":{}},{"cell_type":"markdown","source":"> Use sample method in pandas to observe random obesrvation\n> This is very useful when we are visualizing data after some data analysis","metadata":{}},{"cell_type":"code","source":"print(train_data_sub.sample(5))","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:40.986586Z","iopub.execute_input":"2022-02-22T14:26:40.987148Z","iopub.status.idle":"2022-02-22T14:26:41.197121Z","shell.execute_reply.started":"2022-02-22T14:26:40.987099Z","shell.execute_reply":"2022-02-22T14:26:41.196372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Data Info","metadata":{}},{"cell_type":"code","source":"train_data_sub.info()","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:41.198286Z","iopub.execute_input":"2022-02-22T14:26:41.199039Z","iopub.status.idle":"2022-02-22T14:26:41.219205Z","shell.execute_reply.started":"2022-02-22T14:26:41.199005Z","shell.execute_reply":"2022-02-22T14:26:41.218263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that timestamp and Asset_ID has been read as int64. Asset_ID should be read as a string var.","metadata":{}},{"cell_type":"markdown","source":"**Data Describe**","metadata":{}},{"cell_type":"code","source":"percentile_list = [0.01, 0.05]\npercentile_list.extend([i/10 for i in range(1, 10, 1)]) # Notice that we used extend instead of append \npercentile_list.extend([0.95, 0.99])\n\nprint(train_data_sub.describe(percentiles = percentile_list, include = 'all').T)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:41.220535Z","iopub.execute_input":"2022-02-22T14:26:41.220790Z","iopub.status.idle":"2022-02-22T14:26:43.172450Z","shell.execute_reply.started":"2022-02-22T14:26:41.220753Z","shell.execute_reply":"2022-02-22T14:26:43.171348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Target      : Has some missing values(131K) - Observe the count to find this\n- Count       : 286 trades take place per minute(Max : 165K - 47X of 99th Percentile Value)\n- Open        : 50% of the times, trades open below $15(65K Max Traded Value in a Minute)\n- High, Low, and Close  : Almost same distribution as of open price in a minute(Not much variation in price at minute level)\n- Volume      : 289K average volume per minute. 50% of the time per minute trade volume is less than 1300. Highest volume in a minute is 596M(100X of volume at 99th percentile)\n- VWAP        : Volume weighted average price is 1431.(Max : 65K)\n- Target      : 99% of the time, residualized return is less than 0.02. 50% of the time, the return is in -ve","metadata":{}},{"cell_type":"markdown","source":"> Unique Values in each feature","metadata":{}},{"cell_type":"code","source":"print(train_data_sub.nunique())","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:43.174146Z","iopub.execute_input":"2022-02-22T14:26:43.174470Z","iopub.status.idle":"2022-02-22T14:26:47.798436Z","shell.execute_reply.started":"2022-02-22T14:26:43.174427Z","shell.execute_reply":"2022-02-22T14:26:47.797570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Null Values \\n')\nprint(train_data_sub.isnull().sum())","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:47.799483Z","iopub.execute_input":"2022-02-22T14:26:47.799691Z","iopub.status.idle":"2022-02-22T14:26:47.881794Z","shell.execute_reply.started":"2022-02-22T14:26:47.799665Z","shell.execute_reply":"2022-02-22T14:26:47.880855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Data Analysis**","metadata":{}},{"cell_type":"markdown","source":"We have already observed that Open, Low, High, and Close distributions are almost identical. Let's validate this by using **correlation plot** B/W these features. ","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ncorr = train_data_sub[['Open', 'Low', 'High', 'Close']].corr()\ncorr.style.background_gradient(cmap='coolwarm')","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:47.882961Z","iopub.execute_input":"2022-02-22T14:26:47.883176Z","iopub.status.idle":"2022-02-22T14:26:48.320871Z","shell.execute_reply.started":"2022-02-22T14:26:47.883149Z","shell.execute_reply":"2022-02-22T14:26:48.319990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axs = plt.subplots(2,3, figsize = (14, 8))\nfig.suptitle('Correlation Plots')\n\naxs[0,0].plot(train_data_sub['Open'], train_data_sub['Low'])\naxs[0,0].set_title('Open vs Low')\n\naxs[0,1].plot(train_data_sub['Open'], train_data_sub['High'])\naxs[0,1].set_title('Open vs High')\n\naxs[0,2].plot(train_data_sub['Open'], train_data_sub['Close'])\naxs[0,2].set_title('Open vs Close')\n\naxs[1,0].plot(train_data_sub['Low'], train_data_sub['High'])\naxs[1,0].set_title('Low vs High')\n\naxs[1,1].plot(train_data_sub['Low'], train_data_sub['Close'])\naxs[1,1].set_title('Low vs Close')\n\naxs[1,2].set_title('High vs Close')\naxs[1,2].plot(train_data_sub['High'], train_data_sub['Close'])","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:26:48.322298Z","iopub.execute_input":"2022-02-22T14:26:48.322686Z","iopub.status.idle":"2022-02-22T14:27:20.023102Z","shell.execute_reply.started":"2022-02-22T14:26:48.322642Z","shell.execute_reply":"2022-02-22T14:27:20.022463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is obvious that any one of the feature has sufficient information and can cover the nature of other three for all purposes.","metadata":{}},{"cell_type":"markdown","source":"**Box Plot for Numeric Features**\n\nThough we have already looked at the precentile distribution of each feature, another way of achieving the same thing is using Box Plots, and ofcourse it is a visual represenation so much easy to understand.","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nsns.set_style(\"darkgrid\")\n\nfig, axs = plt.subplots(2, 2, figsize= (16, 10))\n\nsns.boxplot(x = train_data_sub['Count'], ax = axs[0,0])\nsns.boxplot(x = train_data_sub['Close'], ax = axs[0,1])\nsns.boxplot(x = train_data_sub['Volume'],ax = axs[1,0])\nsns.boxplot(x = train_data_sub['VWAP'],  ax = axs[1,1])","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:27:20.024062Z","iopub.execute_input":"2022-02-22T14:27:20.024781Z","iopub.status.idle":"2022-02-22T14:27:31.863584Z","shell.execute_reply.started":"2022-02-22T14:27:20.024732Z","shell.execute_reply":"2022-02-22T14:27:31.862694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is a lot of values beyond upper whisker(Q1 + 1.5 * IQR) : **Outliers**\n\nIQR : Q3 - Q1","metadata":{}},{"cell_type":"markdown","source":"**Exploratory Data Analysis Using Existing Packages**\n* What we have done so far, is to analyze the data through existing methods in pandas\n* Now we will use existing packages \n> * My personal favourite is dataprep\n> Documentation : https://dataprep.ai/\n* Other packages : AutoViz, PandasProfiling, Lux","metadata":{}},{"cell_type":"code","source":"!pip install dataprep","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:27:31.864872Z","iopub.execute_input":"2022-02-22T14:27:31.865126Z","iopub.status.idle":"2022-02-22T14:27:58.198440Z","shell.execute_reply.started":"2022-02-22T14:27:31.865097Z","shell.execute_reply":"2022-02-22T14:27:58.197295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from dataprep.eda import plot, plot_correlation, create_report, plot_missing","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:27:58.200141Z","iopub.execute_input":"2022-02-22T14:27:58.200563Z","iopub.status.idle":"2022-02-22T14:28:03.417477Z","shell.execute_reply.started":"2022-02-22T14:27:58.200521Z","shell.execute_reply":"2022-02-22T14:28:03.416650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(train_data_sub)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:28:03.418634Z","iopub.execute_input":"2022-02-22T14:28:03.418872Z","iopub.status.idle":"2022-02-22T14:28:30.305902Z","shell.execute_reply.started":"2022-02-22T14:28:03.418842Z","shell.execute_reply":"2022-02-22T14:28:30.304617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* While analyzing the data, we missed the distribution similarity of VWAP with Low, High, Open, and Close  \n* Almost all features are skewed\n* Asset_ID 1, 2, 5, 6, 7, 9 have same volume","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\ncreate_report(train_data_sub)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:28:30.307856Z","iopub.execute_input":"2022-02-22T14:28:30.308216Z","iopub.status.idle":"2022-02-22T14:30:52.889114Z","shell.execute_reply.started":"2022-02-22T14:28:30.308166Z","shell.execute_reply":"2022-02-22T14:30:52.887307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(train_data_sub, 'Volume')","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:30:52.891412Z","iopub.execute_input":"2022-02-22T14:30:52.892047Z","iopub.status.idle":"2022-02-22T14:31:01.587732Z","shell.execute_reply.started":"2022-02-22T14:30:52.891976Z","shell.execute_reply":"2022-02-22T14:31:01.586956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Assets Details Data**","metadata":{}},{"cell_type":"code","source":"asset_details_data = pd.read_csv('/kaggle/input/g-research-crypto-forecasting/asset_details.csv')","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:01.588849Z","iopub.execute_input":"2022-02-22T14:31:01.589931Z","iopub.status.idle":"2022-02-22T14:31:01.604269Z","shell.execute_reply.started":"2022-02-22T14:31:01.589879Z","shell.execute_reply":"2022-02-22T14:31:01.603493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"asset_details_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:01.605530Z","iopub.execute_input":"2022-02-22T14:31:01.605961Z","iopub.status.idle":"2022-02-22T14:31:01.617407Z","shell.execute_reply.started":"2022-02-22T14:31:01.605924Z","shell.execute_reply":"2022-02-22T14:31:01.616528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"create_report(asset_details_data)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:01.622425Z","iopub.execute_input":"2022-02-22T14:31:01.622709Z","iopub.status.idle":"2022-02-22T14:31:04.339669Z","shell.execute_reply.started":"2022-02-22T14:31:01.622679Z","shell.execute_reply":"2022-02-22T14:31:04.338636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Add Asset Name to Feature Data**","metadata":{}},{"cell_type":"code","source":"train_data_sub = train_data_sub.merge(asset_details_data, on = 'Asset_ID')","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:04.354217Z","iopub.execute_input":"2022-02-22T14:31:04.354473Z","iopub.status.idle":"2022-02-22T14:31:05.150171Z","shell.execute_reply.started":"2022-02-22T14:31:04.354440Z","shell.execute_reply":"2022-02-22T14:31:05.149225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_sub.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:05.151398Z","iopub.execute_input":"2022-02-22T14:31:05.151651Z","iopub.status.idle":"2022-02-22T14:31:05.168421Z","shell.execute_reply.started":"2022-02-22T14:31:05.151620Z","shell.execute_reply":"2022-02-22T14:31:05.167576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Distribution of weight by asset name","metadata":{}},{"cell_type":"code","source":"def plot_barh(df, xvar, yvar):\n    fig = plt.figure(figsize = (10, 5))\n    plt.xticks(rotation='vertical')\n    plt.bar(df[xvar], df[yvar])","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:05.169437Z","iopub.execute_input":"2022-02-22T14:31:05.170321Z","iopub.status.idle":"2022-02-22T14:31:05.181608Z","shell.execute_reply.started":"2022-02-22T14:31:05.170271Z","shell.execute_reply":"2022-02-22T14:31:05.180844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(asset_details_data.sort_values('Weight', ascending = False))\nplot_barh(asset_details_data, 'Asset_Name', 'Weight')","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:05.183245Z","iopub.execute_input":"2022-02-22T14:31:05.184120Z","iopub.status.idle":"2022-02-22T14:31:05.559032Z","shell.execute_reply.started":"2022-02-22T14:31:05.184068Z","shell.execute_reply":"2022-02-22T14:31:05.558266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Top 5 currencies by Weight:**\n* Bitcoin\n* Ethereum\n* Cardano\n* Binance Coin\n* Dogecoin\n","metadata":{}},{"cell_type":"code","source":"train_data_sub['Asset_Name'].value_counts(normalize = True)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:05.560216Z","iopub.execute_input":"2022-02-22T14:31:05.561067Z","iopub.status.idle":"2022-02-22T14:31:06.055788Z","shell.execute_reply.started":"2022-02-22T14:31:05.561029Z","shell.execute_reply":"2022-02-22T14:31:06.054974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Top 5 currency by # recods:**\n* Ethereum\n* Litecoin\n* EOS.IO\n* Bitcoin\n* Bitcoin Cash\n\n> All the top 5 coins have 8.1% of volume","metadata":{}},{"cell_type":"markdown","source":"**Total Traded Volume of Each Cypto**\n\n> In the below code section, we will also see an example of operator chaining in pandas","metadata":{}},{"cell_type":"code","source":"volume_traded_by_asset = (train_data_sub.groupby('Asset_Name')['Volume']\n                                        .agg(['sum', 'mean'])\n                                        .reset_index()\n                                        .rename(columns = {'sum' : 'TradedVolume', 'mean' : 'AvgVolTraded'})\n                                        .sort_values('AvgVolTraded', ascending = False))\nprint(volume_traded_by_asset)\nplot_barh(volume_traded_by_asset, 'Asset_Name', 'TradedVolume')\nplot_barh(volume_traded_by_asset, 'Asset_Name', 'AvgVolTraded')","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:06.057227Z","iopub.execute_input":"2022-02-22T14:31:06.057462Z","iopub.status.idle":"2022-02-22T14:31:07.213397Z","shell.execute_reply.started":"2022-02-22T14:31:06.057433Z","shell.execute_reply":"2022-02-22T14:31:07.212471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"**Derived Features**\n\nWe can also create some hypothesis about the data and validate them by creating features. These features will help us get lift at modeling phase.\nFeatures:\n1. Units Sold per Minute - Volume/Count\n2. \n","metadata":{}},{"cell_type":"markdown","source":"**Units Sold per Minute**","metadata":{}},{"cell_type":"code","source":"train_data_sub['UnitsSoldPerMinute'] = train_data_sub['Volume']/train_data_sub['Count']","metadata":{"execution":{"iopub.status.busy":"2022-02-22T14:31:07.214518Z","iopub.execute_input":"2022-02-22T14:31:07.214751Z","iopub.status.idle":"2022-02-22T14:31:07.245473Z","shell.execute_reply.started":"2022-02-22T14:31:07.214723Z","shell.execute_reply":"2022-02-22T14:31:07.244557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"WIP....","metadata":{}}]}