{"cells":[{"metadata":{"_uuid":"d07f853a51596448ab827b9d39279a5876e900ed"},"cell_type":"markdown","source":"# Google Analytics Customer Revenue Prediction\n---\n\nStan Wang (16446054), Dale Euinton (14026002), and Hang Zhao (18045261)\n\n## Abstract\nWe present a kernel with an iterative workflow to participate a Kaggle competition. We first do exploratory data analysis on raw data to get a big picture of data set. Then we make the first prediction with **LightGBM** as the baseline of this competition. The **LightGBM** will generate the _feature importance_ at meanwhile. After that, we will enter the second round of EDA and feature engineering base on the feature importance. After address the more important feature we fit it to the **LightGBM** again. This project evaluates the performance of this work-flow and hopes it can contribute some knowledge to the community especially to those who participate in Kaggle competition the first time."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from __future__ import division\nimport numpy as np\nimport pandas as pd \nimport os\nimport timeit\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5770f2af736df0eb707567f9cd2b2a9c91876410","_kg_hide-input":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nfrom plotly.offline import init_notebook_mode, iplot\nimport plotly.graph_objs as go\nfrom plotly import tools\n# link pandas and plotly\n# import cufflinks as cf\n\ninit_notebook_mode(connected=True)\n# cf.set_config_file(offline=True, world_readable=True, theme='ggplot')\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d2c72bd4ac553dd5c5b6147d7f14ffc1df0813b","_kg_hide-input":true},"cell_type":"code","source":"from matplotlib import rcParams\nrcParams['figure.figsize'] = 22, 14\nrcParams['axes.titlesize'] = 24\nrcParams['axes.labelsize'] = 20\nrcParams['xtick.labelsize'] = 16\nrcParams['ytick.labelsize'] = 16\nrcParams['legend.fontsize'] = 14","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60610c832822d1d127c04b9df2fd31c81c2feb48"},"cell_type":"markdown","source":"Functions that we will use, but hide in the beginning to increase the readability."},{"metadata":{"trusted":true,"_uuid":"63628dc1d047ac18ee64c6f5ecd54521d6c95e79","_kg_hide-input":true},"cell_type":"code","source":"def sub_boxenplots(x, data, y=\"totals.transactionRevenue_ln\", rot=15):\n    order = data[x].unique()\n    fig, axes = plt.subplots(ncols=2, nrows=1, squeeze=False, figsize=(22, 8))\n    sns.boxenplot(x=x, y=y, data=data, ax=axes[0, 0], order=order)\n    axes[0, 0].set_title(\"All Instances\")\n    axes[0, 0].set_xticklabels(axes[0, 0].get_xticklabels(), rotation=rot, ha='right')\n    sns.boxenplot(x=x, y=y, data=data.loc[data[y] > 0, :], ax=axes[0, 1], order=order)\n    axes[0, 1].set_title(\"Instances With Non-zero Revenue\")\n    axes[0, 1].set_xticklabels(axes[0, 1].get_xticklabels(), rotation=rot, ha='right')\n    fig.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a3c2ff75ba4620bb5415435db1756df6006933c","_kg_hide-input":true},"cell_type":"code","source":"prefix = \"../input/ga-customer-revenue-prediction/\"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3ecd73723669a998f2cf7258a767874e49743054"},"cell_type":"markdown","source":"Thanks to this awesome [kernel](https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue) that save the trouble for us to load the data in proper type."},{"metadata":{"trusted":true,"_uuid":"f674eb19ad7a186a9cda85496b33f736f257346c","_kg_hide-input":true},"cell_type":"code","source":"import json\nfrom pandas.io.json import json_normalize\ndef load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    read_start = timeit.default_timer()\n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows,\n                     parse_dates=['date']\n                    )\n    read_end = timeit.default_timer()\n    print(\"Finish reading {0}, time usage: {1}\".format(csv_path, read_end - read_start))\n    \n    process_start = timeit.default_timer()\n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    process_end = timeit.default_timer()\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}, time usage: \", process_end - process_start)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"58e6796396ffead7b6da1e38608136af5642964e"},"cell_type":"code","source":"# Just for convinence, uncomment before submite the codes\nX_train_df = load_df(prefix + \"train.csv\")\nX_test_df = load_df(prefix + \"test.csv\")\n\nprint(\"Done!\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0497da9000e5b2c397df63b52ad8eaa8bf783e8"},"cell_type":"markdown","source":"The data set needs to be pre-preocessed before it can be used."},{"metadata":{"trusted":true,"_uuid":"4cb600b7670c61e015703c9cf35d541e7001276a"},"cell_type":"code","source":"X_train_df.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6e67cc70fafbc6efc57db54f56bc33b9b8ce46b5","_kg_hide-input":true},"cell_type":"code","source":"submit_id = X_test_df[\"fullVisitorId\"].unique()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7beb0c0e17d31042b2ed4a2500dbbf47eab34fb2"},"cell_type":"markdown","source":"# EDA\n---\n## Target Visualisation\n\n**We are predicting the **natural log**  of the sum of all transactions **per user**. For every user in the test set, the target is:\n$$\ny_{user} = \\sum_{i=1}^{n}{transaction_{user_i}}\n$$\n$$\ntarget_{user} = \\ln(y_{user} + 1)\n$$\nGive a look at the target distribution will give us a big picture of what we are predicting. Before we proceed, we will generate the new target as the competition requested."},{"metadata":{"trusted":true,"_uuid":"a761a719961b6cd5c3d0286e7ca2ceca08475761"},"cell_type":"code","source":"X_train_df.loc[:, \"totals.transactionRevenue_ln\"] = np.log1p(X_train_df[\"totals.transactionRevenue\"].fillna(0).astype(\"float\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a40e70d8f857216b39756a3008ff100f327aaa0b"},"cell_type":"markdown","source":"> The 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue.\n\nLet's take a look at the ratio of zero and non-zero revenue instances."},{"metadata":{"trusted":true,"_uuid":"63b54c6addc7068d906ee96383ee374844cc36ca","_kg_hide-input":true},"cell_type":"code","source":"labels = ['Zero revenue instance', 'Non-zero revenue instance']\nvalues = [X_train_df.loc[X_train_df[\"totals.transactionRevenue_ln\"] == 0, \"totals.transactionRevenue_ln\"].count(),\n         X_train_df.loc[X_train_df[\"totals.transactionRevenue_ln\"] != 0, \"totals.transactionRevenue_ln\"].count()]\ntrace = go.Pie(labels=labels, values=values)\nlayout = dict(\n    title = 'Instance Revenue'\n)\n\nfig = dict(data=[trace], layout=layout)\niplot(fig, filename='basic_pie_chart')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f5ae217d60ceb224753b13597a31b643e695f7f"},"cell_type":"markdown","source":"Well, 80/20 seems too optimistic. Let's merge all the instances to unique customers, see if things get any better."},{"metadata":{"trusted":true,"_uuid":"1ddc182d565c35d2711a08580f931d27b4ee9cdf","_kg_hide-input":true},"cell_type":"code","source":"sum_target_series = X_train_df.groupby(\"fullVisitorId\")[\"totals.transactionRevenue_ln\"].sum()\n\nlabels = ['Zero revenue customer', 'Non-zero customer']\nvalues = [sum_target_series[sum_target_series == 0].count(),\n         sum_target_series[sum_target_series != 0].count()]\ntrace = go.Pie(labels=labels, values=values)\nlayout = dict(\n    title = 'Unique Customer Revenue'\n)\n\nfig = dict(data=[trace], layout=layout)\niplot(fig, filename='basic_pie_chart')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"75a9c15225c9148715eca3f35a8724ec6142067c"},"cell_type":"markdown","source":"The customer that will generate revenue is the objective of this project. However, since then we can infer that a pre-classified the customer who will not make any revenues will help the later regression task.\n\nLet's take a look at the boxplot of the customers that will generate revenues."},{"metadata":{"trusted":true,"_uuid":"ee1d1519eae422ab634ce4937715b1ae6af295c9","_kg_hide-input":true},"cell_type":"code","source":"trace = go.Box(y=X_train_df.loc[X_train_df[\"totals.transactionRevenue_ln\"] > 0, \"totals.transactionRevenue_ln\"],\n              name='Instance')\nlayout = dict(\n    title = 'Instance With Revenue Greater Than Zero'\n)\n\nfig = dict(data=[trace], layout=layout)\niplot(fig, filename='basic_pie_chart')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e9822d3c5951f3a3b809b7eb507d2bc105e31dd9"},"cell_type":"markdown","source":"Then let's take a look at the unique user revenues."},{"metadata":{"trusted":true,"_uuid":"16dc96b6a9d9146ea7d79f61de0d40bd66346dd2","_kg_hide-input":true},"cell_type":"code","source":"trace = go.Box(y=sum_target_series[sum_target_series > 0], name=\"Unique User\")\niplot([trace])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"753a1ef8498355ad5d018fe49508a222e0536b8a"},"cell_type":"markdown","source":"Even we filter all the zeros, the data set is still extremely **unbalanced,** we should be careful with the outliers."},{"metadata":{"_uuid":"25fb583896bd41d167ed7c1230fd25d29219233f"},"cell_type":"markdown","source":"# Data Preprocessing\n---\nThanks for the [SRK](https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue)'s kernel pointed out that there are constant value columns which are invalid in this competiton. Drop them before further processing"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"29b3fae4b2bd0a31194cf4b71a57ad3fcda7603a","_kg_hide-input":true},"cell_type":"code","source":"to_drop_cols = [c for c in X_train_df.columns if X_train_df[c].nunique(dropna=False)==1 ]\n\nX_train_df.drop(to_drop_cols, axis=1, inplace=True)\nX_test_df.drop(to_drop_cols, axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41eb573f4c2e73449df1c933e4691c1cf7708e60"},"cell_type":"markdown","source":"Let's check what is the difference between the testing set and training set in features"},{"metadata":{"trusted":true,"_uuid":"9d57ad7f52a0fac071e2f239ef4d6a8382bcd2ff","_kg_hide-input":true},"cell_type":"code","source":"print(\"Training set shape: \", X_train_df.shape, \" Testing set shape: \", X_test_df.shape)\nprint(\"Difference features between two sets:\")\nfor col in X_train_df.columns:\n    if col not in X_test_df:\n        print(col)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d19b9d455087ca3078248a18d68a3742829b6d7d"},"cell_type":"markdown","source":"Let's exmine the content in the differece feature `trafficSource.campaignCode`"},{"metadata":{"trusted":true,"_uuid":"f7ac5f599a20b2f43dc85a33b1c87a75ad15a160","_kg_hide-input":true,"scrolled":true},"cell_type":"code","source":"X_train_df[\"trafficSource.campaignCode\"].unique()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a4164a9c424f89c71ee681211ad7dff59b64a73"},"cell_type":"markdown","source":"It seems that we can drop this column in training set since it doesn't provide any sensible information.\n\nAnd next, we will concatenate the training set and testing set together, just for saving the redundant process that we need to perform on both datasets."},{"metadata":{"trusted":true,"_uuid":"63155794b6b0708353b5716625a15051e632b852","_kg_hide-input":true},"cell_type":"code","source":"X_train_df.drop(\"trafficSource.campaignCode\", axis=1, inplace=True)\n\nX_test_df = pd.concat([X_test_df, \n                       pd.Series(np.nan, name='totals.transactionRevenue'), \n                       pd.Series(np.nan, name='totals.transactionRevenue_ln')], \n                      axis=1)\n\nX_all_df = pd.concat([X_train_df, X_test_df], ignore_index=True, sort=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01b78b6ea1c182fe481021381eab97e14d5964ea"},"cell_type":"markdown","source":"Session ID is for internet session only, we don't need it. And the `visitId` we don't need it since the `fullVisitorId` has the information"},{"metadata":{"trusted":true,"_uuid":"f7e16b85556fcc8a33c78cdd518a1c1a0cd2fce5","_kg_hide-input":false,"scrolled":true},"cell_type":"code","source":"X_all_df.drop(['sessionId', 'visitId'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"adc45dba5a1813d3df754868a1937836e64f8c45"},"cell_type":"markdown","source":"### Cast The Datatype To The Right One\nWe want to make sure all the datatype is the right one and we want to change all the string object type to the **category** type as python has a horrible run-time to deal with string. However, we will keep the ID as the string since the **category**  will only help if there are categories."},{"metadata":{"trusted":true,"_uuid":"f627f9ebe9b20061114cfb66cb4995e65e51384d","_kg_hide-input":true},"cell_type":"code","source":"obj_cols = [column for column in X_all_df.columns if X_all_df[column].dtype == object]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54575f6409998dd84e769cee98d09224a805f40f","_kg_hide-input":true},"cell_type":"code","source":"for col in obj_cols:\n    try:\n        X_all_df.loc[:, col] = pd.to_numeric(X_all_df.loc[:, col], errors='raise')\n#         print(\"Successfully parse column: \", col)\n    except ValueError as e:\n        # mute the string column that cannot be converted to numeric values.\n        pass\n#         print(\"Unable to parse column: \", col)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"7cf0b7128e5973de488fcf52ecf35171fe93626c","_kg_hide-input":true},"cell_type":"code","source":"for col in X_all_df.select_dtypes(include='object'):\n    if col == \"fullVisitorId\":\n        continue\n    try:\n        X_all_df.loc[:, col] = pd.Categorical(X_all_df.loc[:, col])\n#         print(\"Successfully parse column: \", col)\n    except ValueError as e:\n        print(\"Unable to parse column: \", col)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"de3771e0e7851fd2cf4b1a792235c1c4e49d6d8d"},"cell_type":"markdown","source":"Lastly, change all the boolean features to right type"},{"metadata":{"trusted":true,"_uuid":"f5eb065da631abbf39ccd5a0a11c7acbe4503a08","_kg_hide-input":true},"cell_type":"code","source":"bool_cols = []\nfor col in X_all_df:\n    if X_all_df[col].nunique() == 2:\n        bool_cols.append(col)\nX_all_df.loc[:, bool_cols] = X_all_df.loc[:, bool_cols].astype('bool')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2311446c44c06b5365fa164e96ca98d588421dd5"},"cell_type":"markdown","source":"### Fill The NaN Values\n`NaN` values are a familiar enemy for all data scientists. Before we drop or fill them up, we need to investigate the features with `NaN` values."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"15bdf92c0f348c1665c4b17df791c1b862f4eeae"},"cell_type":"code","source":"def investigate(col_name, data):\n    print(\"The number of unique category\")\n    print(col_name, \": \", data[col_name].nunique())\n    print(\"*\" * 20)\n    print(\"Value count\")\n    print(data[col_name].value_counts())","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d1735d088c2a70f9616b3fbcce10b23751af3686"},"cell_type":"code","source":"def ratio_nan(data):\n    num_data = X_all_df.shape[0]\n    null_sum = data.isnull().sum()\n    null_val_features = null_sum[null_sum > 0]\n    if \"totals.transactionRevenue_ln\" in null_val_features:\n        null_val_features.drop([\"totals.transactionRevenue\", \"totals.transactionRevenue_ln\"], inplace=True)\n    print(null_val_features/ num_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a2d10cf99b4a64733018306fb7d0218f173bf71"},"cell_type":"code","source":"print(\"The features with NaN value percentage: \")\nratio_nan(X_all_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2aaffc5f0a4d8717f8288d53e0f1ef87547984e2"},"cell_type":"markdown","source":">A bounce occurs when a web site visitor only views a single page on a website, that is, the visitor leaves a site without visiting any other pages before a specified session-timeout occurs.\n\nFrom here we can infer that the instance who bounce away will never generate any revenue. Let's explore it."},{"metadata":{"trusted":true,"_uuid":"136c28b7803c05e85ab2d5610708fe10db500cff","_kg_hide-input":true},"cell_type":"code","source":"X_all_df.loc[:, \"totals.bounces\"].fillna(0, inplace=True)\nX_all_df.loc[:, \"totals.bounces\"] = X_all_df.loc[:, \"totals.bounces\"].astype('bool')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"097501d2f087f6589e62acc7a4f01e5b407d47b6"},"cell_type":"markdown","source":"Let's take a look at the revenues which are corresponding to the bounce feature."},{"metadata":{"trusted":true,"_uuid":"c506338a7fca20648ee9a6c93ac60e84e32611d5"},"cell_type":"code","source":"sub_boxenplots(data=X_all_df.loc[X_all_df[\"totals.transactionRevenue_ln\"].notnull(), :], x=\"totals.bounces\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a0aeb609325bd91a50d9e5d871ca61355af22e11"},"cell_type":"markdown","source":"The assumption has been proven that if an instance bounces away, there is no way that he or she can go to the checkout page to pay the money. This is an excellent feature that can help us quickly **shrink** the size of our data. We will drop all the rows with the true bounce and confidentially infer 0 revenue to the testing set with true bounce."},{"metadata":{"trusted":true,"_uuid":"aa88d198e9b3a9b967e5d2104c7d6956e2a1676d","_kg_hide-input":true},"cell_type":"code","source":"X_all_df.drop(X_all_df.loc[X_all_df[\"totals.bounces\"], \"fullVisitorId\"].index, axis=0, inplace=True)\n\n# We don't need this feature anymore\nX_all_df.drop(\"totals.bounces\", axis=1, inplace=True)\n\nX_all_df.reset_index(drop=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c1e67477f7abf2c1fac7cf189886da6da617ea90"},"cell_type":"markdown","source":"We will skip some sample features that we will correct them silently."},{"metadata":{"trusted":true,"_uuid":"4a65b0a0eb73a8fa84d2bd8f67849a887d0f2044","_kg_hide-input":true},"cell_type":"code","source":"X_all_df.loc[:, \"totals.newVisits\"].fillna(0, inplace=True)\nX_all_df.loc[:, \"totals.newVisits\"] = X_all_df.loc[:, \"totals.newVisits\"].astype('bool')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"218f97f05bea4a46617867a27bd59f83ea1e170b"},"cell_type":"markdown","source":"> A pageview (or pageview hit, page tracking hit) is an instance of a page being loaded (or reloaded) in a browser. Pageviews is a metric defined as the total number of pages viewed."},{"metadata":{"trusted":true,"_uuid":"613f93de5c0c4ce279fe4db060456e3390eee471","_kg_hide-input":true},"cell_type":"code","source":"X_all_df.loc[:, \"totals.pageviews\"].fillna(0, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c6146a21d2a792468ccc8f8c99c3d254753b085"},"cell_type":"markdown","source":"Google traffic source ads contents are highly missing.\n<table>\n    <tbody>\n         <tr>\n          <td><code>trafficSource.adContent</code></td>\n          <td>STRING</td>\n          <td>The ad content of the traffic source. Can be set by the utm_content URL parameter.</td>\n        </tr>\n        <tr>\n          <td><code>trafficSource.<br>\n          adwordsClickInfo.adNetworkType</code></td>\n          <td>STRING</td>\n          <td>Network Type. Takes one of the following values: {“Google Search\", \"Content\", \"Search partners\", \"Ad Exchange\", \"Yahoo Japan Search\", \"Yahoo Japan AFS\", “unknown”}</td>\n        </tr>\n        <tr>\n              <td><code>trafficSource.<br>\n              adwordsClickInfo.gclId</code></td>\n              <td>STRING</td>\n              <td>The Google Click ID.</td>\n            </tr>\n            <tr>\n              <td><code>trafficSource.<br>\n              adwordsClickInfo.isVideoAd</code></td>\n              <td>BOOLEAN</td>\n              <td>True if it is a Trueview video ad.</td>\n            </tr>\n            <tr>\n              <td><code>trafficSource.<br>\n              adwordsClickInfo.page</code></td>\n              <td>INTEGER</td>\n              <td>Page number in search results where the ad was shown.</td>\n            </tr>\n            <tr>\n              <td><code>trafficSource.<br>\n              adwordsClickInfo.slot</code></td>\n              <td>STRING</td>\n              <td>Position of the Ad. Takes one of the following values:{“RHS\", \"Top\"}</td>\n            </tr>\n            <tr>\n              <td><code>trafficSource.isTrueDirect</code></td>\n              <td>BOOLEAN</td>\n              <td>True if the source of the session was Direct (meaning the user typed the name of your website URL into the browser or came to your site via a bookmark), This field will also be true if 2 successive but distinct sessions have exactly the same campaign details. Otherwise NULL.</td>\n            </tr>\n            <tr>\n              <td><code>trafficSource.keyword</code></td>\n              <td>STRING</td>\n              <td>The keyword of the traffic source, usually set when the trafficSource.medium is \"organic\" or \"cpc\". Can be set by the utm_term URL parameter.</td>\n            </tr>\n            <tr>\n              <td><code>trafficSource.referralPath</code></td>\n              <td>STRING</td>\n              <td>If trafficSource.medium is \"referral\", then this is set to the path of the referrer. (The host name of the referrer is in trafficSource.source.)</td>\n            </tr>\n    </tbody>\n</table>"},{"metadata":{"_uuid":"1ec926fb3298efefc3a2394c73ff08d529666350"},"cell_type":"markdown","source":"From the description, we can infer that `NULL` values represent that the user doesn't click the ads?  Let's investigate if the rest features with `NaN` values will affect the revenues they generated."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"86fc0af8daf9906b834b8d7f2b31fec7bad6482c"},"cell_type":"code","source":"def plot_cmp_stack(data, col_name):\n    \n    total_null_num = data[col_name].isnull().sum()\n    total_not_null_num = data[col_name].notnull().sum()\n    \n    null_zero_num =  (data.loc[data[col_name].isnull(), \"totals.transactionRevenue_ln\"] == 0).sum()\n    null_non_zero_num = (data.loc[data[col_name].isnull(), \"totals.transactionRevenue_ln\"] != 0).sum()\n    not_null_zero_num = (data.loc[data[col_name].notnull(), \"totals.transactionRevenue_ln\"] == 0).sum()\n    not_null_non_zero_num = (data.loc[data[col_name].notnull(), \"totals.transactionRevenue_ln\"] != 0).sum()\n    \n    trace1 = go.Bar(\n        x=['NULL', 'Not NULL'],\n        y=[null_zero_num/ total_null_num, not_null_zero_num/ total_not_null_num],\n        name='Instances With Zero Revenue',\n        marker=dict(\n            color='rgb(158,202,225)',\n            line=dict(\n                color='rgb(8,48,107)',\n                width=1.5),\n        ),\n        opacity=0.6\n    )\n    \n    trace2 = go.Bar(\n        x=['NULL', 'Not NULL'],\n        y=[null_non_zero_num/ total_null_num, not_null_non_zero_num/ total_not_null_num],\n        name='Instances With Non-Zero Revenue',\n        marker=dict(\n            color='rgb(58,200,225)',\n            line=dict(\n                color='rgb(8,48,107)',\n                width=1.5),\n            ),\n        opacity=0.6        \n    )\n    \n    trace3 = go.Pie(\n        labels = ['Ratio Of Non-zero Revenue With This Feature Is Null', 'Ratio Of Non-zero Revenue With This Feature Is Not NULL'],\n        values = [null_non_zero_num, not_null_non_zero_num],\n        domain = {\"x\": [0.5, 1]},\n        hole = .4,\n        name = 'Instances With Non-zero Revenue',\n        text = ['Non-zero Ratio']\n    )\n    \n    data=[trace1, trace2, trace3]\n    \n    layout=go.Layout(\n        barmode='stack',\n        title = col_name + \": Revenue Comparison Between NULL and Not NULL Value\",\n        yaxis = {'title': 'Percentage'},\n        xaxis = {\n            'domain': [0, 0.5]\n        },\n        showlegend=False\n    )\n    \n    fig = go.Figure(data=data, layout=layout)\n    iplot(fig, filename=col_name + '_stack_bar')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"38eea165fcc3ea5ff773ae369514b887c2194916"},"cell_type":"code","source":"plot_cmp_stack(data=X_train_df, col_name=\"trafficSource.adwordsClickInfo.gclId\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4554dcd5a2dcdc5ce777f47560098f4bfbcb63bf"},"cell_type":"markdown","source":"From the bar plot above, our previous infers failed. In the left pie chart we can tell that this even the user click the ads, it still occupies a small portion of who make revenues."},{"metadata":{"trusted":true,"_uuid":"7960c22abb332ee2686d311fad2efd85e169da7f"},"cell_type":"code","source":"plot_cmp_stack(data=X_train_df, col_name=\"trafficSource.isTrueDirect\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22b379a0ee5eb0fc3a0e7bfeaffa44576e5eee16"},"cell_type":"code","source":"plot_cmp_stack(data=X_train_df, col_name=\"trafficSource.keyword\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"622e4b6e1fb45ff05d6fcb1cfff045f10d9e33f0"},"cell_type":"code","source":"plot_cmp_stack(data=X_train_df, col_name=\"trafficSource.referralPath\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d188808d96a6f5ef2bef4a0d0aec4923cd0cab2"},"cell_type":"markdown","source":"Above three features have a significant share in the instances who contribute revenue. Especially `trafficSource.referralPath`,  almost half of the revenue instances, their `trafficSource.refferalPath` is set."},{"metadata":{"trusted":true,"_uuid":"e9110448dc744b22bdc2bfc802945118c42e4616","_kg_hide-input":true},"cell_type":"code","source":"to_drop_cols = [\"trafficSource.adContent\", \"trafficSource.adwordsClickInfo.adNetworkType\", \n                \"trafficSource.adwordsClickInfo.gclId\", \"trafficSource.adwordsClickInfo.isVideoAd\",\n                \"trafficSource.adwordsClickInfo.page\", \"trafficSource.adwordsClickInfo.slot\"]\n\nX_all_df.drop(to_drop_cols, axis=1, inplace=True)\n\nX_all_df.loc[:, \"trafficSource.isTrueDirect\"] = X_all_df.loc[:, \"trafficSource.isTrueDirect\"].fillna(0).astype('bool')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"5c04c69854dad54f9ad33667496bc057b079b66a"},"cell_type":"code","source":"ratio_nan(X_all_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1214c7f72b30c8e228242c0471aefda1e31a9cda"},"cell_type":"markdown","source":"## Investigation in Data Integrity\n---\nNot all the datasets are 100% accurate. Sometimes, the inaccurate value will hurt the models by giving misleading information. As a human, a programmer will have his/her bias about datasets as well. As a result, we need to check the integrity of the datasets before the machine learning taking place."},{"metadata":{"_uuid":"f337cb8520920832b6f49201ca9493818d7a167a"},"cell_type":"markdown","source":"**Note that the visit number of an  instance is not neccessarily start from 1**"},{"metadata":{"trusted":true,"_uuid":"250272113f3fd93a1316b92eb0761a54c1f2e67b"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"fullVisitorId\"] == \"0824839726118485274\", \"visitNumber\"].sort_values().head(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0e70f69bee0bcb5a6390a19ee4db1508d76ab7af"},"cell_type":"markdown","source":"Investigate more with the unique minium number of instances"},{"metadata":{"trusted":true,"_uuid":"d1e70250fb42b412e14bc114abfb0a9e91e2a0f4"},"cell_type":"code","source":"X_all_df.groupby(\"fullVisitorId\")[\"visitNumber\"].min().unique()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2086e2c302578a223c2da04fd8bc151c292d35e4"},"cell_type":"markdown","source":"Assume `channelGrouping` is` Direct`, its `trafficSource.isTrueDirect` should be **1**"},{"metadata":{"trusted":true,"_uuid":"4ef26c75debf8810bdac5d1dd7b2c0d782357df2"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"channelGrouping\"] == \"Direct\", \"trafficSource.isTrueDirect\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6270ce1a6dd14baf50ab8bd95f288ce601ff1782"},"cell_type":"markdown","source":"Let's do opposite"},{"metadata":{"trusted":true,"_uuid":"7f0c4ca50f17828b7ea4974493d6ceb1222205b9"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"trafficSource.isTrueDirect\"] == 1, \"channelGrouping\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"467cfdefc855c45959f393da91d80491847990f2"},"cell_type":"markdown","source":"Looks like there are wrong values. Let's see if `trafficSource.isTrueDirect` is false."},{"metadata":{"trusted":true,"_uuid":"03b4b9e67fc2c35ff734426dd82fb11d99ee1ff1"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"trafficSource.isTrueDirect\"] == 0, \"channelGrouping\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73b23b2da16aecb8ccc6621eaa3f6989bd637ae1"},"cell_type":"markdown","source":"Here is the description from **Google**\n<table>\n  <tbody>\n    <tr>\n      <td><code>trafficSource.isTrueDirect</code></td>\n      <td>BOOLEAN</td>\n      <td>True if the source of the session was Direct (meaning the user typed the name of your website URL into the browser or came to your site via a bookmark), This field will also be true if 2 successive but distinct sessions have exactly the same campaign details. Otherwise NULL.</td>\n    </tr>\n  </tbody>\n</table>"},{"metadata":{"trusted":true,"_uuid":"93e29532583afc068363246077272021c50344a4"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"trafficSource.isTrueDirect\"] == 1, \"trafficSource.medium\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c91410b854241b236588690e2db2e82abc2952e9"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"trafficSource.isTrueDirect\"] == 0, \"trafficSource.medium\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3f476dfb0b7379541fe43e8842252ed845ba3248"},"cell_type":"markdown","source":"**cost per click (CPC)** is an internet advertising model used to direct traffic to websites, in which an advertiser pays a publisher (typically a website owner or a network of websites) when the ad is clicked.\n\n**Cost-per-thousand impressions (CPM)**: A way to bid where you pay per one thousand views (impressions) on the Google Display Network.\n\n---\nHere we can see there are couple of problems:\n\n* referral doesn't match.\n* (none) value in the trafficSource.medium actually mean direct.\n* trafficSource.isTrueDirect seems have wrong values.\n"},{"metadata":{"trusted":true,"_uuid":"7565df9e253af87377770e9285580b5a4fb8a9d5"},"cell_type":"code","source":"temp_df = X_all_df.loc[X_all_df[\"trafficSource.isTrueDirect\"] == 1, :]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1da9e6e49c45a64dd61b3f24774bd88f895b2087"},"cell_type":"code","source":"temp_df.loc[ temp_df[\"trafficSource.medium\"] == 'referral', 'channelGrouping'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68354fa0b91906a214b3b18fa06fa0776ab90074"},"cell_type":"markdown","source":"Conclusion: `trafficSource.isTrueDirect` have a lot wrong values and `trafficSource.medium` is highly redundant to `channelGrouping`."},{"metadata":{"_uuid":"69057ceec3754f1710c8aecadb3d24ca5d4e86fe"},"cell_type":"markdown","source":"Assume, when keyword is `NULL` means that instance is not using searching engine."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"dd83f5312777fdc2b84df2abb8641a11c45747d8"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"trafficSource.keyword\"].notnull(), \"channelGrouping\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e789ce371b0954d1afe2c57ac857330e6441f0e"},"cell_type":"markdown","source":"It seems the most of it are right, let's see the opposite"},{"metadata":{"trusted":true,"_uuid":"985df20a1d8127691426ffe2e2541d771ffeec07"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"trafficSource.keyword\"].isnull(), \"channelGrouping\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b580ab22930673522f9402960cdf4d3cb62a7e4"},"cell_type":"markdown","source":"There are `Organic Search` **6026** and `Paid Search` **147**  which should have a keyword with it. Let's see if they are mislabeled."},{"metadata":{"trusted":true,"_uuid":"758d04bd4f849aa84a641bb1bab1928b08ba22ff"},"cell_type":"code","source":"temp_df = X_all_df.loc[X_all_df[\"trafficSource.keyword\"].isnull(), :]\ntemp_df.loc[(temp_df[\"channelGrouping\"] == \"Organic Search\") | (temp_df[\"channelGrouping\"] == \"Paid Search\"), \"trafficSource.medium\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c24ccddf2ed3a14d38f38f5760a5bcf2f90adbc1"},"cell_type":"markdown","source":"If we check the value in the `trafficSource.keyword` there is a value called **(not provided)** above should be set to it"},{"metadata":{"trusted":true,"_uuid":"820fbd9a4c1b01255c613e63fba1915b38a80d84"},"cell_type":"code","source":"X_all_df.loc[temp_df.index, \"trafficSource.keyword\"] = \"(not provided)\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a3d2f1531fb3942f7984d3478115797854ab33a"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"trafficSource.referralPath\"].isnull(), \"channelGrouping\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6562466b5e3647fbbdc83f76c6c52ca43e102830"},"cell_type":"markdown","source":"There 14 rows that dosen't have a path, let's assign them with the most common path."},{"metadata":{"trusted":true,"_uuid":"e13064a6d1cb50941c7d755abb2be0ff38846662"},"cell_type":"code","source":"temp_df = X_all_df.loc[X_all_df[\"trafficSource.referralPath\"].isnull(), :]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c06fec41fb38bb3b5702bf5528228fb4e6fa3b91"},"cell_type":"code","source":"X_all_df.loc[temp_df.loc[(temp_df[\"channelGrouping\"] == \"Referral\") | (temp_df[\"channelGrouping\"] == \"Social\")].index, \"trafficSource.referralPath\"] = X_all_df[\"trafficSource.referralPath\"].value_counts().index[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"babb9af4d596de044184c57bc2256cf1a8abc7f4"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"trafficSource.source\"] == \"(direct)\", \"channelGrouping\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ccdf227c5c8cbcf468565c5b36422139639a9cc2"},"cell_type":"markdown","source":"There are two rows mislabeled, change them back to right values."},{"metadata":{"trusted":true,"_uuid":"8f49a6c857ce662f4b2277f0f270a7b7732fd811"},"cell_type":"code","source":"X_all_df.loc[X_all_df[\"trafficSource.source\"] == \"(direct)\", \"channelGrouping\"] = \"Direct\"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b9d00cbbf1b958aeb683425f16efc6d267c9a9a3"},"cell_type":"markdown","source":"Now we can fill up the `NaN` value with some category"},{"metadata":{"trusted":true,"_uuid":"a02f5b5ceb4d4a78316054a3d7ac94d3819d6314"},"cell_type":"code","source":"X_all_df[\"trafficSource.referralPath\"] = X_all_df[\"trafficSource.referralPath\"].cat.add_categories(['Not Referral'])\n\nX_all_df.loc[:, \"trafficSource.referralPath\"] = X_all_df.loc[:, \"trafficSource.referralPath\"].fillna('Not Referral')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d79ba3b98f43074e746ee36ee8fa62492d034fb"},"cell_type":"markdown","source":"### Merge the Category With Small Instance\nThere are features that have a category that with only one or two instances. It will be helpful for both data analysis and prediction to merge them all into a big category. This is proven later by us that it won't affect the final prediction accuracy at all, so we will do the merge because it will help with data visualisation."},{"metadata":{"trusted":true,"_uuid":"8b5cced822495d69f2366ebd0ca7081046387b1f","_kg_hide-input":true},"cell_type":"code","source":"def percentage_counts(data, col_name):\n    total = data[col_name].count()\n    return data[col_name].value_counts()/ total * 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6fe8c68f7c62e5cbf53cae714a6d0d68f415f713","_kg_hide-input":true},"cell_type":"code","source":"def keep_greater_than_percentage_entries(data, col_name, percentage=0.1, new_entry=\"Others\"):\n    percentage_counts_series = percentage_counts(temp_df, col_name)\n    entries_to_keep = percentage_counts_series[percentage_counts_series >= percentage].index\n    data[col_name] = data[col_name].cat.add_categories([new_entry])\n    data.loc[~data[col_name].isin(entries_to_keep), col_name] = new_entry\n    data[col_name] = data[col_name].cat.remove_unused_categories()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d76355208e381402d3137dc5991cb1375262adb8"},"cell_type":"code","source":"features_to_merge = [\n    'device.browser', 'geoNetwork.city', 'geoNetwork.country',\n    'geoNetwork.metro', 'geoNetwork.networkDomain', 'geoNetwork.region',\n    'geoNetwork.subContinent', 'trafficSource.campaign', 'trafficSource.keyword',\n    'trafficSource.referralPath', 'trafficSource.source', 'device.operatingSystem'\n                    ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"97201aa26a3681b4dec44fe2df2cfc55eb0c3cfd"},"cell_type":"code","source":"for feature in features_to_merge:\n    keep_greater_than_percentage_entries(data=X_all_df, col_name=feature)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"414ebfffba56483269c70e910fbe0922141314ad"},"cell_type":"markdown","source":"Split the dataset into training dataset for later analysis"},{"metadata":{"trusted":true,"_uuid":"8de9712c48e9a23a4b59f7ea7b96893d2407be21"},"cell_type":"code","source":"X_train_df = X_all_df.loc[X_all_df[\"totals.transactionRevenue_ln\"].notnull(), :].reset_index(drop=True)\nX_test_df = X_all_df.loc[X_all_df[\"totals.transactionRevenue_ln\"].isnull(), :].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d56bec4d77c99126257cc8b12d0b9d79344edd3"},"cell_type":"markdown","source":"## Features Analysis\n---\nAfter all efforts in preprocessing, we can now look into the features we will deal with. As this is a iterative developing workflow, we will go through some features at first time and focus on the top three most important features."},{"metadata":{"_uuid":"4dbbb7b77c3987841d33e8c1a6be4cebd96dc47a"},"cell_type":"markdown","source":"## Analysis Base On Date"},{"metadata":{"trusted":true,"_uuid":"6626ec4b8e5868c61ebe23a775d46cdbcbe79080"},"cell_type":"code","source":"X_train_df.groupby(\"date\")[\"totals.transactionRevenue_ln\"].agg(['sum', 'count', 'mean']).plot(subplots=True, sharex=True, title=\"Revenue Base On Date\", linewidth=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a29da1537d3a719b00e5702bdadec40905940b05"},"cell_type":"markdown","source":"From the data revenue plots we can tell that, from the middle of November till near the Christmas, there is a surge both in visit counts and sum up revenue. The visit counts drop significantly after the Christmas."},{"metadata":{"_uuid":"f7eeeb039ff3369032e284fe7b64200171eed18c"},"cell_type":"markdown","source":"## Traffic Source Analysis"},{"metadata":{"trusted":true,"_uuid":"5b77e4017ce6a01b72ad1eca9766b3669aaf2820"},"cell_type":"code","source":"sub_boxenplots(x='channelGrouping', data=X_train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4459009d383325e32cc97534f1f285f346ac5048"},"cell_type":"markdown","source":"From the plot we can infer that the instances who comes from `referral` website has the highest ritio of non-zero revenue, while the instances come from `Display` channel have the highest median non-zero revenue."},{"metadata":{"trusted":true,"_uuid":"a5bba1ce30b1602d440930fc1f568dba78f4da79"},"cell_type":"code","source":"sub_boxenplots(x='device.deviceCategory', data=X_train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"563b3ad987c4312161d2e4e290f8c6ff1bddbdab"},"cell_type":"markdown","source":"People who use the desktop are still the main income source of GA store."},{"metadata":{"trusted":true,"_uuid":"036ccc6c786018ea019cac6be5c0cccc66609867"},"cell_type":"code","source":"sub_boxenplots(x='device.isMobile', data=X_train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c687ccb336c3e78157e3f20644781f17d74eace"},"cell_type":"markdown","source":"This information might be redundant to the previous one `device.deviceCategory`. Drop this feature if the feature importance is  not high."},{"metadata":{"trusted":true,"_uuid":"b841d5866da6b614a26b9851d1af04c4f4b129b4"},"cell_type":"code","source":"sub_boxenplots(x='geoNetwork.continent', data=X_train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae52e6af91c3180e372cd2ec5e54955ef956c024"},"cell_type":"markdown","source":"Americas are still the main market for the GA store but among the group who spends the money that `Africa` has the most dense non-zero nature log revenue within the range of 18-22."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"ab9929f1ad56794a64c9bd1ea60423430ce07089"},"cell_type":"code","source":"sub_boxenplots(x='totals.newVisits', data=X_train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c03142da3c7ed98e2de4605494753b521f36bf97"},"cell_type":"code","source":"sub_boxenplots(x='trafficSource.isTrueDirect', data=X_train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5e583ac2b8709c04bec70d7482cf917cc92bfc1"},"cell_type":"markdown","source":"Above two binary features are quite important and intuitive as well. An old user is more likely to make a purchase than a new user. Those who visit website directly are more objective than the other means."},{"metadata":{"trusted":true,"_uuid":"f42730d654e7003ede1c73953d3446474b4a8eb5"},"cell_type":"code","source":"sub_boxenplots(x='trafficSource.campaign', data=X_train_df, rot=20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"543c5eded611ace6123186b4d563db908719df79"},"cell_type":"code","source":"sub_boxenplots(x='trafficSource.medium', data=X_train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c2f6c0282ffcc240523c6a19618de4eafe8f6590","scrolled":false},"cell_type":"code","source":"sub_boxenplots(x='device.operatingSystem', data=X_train_df, rot=20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"708c2db7a56c9e37c1331d99862a8b2f2209227f"},"cell_type":"markdown","source":"We can tell that the people who choose `Chrome OS` will contribute slightly more than other operating system. "},{"metadata":{"_uuid":"7549d10826ed9fa58429ee9b66c3cef6a609ed48"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"44d26112e4c963898dcf837ecbe06603f6c98e76"},"cell_type":"markdown","source":"## Feature Explore On Import Features\nThe features we explore below will be the features that _LightGBM_ thinks it is important. Let's find out are they make sense to us."},{"metadata":{"trusted":true,"_uuid":"fbbe88addd778277eb8d0bec8143c00e31ef20cc"},"cell_type":"code","source":"g = sns.jointplot('totals.hits', 'totals.transactionRevenue_ln', data=X_train_df[X_train_df['totals.transactionRevenue_ln'] > 0],\n                 kind='reg', height=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9040b62769dab4d2478667b79561cdefe31a0a69"},"cell_type":"code","source":"g = sns.jointplot('visitStartTime', 'totals.transactionRevenue_ln', data=X_train_df[X_train_df['totals.transactionRevenue_ln'] > 0],\n                 kind='reg', height=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"83015db0594f4a4a2a33c5dca234f53258eb554d"},"cell_type":"code","source":"g = sns.jointplot('totals.pageviews', 'totals.transactionRevenue_ln', data=X_train_df[X_train_df['totals.transactionRevenue_ln'] > 0],\n                 kind='reg', height=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22ecfd888bad6cbd9592df157dab04d3ef8d840e"},"cell_type":"markdown","source":"**Observation:** Above are the most three important features that _LightGBM_ thinks that are important. `pageviews` and `hits` make sense while `visitStartTime` seems a big mistake. Like Teo mentioned during our presentation that, a single algorithm will have its bias."},{"metadata":{"_uuid":"637b12c5f98d94b71d27616842e076313c39fe90"},"cell_type":"markdown","source":"## Feature Encoding\n---\nString feature is not usable for machine learning algorithm. We need to encode them to either numeric featuers or one-hot encoding features. This deponds on the machine learning algorithm you choose. Before we start, we need to find out the category of the features. Unfortinately, it has to be done by hand pick."},{"metadata":{"trusted":true,"_uuid":"eeee3077d428017036180d83e09b8816433fe8c8","_kg_hide-input":false},"cell_type":"code","source":"cat_feature_list = ['channelGrouping', 'device.browser', 'device.deviceCategory', 'device.operatingSystem', \n                    'geoNetwork.city', 'geoNetwork.continent', 'geoNetwork.country', 'geoNetwork.metro', 'geoNetwork.networkDomain', 'geoNetwork.region', 'geoNetwork.subContinent',\n                    'trafficSource.campaign', 'trafficSource.isTrueDirect', 'trafficSource.keyword', 'trafficSource.medium', 'trafficSource.referralPath', 'trafficSource.source'\n                   ]\nbool_feature_list = ['device.isMobile', 'totals.newVisits']\nid_feature_list = ['fullVisitorId']\nnum_feature_list = ['visitNumber', 'totals.hits', 'totals.pageviews', 'visitStartTime']\ntime_feature_list = ['date']\nlabel_list = ['totals.transactionRevenue', 'totals.transactionRevenue_ln']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c6b5770a7a992285850ed4cea0f16528b1dac3e1"},"cell_type":"markdown","source":"Check if any features are missing."},{"metadata":{"trusted":true,"_uuid":"7ea8e369f6abd6d0a8560e3108a372aae4560c5b","_kg_hide-input":true},"cell_type":"code","source":"def check_features():\n    found = True\n    total_feature_list = id_feature_list + label_list + cat_feature_list + num_feature_list + bool_feature_list + time_feature_list\n    for feature in X_train_df.columns:\n        if feature not in total_feature_list:\n            found = False\n        assert found, \"You forgot \" + feature\n        total_feature_list.remove(feature)\n    if found:\n        if total_feature_list == []:\n            print(\"All the features are found!\")\n        else:\n            print(\"There are features left: \", total_feature_list)\ntry:\n    check_features()\nexcept AssertionError as e:\n    print(e)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b13ba016cb1d44e8bda61ec4ee0592ccca3e0355"},"cell_type":"markdown","source":"Rearrange the features by column categories."},{"metadata":{"trusted":true,"_uuid":"e22d364cf3f4f97a1b62337cbce6663ca7738d87"},"cell_type":"code","source":"X_all_df = X_all_df[id_feature_list + label_list + cat_feature_list + num_feature_list + bool_feature_list + time_feature_list]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6302415e5cd0b6f91cb222eaded0e18ceac8c8d3","scrolled":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b4ebee1c4e2f22aa259317b16de024584c538ac5"},"cell_type":"code","source":"def cat_feature_encoding(data, cat_feature_list):\n    feature_encoder_dict = {}\n    with tqdm(cat_feature_list, desc=cat_feature_list[0]) as t:\n        for cat_feature in t:\n            if cat_feature == \"fullVisitorId\": # leave the ID as it is \n                continue\n            t.set_description_str(cat_feature)\n            my_label_encoder = LabelEncoder()\n            encoded_col = my_label_encoder.fit_transform(data[cat_feature])\n            data.loc[:, cat_feature] = encoded_col\n            feature_encoder_dict[cat_feature] = my_label_encoder\n    return feature_encoder_dict","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01d83e8cc6ba63ce1af51d1583053add8f5b4640"},"cell_type":"markdown","source":"Now split the X_all_df to training and testing dataset again."},{"metadata":{"trusted":true,"_uuid":"6911f40c453f48b1cf48e4e78a24a6bd44b61e78"},"cell_type":"code","source":"X_train_df = X_all_df.loc[X_all_df[\"totals.transactionRevenue_ln\"].notnull(), :].reset_index(drop=True)\nX_test_df = X_all_df.loc[X_all_df[\"totals.transactionRevenue_ln\"].isnull(), :].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3e304f199ddb74ca1f2c21df9a868ddde2ce85df"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3850c852d434e774d230105febbc0e2d657f5521"},"cell_type":"markdown","source":"# Feature Engineering \n**What is going to be achieved in this section:**\n\n* Create custom features\n* Utilize the  [featuretools library](https://www.featuretools.com/) to create useful features\n* Merge them together "},{"metadata":{"trusted":true,"_uuid":"72598f6142a888b87940dd624c63b38766e2af1b"},"cell_type":"code","source":"train_df = X_train_df\ntest_df = X_test_df","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"fa14751b250c442d5195a44650bd627a3f2d0ba8"},"cell_type":"code","source":"def printShapes(name1, df1, name2, df2):\n    print(name1, df1.shape)\n    print(name2, df2.shape)\n    \ndef ratio_nan(data):\n    num_data = data.shape[0]\n    null_sum = data.isnull().sum()\n    null_val_features = null_sum[null_sum > 0]\n    print(null_val_features/num_data)\n    \ndef compare_nan(name1, df1, name2, df2):\n    print(name1)\n    print(ratio_nan(df1))\n    print(\"\\n\" + name2)\n    print(ratio_nan(df2))\n    \nimport datetime as dt\ndef toPandasTimestamp(val):\n    lDate = dt.datetime.fromtimestamp(val)\n    lTimestamp = pd.Timestamp(lDate)\n    return lTimestamp\n    \ndef printTTShapes():\n    printShapes(\"Train Shape:\", train_df, \"Test Shape:\", test_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d36453b0d2690e034e75e45959c743a73a13b6ce"},"cell_type":"markdown","source":"It is useful to create a category that is a pandas timestamp for feature tools library to use. Here we do that."},{"metadata":{"_kg_hide-input":false,"trusted":true,"_uuid":"91e3ec4067f87259137e1904ea5081a1dd11e499"},"cell_type":"code","source":"train_df[\"visitStartTimestamp\"] = train_df[\"visitStartTime\"].apply(toPandasTimestamp)\ntest_df[\"visitStartTimestamp\"] = test_df[\"visitStartTime\"].apply(toPandasTimestamp)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c25ac357749b2a4b2c451520303b41165c044fa1"},"cell_type":"code","source":"printTTShapes()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8984c02f148e54cd2b41d92d2fba49c6b660899c"},"cell_type":"code","source":"compare_nan(\"Train\", train_df, \"Test\", test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"faeb0a4aa6e093acd4d4fd851c44640b95d2c693"},"cell_type":"code","source":"train_df[\"totals.transactionRevenue\"] = train_df[\"totals.transactionRevenue\"].apply(lambda x: 0.0 if np.isnan(x) else x) \ntest_df[\"totals.transactionRevenue\"] = test_df[\"totals.transactionRevenue\"].apply(lambda x: 0.0 if np.isnan(x) else x)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"56097222b08d8c73d02de1549201885980280f7b"},"cell_type":"markdown","source":"## Custom Feature Engineering\n\nIn trying to engineer some new features, I came up with these ideas:\n\n1. Difference in time between min and max visitStartTimes\n2. Number of seconds between hits (a lot will just be zero)\n3. Number of seconds between visits\n4. Number of seconds between pageviews\n\nThe following section will focus on creating these features."},{"metadata":{"trusted":true,"_uuid":"4f7e15fa975a8fe1151fd13b6068d4062d0c7b6e"},"cell_type":"code","source":"columnsOfInterest = [\"fullVisitorId\", \"visitStartTime\", \"totals.hits\", \"totals.pageviews\", \"visitNumber\"]\ntrain_grouped = train_df[columnsOfInterest].groupby(\"fullVisitorId\")\ntest_grouped = test_df[columnsOfInterest].groupby(\"fullVisitorId\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d84c7391da8d5918f5764005a7d6835a40325b1d"},"cell_type":"code","source":"train_ds = train_grouped[\"visitStartTime\"].max() - train_grouped[\"visitStartTime\"].min() # ds difference in seconds\ntest_ds = test_grouped[\"visitStartTime\"].max() - test_grouped[\"visitStartTime\"].min()\n\ntrain_sh = train_grouped[\"totals.hits\"].sum() # sh - summed hits\ntrain_spv = train_grouped[\"totals.pageviews\"].sum() # spv - summed page views\ntrain_lv = train_grouped[\"visitNumber\"].max() #lv - largestVisit number\ntest_sh = test_grouped[\"totals.hits\"].sum() # sh - summed hits\ntest_spv = test_grouped[\"totals.pageviews\"].sum() # spv - summed page views\ntest_lv = test_grouped[\"visitNumber\"].max() #lv - largestVisit number\n\ntrain_seconds_per_hit = train_ds / train_sh\ntrain_seconds_per_pageview = train_ds / train_spv \ntrain_seconds_per_visit = train_ds / train_lv\ntest_seconds_per_hit = test_ds / test_sh\ntest_seconds_per_pageview = test_ds / test_spv \ntest_seconds_per_visit = test_ds / test_lv\n\ntrain_nf = pd.concat([train_ds, train_seconds_per_hit, train_seconds_per_pageview, train_seconds_per_visit], \n                    axis = 1, join = \"outer\", \n                    join_axes = [train_ds.reset_index()[\"fullVisitorId\"]])\ntest_nf= pd.concat([test_ds, test_seconds_per_hit, test_seconds_per_pageview, test_seconds_per_visit], \n                    axis = 1, join = \"outer\", \n                    join_axes = [test_ds.reset_index()[\"fullVisitorId\"]])\n\ntrain_nf.reset_index(inplace = True)\ntrain_nf.columns = [\"fullVisitorId\", \"visitTimeRange\", \"secondsPerHit\", \"secondsPerPageview\", \"secondsPerVisit\"]\ntest_nf.reset_index(inplace = True)\ntest_nf.columns = [\"fullVisitorId\", \"visitTimeRange\", \"secondsPerHit\", \"secondsPerPageview\", \"secondsPerVisit\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c80a1b094b331c964b073478637546d2023472c"},"cell_type":"code","source":"printShapes(\"Train New Features:\", train_nf, \"Test New Features:\", test_nf)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"70fa15d0a76c126a2e743c0486a398019578ac6b"},"cell_type":"markdown","source":"Ensure that we get rid of all NaNs"},{"metadata":{"trusted":true,"_uuid":"115727012c30c77d1d0d08600adebf3aede9ab42"},"cell_type":"code","source":"compare_nan(\"Train New Features\", train_nf, \"Test New Features\", test_nf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d66a8b93dfc281e5a56358d5fbcdc7a0ff24353a"},"cell_type":"code","source":"train_nf = train_nf.apply(lambda x: x.fillna(0) if x.dtype.kind in 'biufc' else x.fillna('.'))\ntest_nf = test_nf.apply(lambda x: x.fillna(0) if x.dtype.kind in 'biufc' else x.fillna('.'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d4f79307b8b0e030d840a31ac94a8b91f58489f"},"cell_type":"code","source":"compare_nan(\"Train New Features\", train_nf, \"Test New Features\", test_nf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7afbb123ea84b9f372f944195a642312efa78bfb"},"cell_type":"code","source":"train_nf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2edcd9c4c75c4ff941ac42a531f166966ba0c3c2"},"cell_type":"code","source":"test_nf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f06cd0b76d532add48b10b3671f7cfcf15238c0"},"cell_type":"markdown","source":"## Feature Tools\n\nHere we utilize the feature tools library to generate some features we know are good. Previously, we have trained on all columns with all entries. However, that takes over two hours to run. So, once we did that and found the best features, we have tried to narrow it down to reduce run time and make our lives easier. "},{"metadata":{"trusted":true,"_uuid":"2865a86d740698a02c358b83a91f6f87291adf0d"},"cell_type":"code","source":"fm_columns = [\"fullVisitorId\", \"totals.pageviews\", \"totals.hits\", \"visitStartTimestamp\", \"geoNetwork.city\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"474b7a42ed36d3007e57cfe2f715b3aab47f90c5"},"cell_type":"code","source":"ftrain_df = train_df[fm_columns].reset_index()\nftest_df = test_df[fm_columns].reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1730ce6f599ff5d5ce210cd9c5d5f2bea7484d2d"},"cell_type":"code","source":"ftrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f45f7237ee1b5ef7791c08a5a075da823e2aace7"},"cell_type":"code","source":"ftest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e406a018544f208220331e2ee355512edc42745d"},"cell_type":"markdown","source":"### EntitySets\nFeaturetools uses EntitySets to represent dataframes and the relationships between them. Here we create an entity set from our whole dataframe and then pull another dataframe out from that (using fullVisitorId). This is very powerful and later allows us to automatically use grouped information. "},{"metadata":{"trusted":true,"_uuid":"907c140038fa5357d7877f8ae3b5ffdf97d100e2"},"cell_type":"code","source":"import featuretools as ft\nfrom featuretools import variable_types as vtype\n\nmy_variable_types = {\n    \"fullVisitorId\" : vtype.Id,\n    \"geoNetwork.city\" : vtype.Categorical\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5170ed3235e0c29742bc5619c3120b68b926a4c"},"cell_type":"code","source":"train_es = ft.EntitySet(id = \"train_data\")\ntest_es = ft.EntitySet(id = \"test_data\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c8280ec1a3ad01c94c0997ef353acf626a0485b3"},"cell_type":"code","source":"train_es.entity_from_dataframe(entity_id = \"train_log\",\n                               dataframe = ftrain_df,\n                               index = \"index\",\n                               time_index = \"visitStartTimestamp\",\n                               variable_types = my_variable_types)\n\ntest_es.entity_from_dataframe(entity_id = \"test_log\",\n                              dataframe = ftest_df,\n                              index = \"index\",\n                              time_index = \"visitStartTimestamp\",\n                              variable_types = my_variable_types)\n\nprint(train_es, test_es)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c876b733df65219e77dc3cf4e29697a2e5111b2c"},"cell_type":"markdown","source":"This is where we pull an entity from the original (train/test_log) and make a new entity. Note: Relationships is no longer empty in the output. "},{"metadata":{"trusted":true,"_uuid":"5906c080a4c4fae2601fd448675ddfe7d6f2456c"},"cell_type":"code","source":"train_es.normalize_entity(base_entity_id = \"train_log\",\n                          new_entity_id = \"visitors\",\n                          index = \"fullVisitorId\")\n\ntest_es.normalize_entity(base_entity_id = \"test_log\",\n                         new_entity_id = \"visitors\",\n                         index = \"fullVisitorId\")\n\nprint(train_es, test_es)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f519190ab7ac1510e04418e1145da31b7ba11dd"},"cell_type":"markdown","source":"In an attempt to make things quicker we have also narrowed down the aggregation and transformation functions to be applied. In the output, you can see the features that will be generated."},{"metadata":{"trusted":true,"_uuid":"e9fafb1deb869ef9abdd8fd4621e44d6a03a274a"},"cell_type":"code","source":"my_aggs = [\"sum\", \"max\", \"min\", \"mean\", \"std\", \"mode\"]\nmy_trans = [\"month\", \"day\"]\n\ntrain_features = ft.dfs(entityset = train_es,\n                        target_entity = \"visitors\",\n                        max_depth = 2,\n                        agg_primitives = my_aggs,\n                        trans_primitives = my_trans,\n                        verbose = 1,\n                        max_features = 100,\n                        n_jobs = 4,\n                        features_only = True)\n\ntest_features = ft.dfs(entityset = test_es,\n                        target_entity = \"visitors\",\n                        max_depth = 2,\n                        agg_primitives = my_aggs,\n                        trans_primitives = my_trans,\n                        verbose = 1,\n                        max_features = 100,\n                        n_jobs = 4,\n                        features_only = True)\n\nprint(\"We're generating %d features for train and %d for test\" % (len(train_features), len(test_features)))\nfor feat in train_features:\n    print(feat)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d90091158cff4ea2f57a1937958777c59284df5"},"cell_type":"markdown","source":"Here is where the actual functions are used and applied. This part takes around 5 minutes."},{"metadata":{"trusted":true,"_uuid":"40562c6d465662bbfeabeb4186bccb812789ee9a"},"cell_type":"code","source":"train_feature_matrix, train_features = ft.dfs(entityset = train_es,\n                                              target_entity = \"visitors\",\n                                              max_depth = 2,\n                                              agg_primitives = my_aggs,\n                                              trans_primitives = my_trans,\n                                              verbose = 1,\n                                              max_features = 100,\n                                              n_jobs = 4)\n\ntest_feature_matrix, test_features = ft.dfs(entityset = test_es,\n                                            target_entity = \"visitors\",\n                                            max_depth = 2,\n                                            agg_primitives = my_aggs,\n                                            trans_primitives = my_trans,\n                                            verbose = 1,\n                                            max_features = 100,\n                                            n_jobs = 4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c60ccc46b28896711a582b688a744926a384ceab"},"cell_type":"code","source":"train_feature_matrix.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5cb6a8a0d7525bb4a5453d6ac02ca9dbe2c2473e"},"cell_type":"code","source":"test_feature_matrix.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"35b1a2d0c9075744892d710c65e17e852ca914f4"},"cell_type":"markdown","source":"Here we clean up the data we have generated."},{"metadata":{"trusted":true,"_uuid":"dae12968a112321f474fd99d2893559ea3477e76"},"cell_type":"code","source":"printShapes(\"Train Feature Matrix:\", train_feature_matrix,\n            \"Test Feature Matrix\", test_feature_matrix)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"450d24bf11dfc2d09753eb141f66a60255ae4092"},"cell_type":"code","source":"compare_nan(\"Train FM\", train_feature_matrix,\n            \"Test FM\", test_feature_matrix)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9c32b251a728ff33422693ba9de4bf08710f0322"},"cell_type":"code","source":"train_feature_matrix = train_feature_matrix.apply(lambda x: x.fillna(0) if x.dtype.kind in 'biufc' else x.fillna('.'))\ntest_feature_matrix = test_feature_matrix.apply(lambda x: x.fillna(0) if x.dtype.kind in 'biufc' else x.fillna('.'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1eeda7bce854f95c456691370948f8e6bf2b8b18"},"cell_type":"code","source":"compare_nan(\"Train FM\", train_feature_matrix,\n            \"Test FM\", test_feature_matrix)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dca81f9a3340ec0b4a169e40be8658880092371c"},"cell_type":"markdown","source":"## Merging Data"},{"metadata":{"trusted":true,"_uuid":"dfafed4bb8e54c0d733746d3136aa529aa356a3c"},"cell_type":"code","source":"train_feature_matrix.reset_index(inplace = True)\ntest_feature_matrix.reset_index(inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f87a65a4a2292e6ff4326110c596bbe0133f44b0"},"cell_type":"code","source":"combined_train = train_nf.merge(train_feature_matrix, how = \"outer\", on = \"fullVisitorId\")\ncombined_test = test_nf.merge(test_feature_matrix, how = \"outer\", on = \"fullVisitorId\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"73f57f6c36f289b1b3fe249189d607b9afc7036a"},"cell_type":"code","source":"combined_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5e248990c3c5897d7822ff1b3dbda691a7d40a5"},"cell_type":"code","source":"combined_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fbde924679af07ff9cf6ca5320d5c5b9c4a1dca3"},"cell_type":"code","source":"compare_nan(\"CTrain\", combined_train, \"CTest\", combined_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a28bfe158aacc4b68af3a10c57100c5bf1a31459"},"cell_type":"code","source":"train_rev = train_df[[\"fullVisitorId\", \"totals.transactionRevenue\"]].groupby(\"fullVisitorId\").sum().reset_index()\ntest_rev = test_df[[\"fullVisitorId\", \"totals.transactionRevenue\"]].groupby(\"fullVisitorId\").sum().reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee388aae601bc7262ad1bef6a2dbaf3d1f3a3cd0"},"cell_type":"code","source":"combined_train_wrev = combined_train.merge(train_rev, how = \"outer\", on = \"fullVisitorId\")\ncombined_test_wrev = combined_test.merge(test_rev, how = \"outer\", on = \"fullVisitorId\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb1f9e441f3f940323188ebdbf7531a99117c6b9"},"cell_type":"code","source":"combined_train_wrev = combined_train_wrev.rename(columns = {\"totals.transactionRevenue\" : \"SUM(transactionRevenue)\"})\ncombined_test_wrev = combined_test_wrev.rename(columns = {\"totals.transactionRevenue\" : \"SUM(transactionRevenue)\"})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5ee0265df4e80432531ee7d5be0ae962249f901f"},"cell_type":"markdown","source":"# LightGBM\n---"},{"metadata":{"trusted":true,"_uuid":"c22549dba74c7a950cd3ed9947b9161e6e7d7ab6"},"cell_type":"code","source":"import lightgbm as lgb\nfrom sklearn.model_selection import KFold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a80b000274d08ecc6905d2170711ceac95a14750"},"cell_type":"code","source":"new_train_features_encoding_dict = cat_feature_encoding(data=combined_train_wrev, cat_feature_list=combined_train_wrev.select_dtypes(include='object').columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"39418116e6dc4ede47b4b937fd529d76de9fed5d"},"cell_type":"code","source":"new_test_features_encoding_dict = cat_feature_encoding(data=combined_test_wrev, cat_feature_list=combined_test_wrev.select_dtypes(include='object').columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1c5bb42ce72680b0ed632f2902bfd40dad0d2760","_kg_hide-input":true},"cell_type":"code","source":"y = np.log1p(combined_train_wrev.iloc[:, -1])\n\nX = combined_train_wrev.iloc[:, 1:-1]\n\nX_pred = combined_test_wrev.iloc[:, 1:-1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5560f2e5bfc7ba926f88fde7541ea3ecfa03f80f"},"cell_type":"markdown","source":"This parameters comes from this [kernel](http://https://www.kaggle.com/ogrellier/user-level-lightgbm-lb-1-4480/code). As we don't want to tune the parameters to control the experiment variables."},{"metadata":{"trusted":true,"_uuid":"ddc9cd277879d6b171a8d1b3969fdad030a7c9a0","_kg_hide-input":true},"cell_type":"code","source":"lgb_clf = lgb.LGBMRegressor(learning_rate=0.03, n_estimators=2000, min_child_weight=np.power(10.0, 2), metric='rmse', \n                             num_leaves=128, reg_alpha=np.power(10.0, -3.2454), reg_lambda = np.power(10.0, -4.8571), silent=True, n_jobs=-1,\n                             colsample_bytree =  0.6810, min_child_samples = 95,  subsample = 0.2217, min_split_gain=np.power(10.0, -4.9380))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"035cc3ca44f33c70c41673ecee1e29ad766e8545","_kg_hide-input":true},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\nkfold = 5\nkf = KFold(n_splits=kfold, shuffle=True)\n\npredicts_result = []\ntest_result = []\nfor train_index, test_index in kf.split(X, y):\n    print(\"#\"*10)\n    X_train, X_val = X.iloc[train_index], X.iloc[test_index]\n    y_train, y_val = y.iloc[train_index], y.iloc[test_index]\n    lgb_clf.fit(X_train, y_train, eval_set=[(X_val, y_val)], early_stopping_rounds=400, verbose=200, eval_metric='rmse') #eval_metric=f1_macro_evaluation)\n#     test_result.append(f1_score(y_pred=lgb_clf.predict(X_val), y_true=y_val, average=\"macro\"))\n    y_val_pred = lgb_clf.predict(X_val)\n    # Modify the value that is below the zero\n    y_val_pred[y_val_pred<0] = 0\n    test_result.append(mean_squared_error(y_true=y_val, y_pred=y_val_pred))\n    y_pred = lgb_clf.predict(X_pred, lgb_clf.best_iteration_)\n    y_pred[y_pred<0] = 0\n    predicts_result.append(y_pred)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38c18f14ce794cb554c21955655cb1f1069b5693"},"cell_type":"markdown","source":"The average test RMSE will much higher than our actual prediction is because we filter up all the bouce instance to reduce the work during the feature engineering."},{"metadata":{"trusted":true,"_uuid":"b7b3957a09b66bd83c13e9e0fb373acda74d6931"},"cell_type":"code","source":"print(\"The average test RMSE is: \", np.mean(np.sqrt(test_result)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"50f5e558d72c10328e84c30780e39ea9a62bb8f6","_kg_hide-input":true},"cell_type":"code","source":"def plot_features(col_list, feature_importances, index, most_important=True):\n    indices = np.argsort(feature_importances)[::-1]\n    indices = indices[:index]\n\n    # Visualise these with a barplot\n    plt.subplots(figsize=(20, 15))\n    g = sns.barplot(y=col_list[indices], x = lgb_clf.feature_importances_[indices], orient='h')\n    g.set_xlabel(\"Relative importance\",fontsize=20)\n    g.set_ylabel(\"Features\",fontsize=20)\n    g.tick_params(labelsize=15)\n    g.set_title(\"LightGBM feature importance\", fontsize=20);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"36e1aa3bc28a4e6fc884c72068ced1907d143615","_kg_hide-input":true},"cell_type":"code","source":"plot_features(col_list=X.columns, feature_importances=lgb_clf.feature_importances_, index=-1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f7afd97548b13919f42266ed33ec8417fea23f9"},"cell_type":"markdown","source":"Merge the prediction with the bounce instance we filter."},{"metadata":{"trusted":true,"_uuid":"8e5493a5e2f5f47c7f93a9d4c9b77f123907436b","_kg_hide-input":true},"cell_type":"code","source":"no_bounce_pred_df = pd.DataFrame({\"fullVisitorId\": combined_test_wrev.fullVisitorId, \"PredictedLogRevenue\": np.mean(predicts_result, axis=0)})\n\nsubmit_df = pd.DataFrame({\"fullVisitorId\": submit_id})\n\nsubmit_df = submit_df.merge(no_bounce_pred_df, on=\"fullVisitorId\", how='left').fillna(0)\n\nsubmit_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87224e7b944b69e135655d68d10920f69f3826e9"},"cell_type":"markdown","source":"Take a look what our prediction looks like."},{"metadata":{"trusted":true,"_uuid":"48e5dffc4e2cea01ab4b39b0ee708cbb9d99707b","_kg_hide-input":true},"cell_type":"code","source":"submit_df[submit_df.PredictedLogRevenue != 0].hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d60badb7d9188f333b027d5acecd661ed07d38ae","scrolled":true,"_kg_hide-input":true},"cell_type":"code","source":"labels = ['Zero revenue customer', 'Non-zero customer']\nvalues = [submit_df[submit_df.PredictedLogRevenue == 0].PredictedLogRevenue.count(),\n         submit_df[submit_df.PredictedLogRevenue != 0].PredictedLogRevenue.count()]\ntrace = go.Pie(labels=labels, values=values)\nlayout = dict(\n    title = 'Unique Customer Revenue Prediction'\n)\n\nfig = dict(data=[trace], layout=layout)\niplot(fig, filename='basic_pie_chart')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6bfa441c23edf64aaa60ec51cd3c697a90a4ca15"},"cell_type":"markdown","source":"# Conclusion\nIn this kernel we have tried three different iterations. \n\n1. Naive - Score : 1.7340\n2. Feature Engineering - Score : 1.5665\n3. Reduced Feature Engineering (for runtime) with custom features added - Score :  1.5372\n\nClearly, 1.5665 was the best iteration. \n\nThis kernel has highlighted the usefulness of Plotly, Featuretools, lightGBM. We would use these all again. "},{"metadata":{"trusted":true,"_uuid":"1db1580a4e7decfd48e4db48c0deae58e106b9d4"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}