{"cells":[{"metadata":{"_uuid":"163a1a9c9d4fc3f61c766fdf860af902bd8e762c"},"cell_type":"markdown","source":"# Google Analytics Customer Revenue Prediction\n\n\n\n### Contents of this Kernel\n\n1. Problem Statement  \n2. Dataset Understanding  \n3. Exploration  \n4. Visitor Profile  \n5. Baseline Model  \n\n## 1. Problem Statement \n\nIn this [competition](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction), the aim is to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. The results of predictions and analysis might lead to more actionable operational changes and a better use of marketing budgets for those companies who choose to use data analysis on top of GA data. This is the starter baseline kernel, I will be updating it frequently. \n\nAs the first step, lets load the required libraries.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport json\nimport bq_helper\nfrom pandas.io.json import json_normalize\nimport seaborn as sns \nimport matplotlib.pyplot as plt \nfrom plotly.offline import init_notebook_mode, iplot\nimport plotly.graph_objs as go\nfrom plotly import tools\ninit_notebook_mode(connected=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d53b5bdd24383e8eb78b163fa3047cd53febfc7d"},"cell_type":"markdown","source":"## 2. Dataset Understanding\n\nThe data is shared in big query and csv format. The csv files contains some filed with json objects. The description about dataset fields is given [here](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/data). Lets read the dataset in csv format and unwrap the json fields. I am using the [function](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/data) shared by @julian in his kernel.  \n\n### 2.1 Dataset Preparation"},{"metadata":{"trusted":true,"_uuid":"5a88e31d7edecf4f92855192f0260f6884dc9da0","_kg_hide-input":true},"cell_type":"code","source":"json_cols = ['device', 'geoNetwork', 'totals', 'trafficSource']\ndef load_df(filename):\n    path = \"../input/\" + filename\n    df = pd.read_csv(path, converters={column: json.loads for column in json_cols}, \n                     dtype={'fullVisitorId': 'str'})\n    \n    for column in json_cols:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}_{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    return df\n\ntrain = load_df(\"train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"673e7c7367a4a5980560baa125e43376d64cd5ff"},"cell_type":"markdown","source":"### 2.2 Dataset Snapshot\n\nLets view the snapshot of the test dataset. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"9e761c2ef85fd6c9c94d7a155d639d6a5424d209"},"cell_type":"code","source":"print (\"There are \" + str(train.shape[0]) + \" rows and \" + str(train.shape[1]) + \" raw columns in this dataset\")\n\nprint (\"Snapshot: \")\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4d55abc37cefb9a341b3271b1c6f703c347237e"},"cell_type":"markdown","source":"### 2.2 Missing Values Percentage\n\nFrom the snapshot we can observe that there are many missing values in the dataset. Let's plot the missing values percentage for columns having missing values. \n\n> The following graph shows only those columns having missing values, all other columns are fine. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"44dfb5f91100da2e89986be56c0ccab162e089e4"},"cell_type":"code","source":"miss_per = {}\nfor k, v in dict(train.isna().sum(axis=0)).items():\n    if v == 0:\n        continue\n    miss_per[k] = 100 * float(v) / len(train)\n    \nimport operator \nsorted_x = sorted(miss_per.items(), key=operator.itemgetter(1), reverse=True)\nprint (\"There are \" + str(len(miss_per)) + \" columns with missing values\")\n\nkys = [_[0] for _ in sorted_x][::-1]\nvls = [_[1] for _ in sorted_x][::-1]\ntrace1 = go.Bar(y = kys, orientation=\"h\" , x = vls, marker=dict(color=\"#d6a5ff\"))\nlayout = go.Layout(title=\"Missing Values Percentage\", \n                   xaxis=dict(title=\"Missing Percentage\"), \n                   height=400, margin=dict(l=300, r=300))\nfigure = go.Figure(data = [trace1], layout = layout)\niplot(figure)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"129c51c109ffa9659c881549cf0b29568d3d2993"},"cell_type":"markdown","source":"> - So we can observe that there are some columns in the dataset having very large number of missing values. \n\n## 3. Exploration - Univariate Analysis \n\nLets perform the univariate analysis and plot some distributions of variables in the dataset\n\n### 3.1 Device Attributes\n\nLets plot the distribution of device attributes"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"707893d0be746d4536b6252a26de7f8f357ea54c"},"cell_type":"code","source":"device_cols = [\"device_browser\", \"device_deviceCategory\", \"device_operatingSystem\"]\n\ncolors = [\"#d6a5ff\", \"#fca6da\", \"#f4d39c\", \"#a9fcca\"]\ntraces = []\nfor i, col in enumerate(device_cols):\n    t = train[col].value_counts()\n    traces.append(go.Bar(marker=dict(color=colors[i]),orientation=\"h\", y = t.index[:15][::-1], x = t.values[:15][::-1]))\n\nfig = tools.make_subplots(rows=1, cols=3, subplot_titles=[\"Visits: Category\", \"Visits: Browser\",\"Visits: OS\"], print_grid=False)\nfig.append_trace(traces[1], 1, 1)\nfig.append_trace(traces[0], 1, 2)\nfig.append_trace(traces[2], 1, 3)\n\nfig['layout'].update(height=400, showlegend=False, title=\"Visits by Device Attributes\")\niplot(fig)\n\n## convert transaction revenue to float\ntrain[\"totals_transactionRevenue\"] = train[\"totals_transactionRevenue\"].astype('float')\n\ndevice_cols = [\"device_browser\", \"device_deviceCategory\", \"device_operatingSystem\"]\n\nfig = tools.make_subplots(rows=1, cols=3, subplot_titles=[\"Mean Revenue: Category\", \"Mean Revenue: Browser\",\"Mean Revenue: OS\"], print_grid=False)\n\ncolors = [\"red\", \"green\", \"purple\"]\ntrs = []\nfor i, col in enumerate(device_cols):\n    tmp = train.groupby(col).agg({\"totals_transactionRevenue\": \"mean\"}).reset_index().rename(columns={\"totals_transactionRevenue\" : \"Mean Revenue\"})\n    tmp = tmp.dropna().sort_values(\"Mean Revenue\", ascending = False)\n    tr = go.Bar(x = tmp[\"Mean Revenue\"][::-1], orientation=\"h\", marker=dict(opacity=0.5, color=colors[i]), y = tmp[col][::-1])\n    trs.append(tr)\n\nfig.append_trace(trs[1], 1, 1)\nfig.append_trace(trs[0], 1, 2)\nfig.append_trace(trs[2], 1, 3)\nfig['layout'].update(height=400, showlegend=False, title=\"Mean Revenue by Device Attributes\")\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fe8fe3d1a2019687fb4533a4a7f59f559becd992"},"cell_type":"markdown","source":"> - There is a significant difference in visits from mobile and tablets, but mean revenue for both of them is very close.  \n> - Interesting to note that maximum visits are from Chrome browser however maximum revenue is collected from visits throught firefox. \n> - Chrome OS users has generated maximum revenue though maximum visits are from windows and macintosh users  \n\n### 3.2 GeoNetwork Attributes "},{"metadata":{"trusted":true,"_uuid":"2fcf66951ad315244c75b7f5bbf68efcb2b2b23e","_kg_hide-input":true},"cell_type":"code","source":"geo_cols = ['geoNetwork_city', 'geoNetwork_continent','geoNetwork_country',\n            'geoNetwork_metro', 'geoNetwork_networkDomain', 'geoNetwork_region','geoNetwork_subContinent']\ngeo_cols = ['geoNetwork_continent','geoNetwork_subContinent']\n\ncolors = [\"#d6a5ff\", \"#fca6da\"]\nfig = tools.make_subplots(rows=1, cols=2, subplot_titles=[\"Visits : GeoNetwork Continent\", \"Visits : GeoNetwork subContinent\"], print_grid=False)\ntrs = []\nfor i,col in enumerate(geo_cols):\n    t = train[col].value_counts()\n    tr = go.Bar(x = t.index[:20], marker=dict(color=colors[i]), y = t.values[:20])\n    trs.append(tr)\n\nfig.append_trace(trs[0], 1, 1)\nfig.append_trace(trs[1], 1, 2)\nfig['layout'].update(height=400, margin=dict(b=150), showlegend=False)\niplot(fig)\n\n\n\n\ngeo_cols = ['geoNetwork_continent','geoNetwork_subContinent']\nfig = tools.make_subplots(rows=1, cols=2, subplot_titles=[\"Mean Revenue: Continent\", \"Mean Revenue: SubContinent\"], print_grid=False)\n\ncolors = [\"blue\", \"orange\"]\ntrs = []\nfor i, col in enumerate(geo_cols):\n    tmp = train.groupby(col).agg({\"totals_transactionRevenue\": \"mean\"}).reset_index().rename(columns={\"totals_transactionRevenue\" : \"Mean Revenue\"})\n    tmp = tmp.dropna().sort_values(\"Mean Revenue\", ascending = False)\n    tr = go.Bar(y = tmp[\"Mean Revenue\"], orientation=\"v\", marker=dict(opacity=0.5, color=colors[i]), x= tmp[col])\n    trs.append(tr)\n\nfig.append_trace(trs[0], 1, 1)\nfig.append_trace(trs[1], 1, 2)\nfig['layout'].update(height=450, margin=dict(b=200), showlegend=False)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"25e72963720b5dc6db0dfe8390dffec59f0f3dec","_kg_hide-input":true},"cell_type":"code","source":"tmp = train[\"geoNetwork_country\"].value_counts()\n\n# plotly globe credits - https://www.kaggle.com/arthurtok/generation-unemployed-interactive-plotly-visuals\ncolorscale = [[0, 'rgb(102,194,165)'], [0.005, 'rgb(102,194,165)'], \n              [0.01, 'rgb(171,221,164)'], [0.02, 'rgb(230,245,152)'], \n              [0.04, 'rgb(255,255,191)'], [0.05, 'rgb(254,224,139)'], \n              [0.10, 'rgb(253,174,97)'], [0.25, 'rgb(213,62,79)'], [1.0, 'rgb(158,1,66)']]\n\ndata = [ dict(\n        type = 'choropleth',\n        autocolorscale = False,\n        colorscale = colorscale,\n        showscale = True,\n        locations = tmp.index,\n        z = tmp.values,\n        locationmode = 'country names',\n        text = tmp.values,\n        marker = dict(\n            line = dict(color = '#fff', width = 2)) )           ]\n\nlayout = dict(\n    height=500,\n    title = 'Visits by Country',\n    geo = dict(\n        showframe = True,\n        showocean = True,\n        oceancolor = '#222',\n        projection = dict(\n        type = 'orthographic',\n            rotation = dict(\n                    lon = 60,\n                    lat = 10),\n        ),\n        lonaxis =  dict(\n                showgrid = False,\n                gridcolor = 'rgb(102, 102, 102)'\n            ),\n        lataxis = dict(\n                showgrid = False,\n                gridcolor = 'rgb(102, 102, 102)'\n                )\n            ),\n        )\nfig = dict(data=data, layout=layout)\niplot(fig)\n\n\ntmp = train.groupby(\"geoNetwork_country\").agg({\"totals_transactionRevenue\" : \"mean\"}).reset_index()\n\n\n\n# plotly globe credits - https://www.kaggle.com/arthurtok/generation-unemployed-interactive-plotly-visuals\ncolorscale = [[0, 'rgb(102,194,165)'], [0.005, 'rgb(102,194,165)'], \n              [0.01, 'rgb(171,221,164)'], [0.02, 'rgb(230,245,152)'], \n              [0.04, 'rgb(255,255,191)'], [0.05, 'rgb(254,224,139)'], \n              [0.10, 'rgb(253,174,97)'], [0.25, 'rgb(213,62,79)'], [1.0, 'rgb(158,1,66)']]\n\ndata = [ dict(\n        type = 'choropleth',\n        autocolorscale = False,\n        colorscale = colorscale,\n        showscale = True,\n        locations = tmp.geoNetwork_country,\n        z = tmp.totals_transactionRevenue,\n        locationmode = 'country names',\n        text = tmp.totals_transactionRevenue,\n        marker = dict(\n            line = dict(color = '#fff', width = 2)) )           ]\n\nlayout = dict(\n    height=500,\n    title = 'Mean Revenue by Countries',\n    geo = dict(\n        showframe = True,\n        showocean = True,\n        oceancolor = '#222',\n        projection = dict(\n        type = 'orthographic',\n            rotation = dict(\n                    lon = 60,\n                    lat = 10),\n        ),\n        lonaxis =  dict(\n                showgrid = False,\n                gridcolor = 'rgb(102, 102, 102)'\n            ),\n        lataxis = dict(\n                showgrid = False,\n                gridcolor = 'rgb(102, 102, 102)'\n                )\n            ),\n        )\nfig = dict(data=data, layout=layout)\niplot(fig)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f70cf1087d649b881072d60340424e6cbf2e218"},"cell_type":"markdown","source":"### 3.3 Traffic Attributes\n\nLets now plot the traffic attributes"},{"metadata":{"trusted":true,"_uuid":"e7c28dc60b66ba44f74739f08adad051d9590e4e","_kg_hide-input":true},"cell_type":"code","source":"fig = tools.make_subplots(rows=1, cols=2, subplot_titles=[\"TrafficSource Campaign (not-set removed)\", \"TrafficSource Medium\"], print_grid=False)\n\ncolors = [\"#d6a5ff\", \"#fca6da\", \"#f4d39c\", \"#a9fcca\"]\nt1 = train[\"trafficSource_campaign\"].value_counts()\nt2 = train[\"trafficSource_medium\"].value_counts()\ntr1 = go.Bar(x = t1.index, y = t1.values, marker=dict(color=colors[3]))\ntr2 = go.Bar(x = t2.index, y = t2.values, marker=dict(color=colors[2]))\ntr3 = go.Bar(x = t1.index[1:], y = t1.values[1:], marker=dict(color=colors[0]))\ntr4 = go.Bar(x = t2.index[1:], y = t2.values[1:])\n\nfig.append_trace(tr3, 1, 1)\nfig.append_trace(tr2, 1, 2)\nfig['layout'].update(height=400, margin=dict(b=100), showlegend=False)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa5abc7d9e091a3d86a494f48ebfd938ea7ba57e"},"cell_type":"markdown","source":"### 3.4 Channel Grouping"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2ae83984b56f3e49857a5157ca05829d98bbbb0b"},"cell_type":"code","source":"tmp = train[\"channelGrouping\"].value_counts()\ncolors = [\"#8d44fc\", \"#ed95d5\", \"#caadf7\", \"#6161b7\", \"#7e7eba\", \"#babad1\"]\ntrace = go.Pie(labels=tmp.index, values=tmp.values, marker=dict(colors=colors))\nlayout = go.Layout(title=\"Channel Grouping\", height=400)\nfig = go.Figure(data = [trace], layout = layout)\niplot(fig, filename='basic_pie_chart')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36a0751e6fe11161b8a3f189306e5803ddb2f735"},"cell_type":"markdown","source":"### 3.5 Visits by date, month and day"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f764642db712ee0433b7adce08f7fe75cd5c879e"},"cell_type":"code","source":"def _add_date_features(df):\n    df['date'] = df['date'].astype(str)\n    df[\"date\"] = df[\"date\"].apply(lambda x : x[:4] + \"-\" + x[4:6] + \"-\" + x[6:])\n    df[\"date\"] = pd.to_datetime(df[\"date\"])\n    \n    df[\"month\"]   = df['date'].dt.month\n    df[\"day\"]     = df['date'].dt.day\n    df[\"weekday\"] = df['date'].dt.weekday\n    return df \n\ntrain = _add_date_features(train)\n\ntmp = train['date'].value_counts().to_frame().reset_index().sort_values('index')\ntmp = tmp.rename(columns = {\"index\" : \"dateX\", \"date\" : \"visits\"})\n\ntr = go.Scatter(mode=\"lines\", x = tmp[\"dateX\"].astype(str), y = tmp[\"visits\"])\nlayout = go.Layout(title=\"Visits by date\", height=400)\nfig = go.Figure(data = [tr], layout = layout)\niplot(fig)\n\n\ntmp = train.groupby(\"date\").agg({\"totals_transactionRevenue\" : \"mean\"}).reset_index()\ntmp = tmp.rename(columns = {\"date\" : \"dateX\", \"totals_transactionRevenue\" : \"mean_revenue\"})\ntr = go.Scatter(mode=\"lines\", x = tmp[\"dateX\"].astype(str), y = tmp[\"mean_revenue\"])\nlayout = go.Layout(title=\"MonthlyRevenue by date\", height=400)\nfig = go.Figure(data = [tr], layout = layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e43270a70ea838268a8005392bb86e7b13e2a0d1"},"cell_type":"code","source":"fig = tools.make_subplots(rows=1, cols=3, subplot_titles=[\"Visits by Month\", \"Visits by MonthDay\", \"Visits by WeekDay\"], print_grid=False)\ntrs = []\nfor i,col in enumerate([\"month\", \"day\", \"weekday\"]):\n    t = train[col].value_counts()\n    tr = go.Bar(x = t.index, marker=dict(color=colors[i]), y = t.values)\n    trs.append(tr)\n\nfig.append_trace(trs[0], 1, 1)\nfig.append_trace(trs[1], 1, 2)\nfig.append_trace(trs[2], 1, 3)\nfig['layout'].update(height=400, showlegend=False)\niplot(fig)\n\n\n\ntmp1 = train.groupby('month').agg({\"totals_transactionRevenue\" : \"mean\"}).reset_index()\ntmp2 = train.groupby('day').agg({\"totals_transactionRevenue\" : \"mean\"}).reset_index()\ntmp3 = train.groupby('weekday').agg({\"totals_transactionRevenue\" : \"mean\"}).reset_index()\n\nfig = tools.make_subplots(rows=1, cols=3, subplot_titles=[\"MeanRevenue by Month\", \"MeanRevenue by MonthDay\", \"MeanRevenue by WeekDay\"], print_grid=False)\ntr1 = go.Bar(x = tmp1.month, marker=dict(color=\"red\", opacity=0.5), y = tmp1.totals_transactionRevenue)\ntr2 = go.Bar(x = tmp2.day, marker=dict(color=\"orange\", opacity=0.5), y = tmp2.totals_transactionRevenue)\ntr3 = go.Bar(x = tmp3.weekday, marker=dict(color=\"green\", opacity=0.5), y = tmp3.totals_transactionRevenue)\n\nfig.append_trace(tr1, 1, 1)\nfig.append_trace(tr2, 1, 2)\nfig.append_trace(tr3, 1, 3)\nfig['layout'].update(height=400, showlegend=False)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e9284994f822fde33ae679f78310604a80635fd6"},"cell_type":"markdown","source":"### 3.6 Visit Number Frequency"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c1889e56a301f467e68a979d4b189ec7e10e3da1"},"cell_type":"code","source":"vn = train[\"visitNumber\"].value_counts()\ndef vn_bins(x):\n    if x == 1:\n        return \"1\" \n    elif x < 5:\n        return \"2-5\"\n    elif x < 10:\n        return \"5-10\"\n    elif x < 50:\n        return \"10-50\"\n    elif x < 100:\n        return \"50-100\"\n    else:\n        return \"100+\"\n    \nvn = train[\"visitNumber\"].apply(vn_bins).value_counts()\n\ntrace1 = go.Bar(y = vn.index[::-1], orientation=\"h\" , x = vn.values[::-1], marker=dict(color=\"#7af9ad\"))\nlayout = go.Layout(title=\"Visit Numbers Distribution\", \n                   xaxis=dict(title=\"Frequency\"),yaxis=dict(title=\"VisitNumber\") ,\n                   height=400, margin=dict(l=300, r=300))\nfigure = go.Figure(data = [trace1], layout = layout)\niplot(figure)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"02b61a16d6323e34bf8ca1bed937cfc97f9ae872"},"cell_type":"markdown","source":"## 4. Visitor Profile \n\nLets create the visitor profile by aggregating the rows for every customer. \n\n### 4.1 Visitor Profile Snapshot"},{"metadata":{"trusted":true,"_uuid":"46841c2028ea593e5d77538e7d9b6648cd1b5472","_kg_hide-input":true},"cell_type":"code","source":"import warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\n\nagg_dict = {}\nfor col in [\"totals_bounces\", \"totals_hits\", \"totals_newVisits\", \"totals_pageviews\", \"totals_transactionRevenue\"]:\n    train[col] = train[col].astype('float')\n    agg_dict[col] = \"sum\"\ntmp = train.groupby(\"fullVisitorId\").agg(agg_dict).reset_index()\ntmp.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"62a62c5484e808b19c79907c9f1590e95effc1f7"},"cell_type":"markdown","source":"### 4.2 Total Transactions Revenue"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c60b224592c69e15e4e286d83cec1de595f317ab"},"cell_type":"code","source":"non_zero = tmp[tmp[\"totals_transactionRevenue\"] > 0][\"totals_transactionRevenue\"]\nprint (\"There are \" + str(len(non_zero)) + \" visitors in the train dataset having non zero total transaction revenue\")\n\nplt.figure(figsize=(12,6))\nsns.distplot(non_zero)\nplt.title(\"Distribution of Non Zero Total Transactions\");\nplt.xlabel(\"Total Transactions\");","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e54cfc3f5751bb7e26bf46bac570e471809789c9"},"cell_type":"markdown","source":"Lets take the natural log on the transactions"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e0e4b4c6a17803195d4c9f1ebedfddfb670a20a4"},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nsns.distplot(np.log1p(non_zero))\nplt.title(\"Log Distribution of Non Zero Total Transactions\");\nplt.xlabel(\"Log - Total Transactions\");","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30f66ab7332348bf1a6837dd272caf5090be0d8f"},"cell_type":"markdown","source":"### 4.3 Visitor Profile Attributes"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"15d9c375b9672344d4b320ed6cd918b3644c3cb0"},"cell_type":"code","source":"def getbin_hits(x):\n    if x < 5:\n        return \"1-5\"\n    elif x < 10:\n        return \"5-10\"\n    elif x < 30:\n        return \"10-30\"\n    elif x < 50:\n        return \"30-50\"\n    elif x < 100:\n        return \"50-100\"\n    else:\n        return \"100+\"\n\ntmp[\"total_hits_bin\"] = tmp[\"totals_hits\"].apply(getbin_hits)\ntmp[\"totals_bounces_bin\"] = tmp[\"totals_bounces\"].apply(lambda x : str(x) if x <= 5 else \"5+\")\ntmp[\"totals_pageviews_bin\"] = tmp[\"totals_pageviews\"].apply(lambda x : str(x) if x <= 50 else \"50+\")\n\nt1 = tmp[\"total_hits_bin\"].value_counts()\nt2 = tmp[\"totals_bounces_bin\"].value_counts()\nt3 = tmp[\"totals_newVisits\"].value_counts()\nt4 = tmp[\"totals_pageviews_bin\"].value_counts()\n\nfig = tools.make_subplots(rows=2, cols=2, subplot_titles=[\"Total Hits per User\", \"Total Bounces per User\", \n                                                         \"Total NewVistits per User\", \"Total PageViews per User\"], print_grid=False)\n\ntr1 = go.Bar(x = t1.index[:20], y = t1.values[:20])\ntr2 = go.Bar(x = t2.index[:20], y = t2.values[:20])\ntr3 = go.Bar(x = t3.index[:20], y = t3.values[:20])\ntr4 = go.Bar(x = t4.index, y = t4.values)\n\nfig.append_trace(tr1, 1, 1)\nfig.append_trace(tr2, 1, 2)\nfig.append_trace(tr3, 2, 1)\nfig.append_trace(tr4, 2, 2)\n\nfig['layout'].update(height=700, showlegend=False)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"696837e6c7acbda716057da56b614d573da77758"},"cell_type":"markdown","source":"## 5. Baseline Model\n\n### 5.1 PreProcessing\n\nAs the preprocessing step, lets identify which columns can be removed. \n- Drop Columns with constant values  \n- Drop Ids and other non relevant columns  "},{"metadata":{"trusted":true,"_uuid":"76b0e7a0a97524585d551559dea1af85bf772d00"},"cell_type":"code","source":"## find constant columns\nconstant_columns = []\nfor col in train.columns:\n    if len(train[col].value_counts()) == 1:\n        constant_columns.append(col)\n\n## non relevant columns\nnon_relevant = [\"visitNumber\", \"date\", \"fullVisitorId\", \"sessionId\", \"visitId\", \"visitStartTime\"]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b2aecfc3bcb874a2277e91ceaaadf1ad74f3619"},"cell_type":"markdown","source":"Lets now also read the test dataset which will be used to make predictions "},{"metadata":{"trusted":true,"_uuid":"98535f489e0257e863f6faf8791b510e28de9bf7"},"cell_type":"code","source":"test = load_df(\"test.csv\")\ntest = _add_date_features(test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec87f79725ac4981469edf9b7e7432965e9b5f06"},"cell_type":"markdown","source":"### 5.2 Handle Categorical Columns"},{"metadata":{"trusted":true,"_uuid":"d350e82391640587747a5a73464a6bf0dfc46559"},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\ncat_cols = [c for c in train.columns if not c.startswith(\"total\")]\ncat_cols = [c for c in cat_cols if c not in constant_columns + non_relevant]\nfor c in cat_cols:\n\n    le = LabelEncoder()\n    train_vals = list(train[c].values.astype(str))\n    test_vals = list(test[c].values.astype(str))\n    \n    le.fit(train_vals + test_vals)\n    \n    train[c] = le.transform(train_vals)\n    test[c] = le.transform(test_vals)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9426c61d48e2c9bd712a1aeef5a56268ca4cbad3"},"cell_type":"markdown","source":"### 5.3 Handle Numerical Columns "},{"metadata":{"trusted":true,"_uuid":"7a4f3ac52e03400c36d694921e4ac812006cf0c7"},"cell_type":"code","source":"def _normalize_numerical_cols(df, isTrain = True):\n    df[\"totals_hits\"] = df[\"totals_hits\"].astype(float)\n    df[\"totals_hits\"] = (df[\"totals_hits\"] - min(df[\"totals_hits\"])) / (max(df[\"totals_hits\"]) - min(df[\"totals_hits\"]))\n\n    df[\"totals_pageviews\"] = df[\"totals_pageviews\"].astype(float)\n    df[\"totals_pageviews\"] = (df[\"totals_pageviews\"] - min(df[\"totals_pageviews\"])) / (max(df[\"totals_pageviews\"]) - min(df[\"totals_pageviews\"]))\n    \n    if isTrain:\n        df[\"totals_transactionRevenue\"] = df[\"totals_transactionRevenue\"].fillna(0.0)\n    return df \n\ntrain = _normalize_numerical_cols(train)\ntest = _normalize_numerical_cols(test, isTrain = False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a552f922a9d8fe9e1793462949b2cdb66016dcb"},"cell_type":"markdown","source":"### 5.4 Generate Training and Validation Sets"},{"metadata":{"trusted":true,"_uuid":"649ffb06b70f05244123f15fd6ca464b5b6dc7bd"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfeatures = [c for c in train.columns if c not in constant_columns + non_relevant]\nfeatures.remove(\"totals_transactionRevenue\")\ntrain[\"totals_transactionRevenue\"] = np.log1p(train[\"totals_transactionRevenue\"].astype(float))\ntrain_x, val_x, train_y, val_y = train_test_split(train[features], train[\"totals_transactionRevenue\"], test_size=0.25, random_state=20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"684ce11b223b59d41a39bae78cc82ab500c6c61b"},"cell_type":"markdown","source":"### 5.5 Train the baseline lightgbm model"},{"metadata":{"trusted":true,"_uuid":"52f8f055d13b4b07b5a16e680a7a2639be7943a8"},"cell_type":"code","source":"import lightgbm as lgb \n\nlgb_params = {\"objective\" : \"regression\", \"metric\" : \"rmse\",\n              \"num_leaves\" : 36, \"learning_rate\" : 0.05, \"bagging_fraction\" : 0.75, \"feature_fraction\" : 0.6, \"bagging_frequency\" : 7}\n    \nlgb_train = lgb.Dataset(train_x, label=train_y)\nlgb_val = lgb.Dataset(val_x, label=val_y)\nmodel = lgb.train(lgb_params, lgb_train, 300, valid_sets=[lgb_val], early_stopping_rounds=50, verbose_eval=100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"90c4ca77e351d0e3949224cca0fece79d3b048ce"},"cell_type":"markdown","source":"### 5.6 Generate Predictions and Submission"},{"metadata":{"trusted":true,"_uuid":"ba5602982a42f068c98a04001cc2061c542535fd"},"cell_type":"code","source":"preds = model.predict(test[features], num_iteration=model.best_iteration)\ntest[\"PredictedLogRevenue\"] = np.expm1(preds)\nsub_df = test.groupby(\"fullVisitorId\").agg({\"PredictedLogRevenue\" : \"sum\"}).reset_index()\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df[\"PredictedLogRevenue\"] =  sub_df[\"PredictedLogRevenue\"].apply(lambda x : 0.0 if x < 0 else x)\nsub_df.to_csv(\"baseline.csv\", index=False)\nsub_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dbdaaa11f6bb61103b1e98c779ae3c2f23062592"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}