{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Problem Statement:\nThe 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue. As such, marketing teams are challenged to make appropriate investments in promotional strategies. In this topic ,Google Analytics Customer Revenue Prediction we need to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data Wrangling","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport datetime\nimport math\nimport matplotlib.mlab as mlab\nfrom pandas.io.json import json_normalize\nimport matplotlib.pyplot as plt\nimport seaborn as sns\ncolor = sns.color_palette()\nfrom plotly import tools\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\nfrom sklearn import model_selection, preprocessing, metrics\nimport lightgbm as lgb\n\npd.options.mode.chained_assignment = None\npd.options.display.max_columns = 999","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def load_df(csv_path='../input/ga-customer-revenue-prediction/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(\"../input/ga-customer-revenue-prediction/train.csv\", \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = load_df(\"../input/ga-customer-revenue-prediction/train.csv\")\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = load_df(\"../input/ga-customer-revenue-prediction/test.csv\")\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.info(),test.info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.shape,test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can now observe that:\n\ntrain.csv has 903,653 rows and 55 columns.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data Preprocessing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let us look at what all are the numerical variables for both train and test sets","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* Train Set:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"numeric_features_train = train.select_dtypes(include=[np.number])\nnumeric_features_train.columns\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Test Set:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"numeric_features_test = test.select_dtypes(include=[np.number])\n\nnumeric_features_test.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us look at what all are the categorical variables for both train and test sets","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* Train Set\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical_features_train = train.select_dtypes(include=[np.object])\ncategorical_features_train.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Test Set:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical_features_test = test.select_dtypes(include=[np.object])\ncategorical_features_test.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us now explore the missing values in both train and test sets.\n\nIt is observed that there are some columns that contains \"not available in demo dataset\" as constant values predominently.So it is not going to be effective if we use these columns in our model prediction.So we can safely delete these features from both train and test datasets as below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print (\"Before removing constant columns - shape of train & test datasets: \", train.shape,test.shape)\ntrain = train.loc[:, (train != train.iloc[0]).any()]\ntest = test.loc[:, (test != test.iloc[0]).any()]\nprint (\"After Removing Constant Columns - shape of train & test datasets: \", train.shape,test.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Missing values for all categorical features in Bar chart Representation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* Train set","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"total_test = categorical_features_train.isnull().sum().sort_values(ascending=False)\npercent = (categorical_features_train.isnull().sum()/categorical_features_train.isnull().count()).sort_values(ascending=False)*100\nmissing_data = pd.concat([total_test, percent], axis=1,join='outer', keys=['Total Missing Count', ' % of Total Observations'])\nmissing_data.index.name ='Feature'\nmissing_data.head(14)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us visualise the missing categorical features for train set:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_values = categorical_features_train.isnull().sum(axis=0).reset_index()\nmissing_values.columns = ['column_name', 'missing_count']\nmissing_values = missing_values.loc[missing_values['missing_count']>0]\nmissing_values = missing_values.sort_values(by='missing_count')\nind = np.arange(missing_values.shape[0])\nwidth = 0.1\nfig, ax = plt.subplots(figsize=(12,3))\nrects = ax.barh(ind, missing_values.missing_count.values, color='b')\nax.set_yticks(ind)\nax.set_yticklabels(missing_values.column_name.values, rotation='horizontal')\nax.set_xlabel(\"Missing Observations Count\")\nax.set_title(\"Missing Categorical Observations in Train Dataset\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"total_test = categorical_features_test.isnull().sum().sort_values(ascending=False)\npercent = (categorical_features_test.isnull().sum()/categorical_features_test.isnull().count()).sort_values(ascending=False)*100\nmissing_data = pd.concat([total_test, percent], axis=1,join='outer', keys=['Total Missing Count', ' % of Total Observations'])\nmissing_data.index.name ='Feature'\nmissing_data.head(12)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us visualise the missing categorical features test:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_values = categorical_features_test.isnull().sum(axis=0).reset_index()\nmissing_values.columns = ['column_name', 'missing_count']\nmissing_values = missing_values.loc[missing_values['missing_count']>0]\nmissing_values = missing_values.sort_values(by='missing_count')\nind = np.arange(missing_values.shape[0])\nwidth = 0.1\nfig, ax = plt.subplots(figsize=(12,3))\nrects = ax.barh(ind, missing_values.missing_count.values, color='b')\nax.set_yticks(ind)\nax.set_yticklabels(missing_values.column_name.values, rotation='horizontal')\nax.set_xlabel(\"Missing Observations Count\")\nax.set_title(\"Missing Categorical Observations in Test Dataset\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Replace NaN values throughout train dataset\ntrain.replace(to_replace=np.nan, value=0, inplace=True)\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Replace NaN values throughout test dataset\ntest.replace(to_replace=np.nan, value=0, inplace=True)\ntest.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cor=train.corr()\ncor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nsns.heatmap(train.corr(),annot=True,cmap='RdBu_r')\nplt.title(\"Correlation Of Each Numerical Features\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cor1=test.corr()\ncor1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nsns.heatmap(test.corr(),annot=True,cmap='RdBu_r')\nplt.title(\"Correlation Of Each Numerical Features\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Visualization and Exploratory Data Analysis","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Univariate Analysis:\nLets perform univariate analysis on some of the variables in the dataset and plot their distributions to unfold patterns or insights about the data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Geo Network Attributes:\n* geoNetwork_city\n* geoNetwork_continent\n* geoNetwork_country\n* geoNetwork_metro\n* geoNetwork_networkDomain\n* geoNetwork_region\n* geoNetwork_subContinent\nSo among the above geoNetwork attributes let us consider 'geoNetwork_country' attribute and visualise customer revenue","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\ncolorscale = [[0, 'rgb(102,194,165)'], [0.0005, 'rgb(102,194,165)'], \n              [0.01, 'rgb(171,221,164)'], [0.02, 'rgb(230,245,152)'], \n              [0.04, 'rgb(255,255,191)'], [0.05, 'rgb(254,224,139)'], \n              [0.10, 'rgb(253,174,97)'], [0.25, 'rgb(213,62,79)'], [1.0, 'rgb(158,1,66)']]\n\ndata = [ dict(\n        type = 'choropleth',\n        autocolorscale = False,\n        colorscale = colorscale,\n        showscale = True,\n        locations = train[\"geoNetwork.country\"].value_counts().index,\n        locationmode = 'country names',\n        z = train[\"geoNetwork.country\"].value_counts().values,\n        marker = dict(\n            line = dict(color = 'rgb(250,250,225)', width = 1)),\n            colorbar = dict( title = 'Customer Visits ')\n            ) \n       ]\n\nlayout = dict(\n    height=600,\n    title = 'World Wide Customer Visit Distribution',\n    geo = dict(\n        showframe = True,\n        showocean = True,\n        oceancolor = 'rgb(28,107,160)',\n        projection = dict(\n        type = 'orthographic',\n            rotation = dict(\n                    lon = 50,\n                    lat = 10),\n        ),\n        lonaxis =  dict(\n                showgrid = True,\n                gridcolor = 'rgb(12, 102, 102)'\n            ),\n        lataxis = dict(\n                showgrid = True,\n                gridcolor = 'rgb(12, 102, 102)'\n                )\n            ),\n        )\nfig = dict(data=data, layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inferences:\n* On the continent plot, we can see that America has both higher number of counts as well as highest number of counts where the revenue is non-zero\n* Though Asia and Europe has high number of counts, the number of non-zero revenue counts from these continents are comparatively low.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Target Variable Exploration:\n\nSince we are predicting the natural log of sum of all transactions of the user, let us sum up the transaction revenue at user level and take a log and then do a scatter plot","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"totals.transactionRevenue\"] = train[\"totals.transactionRevenue\"].astype('float')\ngdf = train.groupby(\"fullVisitorId\")[\"totals.transactionRevenue\"].sum().reset_index()\n\nplt.figure(figsize=(8,6))\nplt.scatter(range(gdf.shape[0]), np.sort(np.log1p(gdf[\"totals.transactionRevenue\"].values)))\nplt.xlabel('index', fontsize=12)\nplt.ylabel('TransactionRevenue', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This confirms the first two lines of the competition overview:\n* The 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue. As such, marketing teams are challenged to make appropriate investments in promotional strategies.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Checking the ratio:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"nzi = pd.notnull(train[\"totals.transactionRevenue\"]).sum()\nnzr = (gdf[\"totals.transactionRevenue\"]>0).sum()\nprint(\"Number of instances in train set with non-zero revenue : \", nzi, \" and ratio is : \", nzi / train.shape[0])\nprint(\"Number of unique customers with non-zero revenue : \", nzr, \"and the ratio is : \", nzr / gdf.shape[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So the ratio of revenue generating customers to customers with no revenue is in the ratio is 1.0\n\nSince most of the rows have non-zero revenues, in the following plots let us have a look at the count of each category of the variable along with the number of instances where the revenue is not zero","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Number of visitors and common visitors","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique visitors in train set : \",train.fullVisitorId.nunique(), \" out of rows : \",train.shape[0])\nprint(\"Number of unique visitors in test set : \",test.fullVisitorId.nunique(), \" out of rows : \",test.shape[0])\nprint(\"Number of common visitors in train and test set : \",len(set(train.fullVisitorId.unique()).intersection(set(test.fullVisitorId.unique())) ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Device Information:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef horizontal_bar_chart(cnt_srs, color):\n    trace = go.Bar(\n        y=cnt_srs.index[::-1],\n        x=cnt_srs.values[::-1],\n        showlegend=False,\n        orientation = 'h',\n        marker=dict(\n            color=color,\n        ),\n    )\n    return trace\n\n# Device Browser\ncnt_srs = train.groupby('device.browser')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace3 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\n# Device Category\ncnt_srs = train.groupby('device.deviceCategory')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace4 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(71, 58, 131, 0.8)')\ntrace5 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(71, 58, 131, 0.8)')\ntrace6 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(71, 58, 131, 0.8)')\n\n# Operating system\ncnt_srs = train.groupby('device.operatingSystem')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace7 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(246, 78, 139, 0.6)')\ntrace8 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10),'rgba(246, 78, 139, 0.6)')\ntrace9 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10),'rgba(246, 78, 139, 0.6)')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=3, cols=3, vertical_spacing=0.04, \n                          subplot_titles=[\"Device Browser - Count\", \"Device Browser - Non-zero Revenue Count\", \"Device Browser - Mean Revenue\",\n                                          \"Device Category - Count\",  \"Device Category - Non-zero Revenue Count\", \"Device Category - Mean Revenue\", \n                                          \"Device OS - Count\", \"Device OS - Non-zero Revenue Count\", \"Device OS - Mean Revenue\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 1, 3)\nfig.append_trace(trace4, 2, 1)\nfig.append_trace(trace5, 2, 2)\nfig.append_trace(trace6, 2, 3)\nfig.append_trace(trace7, 3, 1)\nfig.append_trace(trace8, 3, 2)\nfig.append_trace(trace9, 3, 3)\n\nfig['layout'].update(height=1000, width=1000, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inferences:\n* Device browser distribution looks similar on both the count and count of non-zero revenue plots\n* On the device category front, desktop seem to have higher percentage of non-zero revenue counts compared to mobile devices\n* In device operating system, though the number of counts is more from windows, the number of counts where revenue is not zero is more for Macintosh.\n* Chrome OS also has higher percentage of non-zero revenue counts\n* On the mobile OS side, iOS has more percentage of non-zero revenue counts compared to Android","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Date Exploration for datetime instance\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import datetime\n\ndef scatter_plot(cnt_srs, color):\n    trace = go.Scatter(\n        x=cnt_srs.index[::-1],\n        y=cnt_srs.values[::-1],\n        showlegend=False,\n        marker=dict(\n            color=color,\n        ),\n    )\n    return trace\n\ntrain['date'] = train['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ncnt_srs = train.groupby('date')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_index()\n#cnt_srs.index = cnt_srs.index.astype('str')\ntrace1 = scatter_plot(cnt_srs[\"count\"], 'red')\ntrace2 = scatter_plot(cnt_srs[\"count of non-zero revenue\"], 'blue')\n\nfig = tools.make_subplots(rows=2, cols=1, vertical_spacing=0.08,\n                          subplot_titles=[\"Date - Count\", \"Date - Non-zero Revenue count\"])\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 2, 1)\nfig['layout'].update(height=800, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Date Plots\")\npy.iplot(fig, filename='date-plots')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inferences:\n* We have data from 1 Aug, 2016 to 31 July, 2017 in our training dataset\n* In Nov 2016, though there is an increase in the count of visitors, there is no increase in non-zero revenue counts during that time period (relative to the mean).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test['date'] = test['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ncnt_srs = test.groupby('date')['fullVisitorId'].size()\n\n\ntrace = scatter_plot(cnt_srs, 'red')\n\nlayout = go.Layout(\n    height=400,\n    width=800,\n    paper_bgcolor='rgb(233,233,233)',\n    title='Dates in Test set'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the test set, we have dates from 2 Aug, 2017 to 30 Apr, 2018. So there are no common dates between train and test set. So it might be a good idea to do time based validation for this dataset.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Traffic Source:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Continent\ncnt_srs = train.groupby('trafficSource.source')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'green')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'green')\ntrace3 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'green')\n\n# Sub-continent\ncnt_srs = train.groupby('trafficSource.medium')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace4 = horizontal_bar_chart(cnt_srs[\"count\"], 'purple')\ntrace5 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"], 'purple')\ntrace6 = horizontal_bar_chart(cnt_srs[\"mean\"], 'purple')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=2, cols=3, vertical_spacing=0.08, horizontal_spacing=0.15, \n                          subplot_titles=[\"Traffic Source - Count\", \"Traffic Source - Non-zero Revenue Count\", \"Traffic Source - Mean Revenue\",\n                                          \"Traffic Source Medium - Count\",  \"Traffic Source Medium - Non-zero Revenue Count\", \"Traffic Source Medium - Mean Revenue\"\n                                          ])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 1, 3)\nfig.append_trace(trace4, 2, 1)\nfig.append_trace(trace5, 2, 2)\nfig.append_trace(trace6, 2, 3)\n\nfig['layout'].update(height=1000, width=1000, paper_bgcolor='rgb(233,233,233)', title=\"Traffic Source Plots\")\npy.iplot(fig, filename='traffic-source-plots')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inferences:\n* In the traffic source plot, though Youtube has high number of counts in the dataset, the number of non-zero revenue counts are very less.\n* Google plex has a high ratio of non-zero revenue count to total count in the traffic source plot.\n* On the traffic source medium, \"referral\" has more number of non-zero revenue count compared to \"organic\" medium.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Visitor Profile:\nNow let us look at the visitor profile variables like number of pageviews by the visitor, number of hits by the visitor and see how they look.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Page views\ncnt_srs = train.groupby('totals.pageviews')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(60), 'cyan')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(60), 'cyan')\ntrace5 = horizontal_bar_chart(cnt_srs[\"mean\"].head(60), 'cyan')\n\n# Hits\ncnt_srs = train.groupby('totals.hits')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", 'mean']\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(60), 'black')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(60), 'black')\ntrace6 = horizontal_bar_chart(cnt_srs[\"mean\"].head(60), 'black')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=2, cols=3, vertical_spacing=0.08, horizontal_spacing=0.15, \n                          subplot_titles=[\"Total Pageviews - Count\", \"Total Pageviews - Non-zero Revenue Count\", \"Total Pageviews - Mean Revenue\",\n                                          \"Total Hits - Count\",  \"Total Hits - Non-zero Revenue Count\", \"Total Hits - Mean Revenue\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace5, 1, 3)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\nfig.append_trace(trace6, 2, 3)\n\nfig['layout'].update(height=1000, width=900, paper_bgcolor='rgb(233,233,233)', title=\"Visitor Profile Plots\")\npy.iplot(fig, filename='visitor-profile-plots')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inferences:\n* Count plot shows decreasing nature i.e. we have a very high total count for less number of hits and page views per visitor transaction and the overall count decreases when the number of hits per visitor transaction increases.\n* On the other hand, we can clearly see that when the number of hits / pageviews per visitor transaction increases, we see that there is a high number of non-zero revenue counts.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def line_plot(train):\n    f, axes = plt.subplots(figsize=(18, 6))\n\n    sns.set(style=\"white\")\n    sns.lineplot( data=train, palette=\"tab10\", linewidth=2.5)\n\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = train.groupby(\"date\")[\"totals.transactionRevenue\"].agg(['count'])\nline_plot(data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inferences:\n* The number of transactions is up in December, likely in-line with Christmas\n* There appears to be no increasing trend from the beginning of the dataset to the end\nSeasonality is apparent","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data = test.groupby(\"date\")[\"fullVisitorId\"].agg(['size'])\nline_plot(data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The test dataset shows a very large spike around December, much larger than any in the training dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\n\nagg_dict = {}\nfor col in [\"totals.bounces\", \"totals.hits\", \"totals.newVisits\", \"totals.pageviews\", \"totals.transactionRevenue\"]:\n    train[col] = train[col].astype('float')\n    agg_dict[col] = \"sum\"\ntmp = train.groupby(\"fullVisitorId\").agg(agg_dict).reset_index()\ntmp.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Total Revenue Distribution:\nLooking at the distribution of total revenue without normalisation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"non_zero = tmp[tmp[\"totals.transactionRevenue\"] > 0][\"totals.transactionRevenue\"]\nprint (\"There are \" + str(len(non_zero)) + \" visitors in the train dataset having non zero total transaction revenue\")\n\nplt.figure(figsize=(10,6))\nsns.distplot(non_zero)\nplt.title(\"Distribution of Non-Zero Total Transactions\");\nplt.xlabel(\"Total Transactions\");","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nsns.distplot(np.log1p(non_zero))\nplt.title(\"Natural Log Distribution of Non Zero Total Transactions\");\nplt.xlabel(\"Natural Log - Total Transactions\");","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Variables not in test but in train : \", set(train.columns).difference(set(test.columns)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## find constant columns\nconstant_columns = []\nfor col in train.columns:\n    if len(train[col].value_counts()) == 1:\n        constant_columns.append(col)\n\n## non relevant columns\nnon_relevant = [\"visitNumber\", \"date\", \"fullVisitorId\", \"sessionId\", \"visitId\", \"visitStartTime\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(constant_columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cols_to_drop = constant_columns + ['sessionId']\ntrain = train.drop(cols_to_drop + [\"trafficSource.campaignCode\"], axis=1)\ntest = test.drop(cols_to_drop, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Building","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Creating development and validation splits based on time to build the model. We can take the last two months as validation sample.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n# Impute 0 for missing target values\ntrain[\"totals.transactionRevenue\"].fillna(0, inplace=True)\ntrain_y = train[\"totals.transactionRevenue\"].values\ntrain_id = train[\"fullVisitorId\"].values\ntest_id = test[\"fullVisitorId\"].values\n# label encode the categorical variables and convert the numerical variables to float\ncat_cols = [\"channelGrouping\", \"device.browser\", \n            \"device.deviceCategory\", \"device.operatingSystem\", \n            \"geoNetwork.city\", \"geoNetwork.continent\", \n            \"geoNetwork.country\", \"geoNetwork.metro\",\n            \"geoNetwork.networkDomain\", \"geoNetwork.region\", \n            \"geoNetwork.subContinent\"]\nfor col in cat_cols:\n    print(col)\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(train[col].values.astype('str')) + list(test[col].values.astype('str')))\n    train[col] = lbl.transform(list(train[col].values.astype('str')))\n    test[col] = lbl.transform(list(test[col].values.astype('str')))\n    num_cols = [\"totals.hits\", \"totals.pageviews\", \"visitNumber\", \"visitStartTime\"]    \nfor col in num_cols:\n    train[col] = train[col].astype(float)\n    test[col] = test[col].astype(float)\n\n# Split the train dataset into development and valid based on time \ndev_df = train[train['date']<=datetime.date(2017,5,31)]\nval_df = train[train['date']>datetime.date(2017,5,31)]\ndev_y = np.log1p(dev_df[\"totals.transactionRevenue\"].values)\nval_y = np.log1p(val_df[\"totals.transactionRevenue\"].values)\ndev_X = dev_df[cat_cols + num_cols] \nval_X = val_df[cat_cols + num_cols] \ntest_X = test[cat_cols + num_cols] ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Light Gradient Boosting(With Hypertuning)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_lgb(train_X, train_y, val_X, val_y, test_X):\n    params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\", \n        \"num_leaves\" : 30,\n        \"min_child_samples\" : 100,\n        \"learning_rate\" : 0.1,\n        \"bagging_fraction\" : 0.7,\n        \"feature_fraction\" : 0.5,\n        \"bagging_frequency\" : 5,\n        \"bagging_seed\" : 2018,\n        \"verbosity\" : -1\n    }\n    \n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n    model = lgb.train(params, lgtrain, 1000, valid_sets=[lgval], early_stopping_rounds=100, verbose_eval=100)\n    \n    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n    pred_val_y = model.predict(val_X, num_iteration=model.best_iteration)\n    return pred_test_y, model, pred_val_y\n\n# Training the model #\npred_test, model, pred_val = run_lgb(dev_X, dev_y, val_X, val_y, test_X)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Computing the evaluation metric on the validation data. So we need to do a sum for all the transactions of the user and then do a log transformation on top. Let us also make the values less than 0 to 0 as transaction revenue can only be 0 or more.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import metrics\npred_val[pred_val<0] = 0\nval_pred_df = pd.DataFrame({\"fullVisitorId\":val_df[\"fullVisitorId\"].values})\nval_pred_df[\"transactionRevenue\"] = val_df[\"totals.transactionRevenue\"].values\nval_pred_df[\"PredictedRevenue\"] = np.expm1(pred_val)\n#print(np.sqrt(metrics.mean_squared_error(np.log1p(val_pred_df[\"transactionRevenue\"].values), np.log1p(val_pred_df[\"PredictedRevenue\"].values))))\nval_pred_df = val_pred_df.groupby(\"fullVisitorId\")[\"transactionRevenue\", \"PredictedRevenue\"].sum().reset_index()\nprint(np.sqrt(metrics.mean_squared_error(np.log1p(val_pred_df[\"transactionRevenue\"].values), np.log1p(val_pred_df[\"PredictedRevenue\"].values))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we are getting a validation score of 1.70 using this method against the public leaderboard score of 1.44. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"baseline_lgb.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here is the output that is the predicted Log Revenue depending on the customer data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Principal Component Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\nfrom sklearn import datasets","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop(train.iloc[:, 7:32], inplace = True, axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop(\"date\",inplace=True,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = StandardScaler().fit_transform(train)\nX","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a PCA that will retain 99% of the variance\npca = PCA(n_components=0.80, whiten=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_pca = pca.fit_transform(X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_pca","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Calculating accuracy for x and xpca","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns; sns.set()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"rng = np.random.RandomState(1)\nX = np.dot(rng.rand(2, 2), rng.randn(2, 200)).T\nplt.scatter(X[:, 0], X[:, 1])\nplt.axis('equal')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import PCA\npca = PCA(n_components=2)\npca.fit(X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(pca.components_)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(pca.explained_variance_)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def draw_vector(v0, v1, ax=None):\n    ax = ax or plt.gca()\n    arrowprops=dict(arrowstyle='->',\n                    linewidth=2,\n                    shrinkA=0, shrinkB=0)\n    ax.annotate('', v1, v0, arrowprops=arrowprops)\n\n# plot data\nplt.scatter(X[:, 0], X[:, 1], alpha=0.2)\nfor length, vector in zip(pca.explained_variance_, pca.components_):\n    v = vector * 3 * np.sqrt(length)\n    draw_vector(pca.mean_, pca.mean_ + v)\nplt.axis('equal');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pca = PCA(n_components=1)\npca.fit(X)\nX_pca = pca.transform(X)\nprint(\"original shape:   \", X.shape)\nprint(\"transformed shape:\", X_pca.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_new = pca.inverse_transform(X_pca)\nplt.scatter(X[:, 0], X[:, 1], alpha=0.2)\nplt.scatter(X_new[:, 0], X_new[:, 1], alpha=0.8)\nplt.axis('equal');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Cluster Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import random \nimport numpy as np \nimport matplotlib.pyplot as plt \nfrom sklearn.cluster import KMeans \nfrom sklearn.datasets.samples_generator import make_blobs \n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nX = train.values[:,1:]\n\nClus_dataSet = StandardScaler().fit_transform(X)\nClus_dataSet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Modeling:Lets apply k-means on our dataset, and take look at cluster labels.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"clusterNum = 3\nk_means = KMeans(init = \"k-means++\", n_clusters = clusterNum, n_init = 12)\nk_means.fit(X)\nlabels = k_means.labels_\nprint(labels)\\","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Insights:We assign the labels to each row in dataframe.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"Clus_km\"] = labels\ntrain.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can easily check the centroid values by averaging the features in each cluster.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby('Clus_km').mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, lets look at the distribution based on their visitId and visitNumber :","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(5,5))\nsns.jointplot(x=train.visitId.values,y=train.visitNumber.values,size=10)\nplt.xlabel('visitId', fontsize=18)\nplt.ylabel('visitNumber', fontsize=16)\n\nplt.show()\nsns.despine","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}