{"cells":[{"metadata":{"_uuid":"bb32cc5d2722e96bb02725aaece54c5509bdacb9"},"cell_type":"markdown","source":"**Objective of the notebook:**\n\nIn this notebook, let us explore the given dataset and make some inferences along the way. Also finally we will build a baseline light gbm model to get started. \n\n**Objective of the competition:**\n\nIn this competition, we a’re challenged to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport matplotlib.pyplot as plt\nimport seaborn as sns\ncolor = sns.color_palette()\n\n%matplotlib inline\n\nfrom plotly import tools\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\nfrom sklearn import model_selection, preprocessing, metrics\nimport lightgbm as lgb\n\npd.options.mode.chained_assignment = None\npd.options.display.max_columns = 999","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6868c4286025cdb431e9f6c51e4ee714d594fca"},"cell_type":"markdown","source":"**About the dataset:**\n\nSimilar to most other kaggle competitions, we are given two datasets\n* train.csv\n* test.csv\n\nEach row in the dataset is one visit to the store. We are predicting the natural log of the sum of all transactions per user. \n    \nThe data fields in the given files are \n* fullVisitorId- A unique identifier for each user of the Google Merchandise Store.\n* channelGrouping - The channel via which the user came to the Store.\n* date - The date on which the user visited the Store.\n* device - The specifications for the device used to access the Store.\n* geoNetwork - This section contains information about the geography of the user.\n* sessionId - A unique identifier for this visit to the store.\n* socialEngagementType - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n* totals - This section contains aggregate values across the session.\n* trafficSource - This section contains information about the Traffic Source from which the session originated.\n* visitId - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\n* visitNumber - The session number for this user. If this is the first session, then this is set to 1.\n* visitStartTime - The timestamp (expressed as POSIX time).\n\nAlso it is important to note that some of the fields are in json format. \n\nThanks to this [wonderful kernel](https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook) by [Julian](https://www.kaggle.com/julian3833), we can convert all the json fields in the file to a flattened csv format which generally use in other competitions."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e08fe7f8ec7419ba476cd52664140ed6ae736a8c"},"cell_type":"code","source":"%%time\ntrain_df = load_df()\ntest_df = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"302c2b9449b6173ed44ad1d27d3547573368afd4"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dcaa5e0839118e5e2d9192f450569309e8c3798d"},"cell_type":"markdown","source":"**Target Variable Exploration:**\n\nSince we are predicting the natural log of sum of all transactions of the user, let us sum up the transaction revenue at user level and take a log and then do a scatter plot."},{"metadata":{"trusted":true,"_uuid":"09744db3d19167bcb29c298ee5b248004bdb01c1"},"cell_type":"code","source":"train_df[\"totals.transactionRevenue\"] = train_df[\"totals.transactionRevenue\"].astype('float')\ngdf = train_df.groupby(\"fullVisitorId\")[\"totals.transactionRevenue\"].sum().reset_index()\n\nplt.figure(figsize=(8,6))\nplt.scatter(range(gdf.shape[0]), np.sort(np.log1p(gdf[\"totals.transactionRevenue\"].values)))\nplt.xlabel('index', fontsize=12)\nplt.ylabel('TransactionRevenue', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"45d9283ef187997aaf8e16acf8dbe47b7fc0a046"},"cell_type":"markdown","source":"Wow, This confirms the first two lines of the competition overview.\n    * The 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue. As such, marketing teams are challenged to make appropriate investments in promotional strategies.\nInfact in this case, the ratio is even less.     "},{"metadata":{"trusted":true,"_uuid":"4a86c2c2d3a8191845bd799e1124c6d94fcc04ff"},"cell_type":"code","source":"nzi = pd.notnull(train_df[\"totals.transactionRevenue\"]).sum()\nnzr = (gdf[\"totals.transactionRevenue\"]>0).sum()\nprint(\"Number of instances in train set with non-zero revenue : \", nzi, \" and ratio is : \", nzi / train_df.shape[0])\nprint(\"Number of unique customers with non-zero revenue : \", nzr, \"and the ratio is : \", nzr / gdf.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"569d56cf56f6134872a47e89c8654aff2ca7562c"},"cell_type":"markdown","source":"So the ratio of revenue generating customers to customers with no revenue is in the ratio os 1.3%\n\nSince most of the rows have non-zero revenues, in the following plots let us have a look at the count of each category of the variable along with the number of instances where the revenue is not zero.\n\n** Number of visitors and common visitors:**\n\nNow let us look at the number of unique visitors in the train and test set and also the number of common visitors."},{"metadata":{"trusted":true,"_uuid":"dcc528a15dda0f853516375bd7e2e73aec09ac75"},"cell_type":"code","source":"print(\"Number of unique visitors in train set : \",train_df.fullVisitorId.nunique(), \" out of rows : \",train_df.shape[0])\nprint(\"Number of unique visitors in train set : \",test_df.fullVisitorId.nunique(), \" out of rows : \",test_df.shape[0])\nprint(\"Number of common visitors in train and test set : \",len(set(train_df.fullVisitorId.unique()).intersection(set(test_df.fullVisitorId.unique())) ))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"961119f995a317c8b937571890cf30ac66bd1969"},"cell_type":"markdown","source":"**Columns with constant values: **\n\nLooks like there are quite a few features with constant value in the train set. Let us get the list of these features."},{"metadata":{"trusted":true,"_uuid":"5eed694270d93fd7c2bc0a16a620fd329d59c2c1"},"cell_type":"code","source":"[c for c in train_df.columns if train_df[c].nunique()==1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66245f8c2294398511b88d68538578a962cbdf07"},"cell_type":"markdown","source":"They are quite a few. Since the values are constant, we can just drop them from our feature list and save some memory and time in our modeling process. \n\n**Device Information:**"},{"metadata":{"trusted":true,"_uuid":"c57e7e05554e2e19a694a934ffcef530baef47f4","_kg_hide-input":false},"cell_type":"code","source":"def horizontal_bar_chart(cnt_srs, color):\n    trace = go.Bar(\n        y=cnt_srs.index[::-1],\n        x=cnt_srs.values[::-1],\n        showlegend=False,\n        orientation = 'h',\n        marker=dict(\n            color=color,\n        ),\n    )\n    return trace\n\n# Device Browser\ncnt_srs = train_df.groupby('device.browser')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\n# Device Category\ncnt_srs = train_df.groupby('device.deviceCategory')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(71, 58, 131, 0.8)')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(71, 58, 131, 0.8)')\n\n# Operating system\ncnt_srs = train_df.groupby('device.operatingSystem')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace5 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(246, 78, 139, 0.6)')\ntrace6 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10),'rgba(246, 78, 139, 0.6)')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=3, cols=2, vertical_spacing=0.04, \n                          subplot_titles=[\"Device Browser - Count\", \"Device Browser - Non-zero Revenue Count\",\n                                          \"Device Category - Count\",  \"Device Category - Non-zero Revenue Count\",\n                                          \"Device Operating System - Count\", \"Device Operating System - Non-zero Revenue Count\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\nfig.append_trace(trace5, 3, 1)\nfig.append_trace(trace6, 3, 2)\n\nfig['layout'].update(height=1200, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"832da86bcdc9fd153d4142f4bd5430c37915e271"},"cell_type":"markdown","source":"Inferences:\n* Device browser distribution looks similar on both the count and count of non-zero revenue plots\n* On the device category front, desktop seem to have higher percentage of non-zero revenue counts compared to mobile devices.\n* In device operating system, though the number of counts is more from windows, the number of counts where revenue is not zero is more for Macintosh.\n* Chrome OS also has higher percentage of non-zero revenue counts\n* On the mobile OS side, iOS has more percentage of non-zero revenue counts compared to Android \n\n**Date Exploration:**"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"59eb3e903949d8345fd8747aac85eab66c0f42cd"},"cell_type":"code","source":"import datetime\n\ndef scatter_plot(cnt_srs, color):\n    trace = go.Scatter(\n        x=cnt_srs.index[::-1],\n        y=cnt_srs.values[::-1],\n        showlegend=False,\n        marker=dict(\n            color=color,\n        ),\n    )\n    return trace\n\ntrain_df['date'] = train_df['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ncnt_srs = train_df.groupby('date')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_index()\n#cnt_srs.index = cnt_srs.index.astype('str')\ntrace1 = scatter_plot(cnt_srs[\"count\"], 'red')\ntrace2 = scatter_plot(cnt_srs[\"count of non-zero revenue\"], 'blue')\n\nfig = tools.make_subplots(rows=2, cols=1, vertical_spacing=0.08,\n                          subplot_titles=[\"Date - Count\", \"Date - Non-zero Revenue count\"])\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 2, 1)\nfig['layout'].update(height=800, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Date Plots\")\npy.iplot(fig, filename='date-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f35771ef1b28f298d1a266d24e8d2a77626ccf0"},"cell_type":"markdown","source":"Inferences:\n* We have data from 1 Aug, 2016 to 31 July, 2017 in our training dataset\n* In Nov 2016, though there is an increase in the count of visitors, there is no increase in non-zero revenue counts during that time period."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"08f4a090a5928029e4b47b660afd71d952002146"},"cell_type":"code","source":"test_df['date'] = test_df['date'].apply(lambda x: datetime.date(int(str(x)[:4]), int(str(x)[4:6]), int(str(x)[6:])))\ncnt_srs = test_df.groupby('date')['fullVisitorId'].size()\n\n\ntrace = scatter_plot(cnt_srs, 'red')\n\nlayout = go.Layout(\n    height=400,\n    width=800,\n    paper_bgcolor='rgb(233,233,233)',\n    title='Dates in Test set'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f98518587981622a86b31c3e3ce7ab70a2a349e"},"cell_type":"markdown","source":"In the test set, we have dates from 2 Aug, 2017 to 30 Apr, 2018. So there are no common dates between train and test set. So it might be a good idea to do time based validation for this dataset.\n\n**Geographic Information:**"},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"fcb64598e3f9d0931e140f56c70d4dd5dc6f7c75"},"cell_type":"code","source":"# Continent\ncnt_srs = train_df.groupby('geoNetwork.continent')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(58, 71, 80, 0.6)')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(58, 71, 80, 0.6)')\n\n# Sub-continent\ncnt_srs = train_df.groupby('geoNetwork.subContinent')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"], 'orange')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"], 'orange')\n\n# Network domain\ncnt_srs = train_df.groupby('geoNetwork.networkDomain')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace5 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'blue')\ntrace6 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'blue')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=3, cols=2, vertical_spacing=0.08, horizontal_spacing=0.15, \n                          subplot_titles=[\"Continent - Count\", \"Continent - Non-zero Revenue Count\",\n                                          \"Sub Continent - Count\",  \"Sub Continent - Non-zero Revenue Count\",\n                                          \"Network Domain - Count\", \"Network Domain - Non-zero Revenue Count\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\nfig.append_trace(trace5, 3, 1)\nfig.append_trace(trace6, 3, 2)\n\nfig['layout'].update(height=1200, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Geography Plots\")\npy.iplot(fig, filename='geo-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"acba2fcee5fe10d5985a7daf9cc88aaa7a55f4b0"},"cell_type":"markdown","source":"Inferences:\n* On the continent plot, we can see that America has both higher number of counts as well as highest number of counts where the revenue is non-zero\n* Though Asia and Europe has high number of counts, the number of non-zero revenue counts from these continents are comparatively low. \n* We can infer the first two points from the sub-continents plot too.\n* If the network domain is \"unknown.unknown\" rather than \"(not set)\", then the number of counts with non-zero revenue tend to be lower. \n\n**Traffic Source:**\n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"be180a4dbb76e68b449487a6dbc214a7fac35f61","scrolled":true},"cell_type":"code","source":"# Continent\ncnt_srs = train_df.groupby('trafficSource.source')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'green')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'green')\n\n# Sub-continent\ncnt_srs = train_df.groupby('trafficSource.medium')['totals.transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"], 'purple')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"], 'purple')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=2, cols=2, vertical_spacing=0.08, horizontal_spacing=0.15, \n                          subplot_titles=[\"Traffic Source - Count\", \"Traffic Source - Non-zero Revenue Count\",\n                                          \"Traffic Source Medium - Count\",  \"Traffic Source Medium - Non-zero Revenue Count\",\n                                          ])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\n\nfig['layout'].update(height=1000, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Traffic Source Plots\")\npy.iplot(fig, filename='traffic-source-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f36fe7649aba0b26a3026dd0fb6ec8e8b9b69168"},"cell_type":"markdown","source":"Inferences:\n* In the traffic source plot, though Youtube has high number of counts in the dataset, the number of non-zero revenue counts are very less. \n* Google plex has a high ratio of non-zero revenue count to total count in the traffic source plot. \n* On the traffic source medium, \"referral\" has more number of non-zero revenue count compared to \"organic\" medium.\n\n**Visitor Profile:**\n\nNow let us look at the visitor profile variables like number of pageviews by the visitor, number of hits by the visitor and see how they look."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6b4042af5aa1c3b5aa79ba6d60d4af3554e510e7"},"cell_type":"code","source":"\n# Page views\ncnt_srs = train_df.groupby('totals.pageviews')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(60), 'cyan')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(60), 'cyan')\ntrace5 = horizontal_bar_chart(cnt_srs[\"mean\"].head(60), 'cyan')\n\n# Hits\ncnt_srs = train_df.groupby('totals.hits')['totals.transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", 'mean']\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(60), 'black')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(60), 'black')\ntrace6 = horizontal_bar_chart(cnt_srs[\"mean\"].head(60), 'black')\n\n# Creating two subplots\nfig = tools.make_subplots(rows=2, cols=3, vertical_spacing=0.08, horizontal_spacing=0.15, \n                          subplot_titles=[\"Total Pageviews - Count\", \"Total Pageviews - Non-zero Revenue Count\", \"Total Pageviews - Mean Revenue\",\n                                          \"Total Hits - Count\",  \"Total Hits - Non-zero Revenue Count\", \"Total Hits - Mean Revenue\"])\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace5, 1, 3)\nfig.append_trace(trace3, 2, 1)\nfig.append_trace(trace4, 2, 2)\nfig.append_trace(trace6, 2, 3)\n\nfig['layout'].update(height=1200, width=900, paper_bgcolor='rgb(233,233,233)', title=\"Visitor Profile Plots\")\npy.iplot(fig, filename='visitor-profile-plots')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa7d921a36f47ed34e55fc566f9781e625432db0"},"cell_type":"markdown","source":"Inferences:\n* Both these variables look very predictive\n* Count plot shows decreasing nature i.e. we have  a very high total count for less number of hits and page views per visitor transaction and the overall count decreases when the number of hits per visitor transaction increases.\n* On the other hand, we can clearly see that when the number of hits / pageviews per visitor transaction increases, we see that there is a high number of non-zero revenue counts. \n\n**Baseline Model:**\n\nNow let us build a baseline model on this dataset. Before we start building models, let us look at the variable names which are there in train dataset and not in test dataset. "},{"metadata":{"trusted":true,"_uuid":"d008bbab982f36a922928f1dbe079202c380df33"},"cell_type":"code","source":"print(\"Variables not in test but in train : \", set(train_df.columns).difference(set(test_df.columns)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0235902dbb12f2ac27c2c0442ce6b6ab50e9e61"},"cell_type":"markdown","source":"So apart from target variable, there is one more variable \"trafficSource.campaignCode\" not present in test dataset. So we need to remove this variable while building models. Also we can drop the constant variables which we got earlier.\n\nAlso we can remove the \"sessionId\" as it is a unique identifier of the visit."},{"metadata":{"trusted":true,"_uuid":"13d15eed12437ae974f3ec03af6a2b6fba75ec4c"},"cell_type":"code","source":"cols_to_drop = ['socialEngagementType',\n 'device.browserSize',\n 'device.browserVersion',\n 'device.flashVersion',\n 'device.language',\n 'device.mobileDeviceBranding',\n 'device.mobileDeviceInfo',\n 'device.mobileDeviceMarketingName',\n 'device.mobileDeviceModel',\n 'device.mobileInputSelector',\n 'device.operatingSystemVersion',\n 'device.screenColors',\n 'device.screenResolution',\n 'geoNetwork.cityId',\n 'geoNetwork.latitude',\n 'geoNetwork.longitude',\n 'geoNetwork.networkLocation',\n 'totals.bounces',\n 'totals.newVisits',\n 'totals.visits',\n 'trafficSource.adwordsClickInfo.criteriaParameters',\n 'trafficSource.adwordsClickInfo.isVideoAd',\n 'trafficSource.isTrueDirect',\n 'sessionId']\n\ntrain_df = train_df.drop(cols_to_drop + [\"trafficSource.campaignCode\"], axis=1)\ntest_df = test_df.drop(cols_to_drop, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f5ae722a5beceacaf32f52a6691a57f05884be0"},"cell_type":"markdown","source":"Now let us create development and validation splits based on time to build the model. We can take the last two months as validation sample."},{"metadata":{"trusted":true,"_uuid":"1b4cc2fa9b1580c9d8a7ad43b5f75657e150fba3"},"cell_type":"code","source":"# Impute 0 for missing target values\ntrain_df[\"totals.transactionRevenue\"].fillna(0, inplace=True)\ntrain_y = train_df[\"totals.transactionRevenue\"].values\ntrain_id = train_df[\"fullVisitorId\"].values\ntest_id = test_df[\"fullVisitorId\"].values\n\n\n# label encode the categorical variables and convert the numerical variables to float\ncat_cols = [\"channelGrouping\", \"device.browser\", \"device.deviceCategory\", \"device.operatingSystem\", \n            \"geoNetwork.city\", \"geoNetwork.continent\", \"geoNetwork.country\", \"geoNetwork.metro\",\n            \"geoNetwork.networkDomain\", \"geoNetwork.region\", \"geoNetwork.subContinent\", \"trafficSource.adContent\", \n            \"trafficSource.adwordsClickInfo.adNetworkType\", \"trafficSource.adwordsClickInfo.gclId\", \n            \"trafficSource.adwordsClickInfo.page\", \"trafficSource.adwordsClickInfo.slot\", \"trafficSource.campaign\",\n            \"trafficSource.keyword\", \"trafficSource.medium\", \"trafficSource.referralPath\", \"trafficSource.source\"]\nfor col in cat_cols:\n    print(col)\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(train_df[col].values.astype('str')) + list(test_df[col].values.astype('str')))\n    train_df[col] = lbl.transform(list(train_df[col].values.astype('str')))\n    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))\n\n\nnum_cols = [\"totals.hits\", \"totals.pageviews\", \"visitNumber\", \"visitStartTime\"]    \nfor col in num_cols:\n    train_df[col] = train_df[col].astype(float)\n    test_df[col] = test_df[col].astype(float)\n\n# Split the train dataset into development and valid based on time \ndev_df = train_df[train_df['date']<=datetime.date(2017,5,31)]\nval_df = train_df[train_df['date']>datetime.date(2017,5,31)]\ndev_y = np.log1p(dev_df[\"totals.transactionRevenue\"].values)\nval_y = np.log1p(val_df[\"totals.transactionRevenue\"].values)\n\ndev_X = dev_df[cat_cols + num_cols] \nval_X = val_df[cat_cols + num_cols] \ntest_X = test_df[cat_cols + num_cols] ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"3de965f248a0bf1a536d9230ef0f3cb9789b0763"},"cell_type":"code","source":"# custom function to run light gbm model\ndef run_lgb(train_X, train_y, val_X, val_y, test_X):\n    params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\", \n        \"num_leaves\" : 55,\n        \"min_child_samples\" : 100,\n        \"learning_rate\" : 0.03,\n        \"bagging_fraction\" : 0.78, # 78,\n        \"feature_fraction\" : 0.78, # 58,78\n        \"bagging_frequency\" : 9,\n        \"bagging_seed\" : 42,\n        \"verbosity\" : -1,\n        \"reg_alpha\" : 0.48,\n        \"reg_lambda\" : 0.48 # 0.48,\n    }\n\n    \n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n#     cv_lgb = lgb.cv(params, lgtrain, num_boost_round=700, nfold=3, \n#                     verbose_eval=20, early_stopping_rounds=40)\n    model = lgb.train(params, lgtrain, 7000, valid_sets=[lgval], early_stopping_rounds=100, verbose_eval=100)\n    \n    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n    pred_val_y = model.predict(val_X, num_iteration=model.best_iteration)\n    return pred_test_y, model, pred_val_y\n\n# Training the model #\npred_test, model, pred_val = run_lgb(dev_X, dev_y, val_X, val_y, test_X)\n\n# Training until validation scores don't improve for 100 rounds.\n# [100]\tvalid_0's rmse: 1.69974\n# [200]\tvalid_0's rmse: 1.68763\n# [300]\tvalid_0's rmse: 1.68567\n# [400]\tvalid_0's rmse: 1.68585\n# Early stopping, best iteration is:\n# [316]\tvalid_0's rmse: 1.68553","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"574dae973bc73351f6325db2a596a008fc8f8338"},"cell_type":"markdown","source":"There are some negative values in the predictions. So let us make them 0. Also let us sum up the prediction of multiple instances of a vistor to get one value per visitor. Finally let us take log value of these predictions. "},{"metadata":{"trusted":true,"_uuid":"b8d88c9e72313e4e4aa2551455d8058e943bb0ff"},"cell_type":"code","source":"from sklearn import metrics\npred_val[pred_val<0] = 0\nval_pred_df = pd.DataFrame({\"fullVisitorId\":val_df[\"fullVisitorId\"].values})\nval_pred_df[\"transactionRevenue\"] = val_df[\"totals.transactionRevenue\"].values\nval_pred_df[\"PredictedRevenue\"] = np.expm1(pred_val)\n#print(np.sqrt(metrics.mean_squared_error(np.log1p(val_pred_df[\"transactionRevenue\"].values), np.log1p(val_pred_df[\"PredictedRevenue\"].values))))\nval_pred_df = val_pred_df.groupby(\"fullVisitorId\")[\"transactionRevenue\", \"PredictedRevenue\"].sum().reset_index()\nprint(np.sqrt(metrics.mean_squared_error(np.log1p(val_pred_df[\"transactionRevenue\"].values), np.log1p(val_pred_df[\"PredictedRevenue\"].values))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a1b1b6dd5b277d59f8a4a06752c23bb8bd7a5ea"},"cell_type":"code","source":"sub_df = pd.DataFrame({\"fullVisitorId\":test_id})\npred_test[pred_test<0] = 0\nsub_df[\"PredictedLogRevenue\"] = np.expm1(pred_test)\nsub_df = sub_df.groupby(\"fullVisitorId\")[\"PredictedLogRevenue\"].sum().reset_index()\nsub_df.columns = [\"fullVisitorId\", \"PredictedLogRevenue\"]\nsub_df[\"PredictedLogRevenue\"] = np.log1p(sub_df[\"PredictedLogRevenue\"])\nsub_df.to_csv(\"baseline_lgb.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"34da0a2ac754fb22062cd99f44f2422da7b35450"},"cell_type":"code","source":"sub_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4cbf691d8f8ce6bebe84b17f729cf7c7a0ad8c9b"},"cell_type":"markdown","source":"This model with a **valid score of 1.43** \n\n**Feature Importance:**\n\nNow let us have a look at the important features of the light gbm model."},{"metadata":{"trusted":true,"_uuid":"7a0da4195ac3e9683c29579b7700a21eb58fdcb8"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,18))\nlgb.plot_importance(model, max_num_features=50, height=0.8, ax=ax)\nax.grid(False)\nplt.title(\"LightGBM - Feature Importance\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0458610aa52e885eaeaf83db91312d379424d15"},"cell_type":"markdown","source":"\"totals.pageviews\" turn out to be the most important feature followed by \"visitStartTime\". \"totals.hits\" and \"visitNumber\" are next in line. "},{"metadata":{"_uuid":"e2ed97686a372b89e874864f271765cdcbe9129b"},"cell_type":"markdown","source":"**More to come. Stay tuned.!**"},{"metadata":{"trusted":true,"_uuid":"c0e1caa5f3b0899107a951f4f0c0137888c0219f"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}