{"cells":[{"metadata":{"_uuid":"91d5dd6f-0d28-4f68-95d7-9db251e01579","_cell_guid":"2912c8bb-1ce8-4f9e-80c4-490aee0f7fd6","trusted":true},"cell_type":"markdown","source":"# Google Analytics Customer Revenue Prediction"},{"metadata":{"_uuid":"df44478d-8ef4-411c-934f-1b196a6e5498","_cell_guid":"7baaa1bd-36da-4c58-a9e1-b01b2d757b7e","trusted":true},"cell_type":"markdown","source":"## 0. Project Description and Methodology:\nThe 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue. As such, marketing teams are challenged to make appropriate investments in promotional strategies.\n\nIn this project, we'll analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. Hopefully, the outcome will be more actionable operational changes and a better use of marketing budgets for those companies who choose to use data analysis on top of GA data.\n\n<a href=\"https://shop.googlemerchandisestore.com/\">\n<img src=\"https://shop.googlemerchandisestore.com/store/20160512512/assets/items/images/GGOEGBJQ125099.jpg\" style=\"width:300px;height:300px;\" alt=\"Google Tote bage\">\n</a>"},{"metadata":{"_uuid":"092498b8-dfc8-4737-b052-3c27855d8f43","_cell_guid":"a529c59d-9de4-47df-9aa6-aa728f504ade","trusted":true},"cell_type":"markdown","source":"In this project, we'll utilize the process of we learned in class as a guideline to go through the analysis:\n\n* [**<b>1. Business understanding</b>**](#1)\n    - In this part, we'll conduct some peripheral research to get a more comprehensive understanding of the whole business model. \n* [**<b>2. Data understanding</b>**](#2)\n    - Dig into the available dataset and try to comprehent the business and extract the data which is needed. Handling missing values.\n* [**<b>3. Data preparation</b>**](#3)\n    - Understand the dependencies between attributes. Analyzing the target variables. Transforming data formats to standard data format.\n* [**<b>4. Modeling</b>**](#4)\n    - Selecting more accurate classfier or regression engine based on the charactristic any of them have.\n    - Train models that are pertinent to the dataset."},{"metadata":{"_uuid":"f07ce3a3-2e5d-4c70-b698-4fba5a613a13","_cell_guid":"a2eb0573-cbe1-4233-8410-cdf6f55799fb","trusted":true},"cell_type":"markdown","source":"# 1. Business Understanding: <a id=\"1\"></a>"},{"metadata":{"_uuid":"1b3a5ac5-afe2-44dd-afd3-4d85d91e5410","_cell_guid":"97ae760d-0624-4418-8c9f-219c296d2b90","trusted":true},"cell_type":"markdown","source":"Google Merchandise Store is official store of Google that contains merchandises that are produced by the company. The company is currently looking for new and effective ways of marketing this website to make customers aware about their merchandises and connect with them appropriately. For this, a Digital Media Marketing Plan must be developed for the company to achieve this agenda and to effectively develop integrated marketing communication strategy for the company."},{"metadata":{"_uuid":"9c50c7e0-5a1a-425f-8549-dd5011b368d3","_cell_guid":"43edd0b4-f61b-4c76-abdb-e61992be16d0","trusted":true},"cell_type":"markdown","source":"# 2. Data understanding<a id=\"2\"></a>"},{"metadata":{"_uuid":"6949c6a7-0db1-4b15-b694-9a91ac4d6e85","_cell_guid":"6603775e-3787-4dc7-923b-27cd0c8877bd","trusted":true},"cell_type":"markdown","source":"* **<b>A. IMPORTS</b>**\n\nImporting necessary packages and libraries."},{"metadata":{"_uuid":"bb77e1a3-2306-4908-9c76-7888142e33a8","_cell_guid":"6918d13c-03e7-4421-a1f4-3126be9e8be0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport json\nimport matplotlib.pyplot as plt\nimport datetime as datetime\nfrom datetime import timedelta, date\nimport seaborn as sns\nimport matplotlib.cm as CM\nimport collections\nimport lightgbm as lgb\nfrom sklearn import preprocessing\nfrom sklearn.tree import DecisionTreeRegressor , plot_tree\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.model_selection import GridSearchCV, train_test_split\nimport warnings\nwarnings.filterwarnings('ignore')\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14a39aac-ecf6-497d-8f01-c5c9cefb3694","_cell_guid":"bf61a707-4745-42d4-837a-5e097acf4582","trusted":true},"cell_type":"markdown","source":"* **B. READING DATA**\n\nFirst of all, get a sense of the whole dataset by reading the description.\n\nReading data and caughting a glimpse of what data it is."},{"metadata":{"_uuid":"b5ffc278-fd40-4ce7-862f-d03242dd40f9","_cell_guid":"7e0dd64e-ca51-4748-9f58-999f5a394fb8","trusted":true},"cell_type":"code","source":"path = \"../input/ga-customer-revenue-prediction/train.csv\"\ntrain_data = pd.read_csv(path)\ntrain_data.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61d82a39-bb05-4d16-b182-79705c6e6b19","_cell_guid":"bb5e0c2a-32a3-490d-8c00-ea5e453f7eaa","trusted":true},"cell_type":"markdown","source":"By using the shape attribute of the pandas dataframe, we could see the train_data has a dimension of 903653 * 12."},{"metadata":{"_uuid":"13e8e09b-5bd1-44e9-b943-5f3f77e86dd1","_cell_guid":"ea5d3550-55d7-47e1-917c-00ed3f2b0e07","trusted":true},"cell_type":"code","source":"print (train_data.shape)\nprint(\"\\n\")\nprint(list(train_data.columns.values))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fbbc34b3-1064-47a1-a3a0-6b57c96ce3dd","_cell_guid":"d2c1ab52-e055-4f7e-81bf-96c3a15154fa","trusted":true},"cell_type":"markdown","source":"* **C. FEATURES DESCRIPTION**\n\nReferring back to Data description to further understanding the features.\n\n*     channelGrouping - The channel via which the user came to the Store.\n*     date - The date on which the user visited the Store.\n*     device - The specifications for the device used to access the Store.\n*     fullVisitorId- A unique identifier for each user of the Google Merchandise Store.\n*     geoNetwork - This section contains information about the geography of the user.\n*     sessionId - A unique identifier for this visit to the store.\n*     socialEngagementType - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n*     totals - This section contains aggregate values across the session.\n*     trafficSource - This section contains information about the Traffic Source from which the session originated.\n*     visitId - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\n*     visitNumber - The session number for this user. If this is the first session, then this is set to 1.\n*     visitStartTime - The timestamp (expressed as POSIX time)."},{"metadata":{"_uuid":"d1330a4f-6d8b-47fd-bee9-3774df5f07c6","_cell_guid":"fb3f7139-8a20-4def-b9f5-6632b27ef957","trusted":true},"cell_type":"markdown","source":"* **D. INDIVIDUAL FEATURE UNDERSTANDING**\n\nIn this section, we're going to analyze these features separately to get a deeper understanding."},{"metadata":{"_uuid":"44e7084c-07e7-496b-a719-42316d3bcd0e","_cell_guid":"6c0f9dfb-bacd-40b6-a1e7-8c0fd45b09ff","trusted":true},"cell_type":"markdown","source":"**- D1. channelGrouping**\n\nAs decribed in the decription, the channelGrouping feature indicates the different channels which lead the customers to visit the GStore, it is quite an important feature to let us understand more precisely where the customers are from."},{"metadata":{"_uuid":"c2a9d01b-bd3a-4572-acd3-9299c4e1ddf1","_cell_guid":"2c83aec2-90d8-4431-a31d-d4763b24d10c","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"tmp = train_data.channelGrouping.value_counts()\nlabels = tmp.index\nsizes = tmp.values\n\nfig1, ax1 = plt.subplots(figsize=(8,8))\nax1.pie(sizes, labels=labels, autopct='%1.1f%%', startangle=90, textprops=dict(color=\"w\"))\nax1.legend(labels, loc=\"center right\",bbox_to_anchor=(0.8, 0, 0.5, 1))\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2bb7a634-9baf-4723-a1f3-321b2400d8e4","_cell_guid":"6b72beb0-9b32-4611-83d1-66137c51a1f5","trusted":true},"cell_type":"code","source":"snstemp = pd.DataFrame(train_data.channelGrouping.value_counts())\nsns.barplot(y = snstemp.index, x = snstemp.iloc[:,0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5dcca7b1-4271-43f6-8421-a7c88aedf1c0","_cell_guid":"fa0010db-42fa-495b-8558-4851f7a58268","trusted":true},"cell_type":"markdown","source":"We can see a general distribution of this feature."},{"metadata":{"_uuid":"65514682-8989-4310-902f-e3cf71f229ff","_cell_guid":"fe227859-c067-4cf4-9388-c9ec1fd8ec5c","trusted":true},"cell_type":"markdown","source":"**- D2. date & visitStartTime **\n\nThe two varialbes, date and visitStartTime, are related to time and can be used in time dependent analyzes specially TimeSeries.\n\nFirst of all, we can use the first row to have a glance of the variables."},{"metadata":{"_uuid":"1e78a463-0a91-4241-b07d-9f61ddc635da","_cell_guid":"3b646220-13fd-4db0-b11b-bce7a6f70047","trusted":true},"cell_type":"code","source":"print(\"date :{}, visitStartTime:{}\".format(train_data.head(1).date[0],train_data.head(1).visitStartTime[0]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"70c2948d-18b7-428a-8044-e87311bf96e3","_cell_guid":"9611f197-d2cd-48e6-821f-b1ce551fbf19","trusted":true},"cell_type":"markdown","source":"Thus, we could see that date is stored in String and should be converted to pandas datetime format. While, visitStartTime is stored in epoch unix format and should be converted to pandas datetime format."},{"metadata":{"_uuid":"e78b418e-909b-4ece-9a9f-361dda731746","_cell_guid":"4d076569-22eb-441e-9eb0-e5f20f2475e1","trusted":true},"cell_type":"code","source":"train_data[\"date\"] = pd.to_datetime(train_data[\"date\"],format=\"%Y%m%d\")\ntrain_data[\"visitStartTime\"] = pd.to_datetime(train_data[\"visitStartTime\"],unit='s')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"471ff8ce-8412-46ef-ae46-21f69e2c1811","_cell_guid":"5d7cc016-b9d1-4f56-983c-a70bb23d85c7","trusted":true},"cell_type":"code","source":"train_data.head()[[\"date\",\"visitStartTime\"]]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aec2a559-b188-47da-98cb-de462d7a9a3b","_cell_guid":"7c5c2274-eb59-4d45-835c-9c3786b7a283","trusted":true},"cell_type":"markdown","source":"Through such conversion, we could get a more intuitive understanding of the visit time related to every row."},{"metadata":{"_uuid":"ada21a59-847a-4cba-9d91-6bffcf398d57","_cell_guid":"b2dcea24-88ed-4fa7-85dc-b51b197e6be2","trusted":true},"cell_type":"markdown","source":"** - D3. device **\n\nThe device variable is stored in json format. We need to extract its fields and analyze them by utilizing json library to deserializing json values into a pandas dataframe."},{"metadata":{"_uuid":"82ff508e-a55d-4cac-8b75-f025ddb54c68","_cell_guid":"945340b9-7ee8-4be1-8a87-42ef1f58b291","trusted":true},"cell_type":"code","source":"tmp_device_df = pd.DataFrame(train_data.device.apply(json.loads).tolist())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b366cc4-4acd-4c38-9e26-2f7c0ed5af08","_cell_guid":"999b32af-98bb-41ba-8c26-56326ca58f6c","trusted":true},"cell_type":"code","source":"# replace the JSON's \"not available in demo dataset\" and \"(not set)\" to NULL\ntmp_device_df = tmp_device_df.replace([\"not available in demo dataset\",\"(not set)\"], pd.NaT)\ntmp_device_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a28b7e8d-c000-4a0c-aa49-cc279a89dd7b","_cell_guid":"da0eed41-0ac4-4a2f-9bf8-8bcf890c1965","trusted":true},"cell_type":"code","source":"tmp_device_df.nunique()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"affe5018-cd8b-4bd2-8fde-205c45f72618","_cell_guid":"f09ab58c-7426-45f8-9516-ae26d04992e5","trusted":true},"cell_type":"markdown","source":"Keys existed in device attribute are listed above and we could use the nunique function to list out how many unique values are there in one key. We could see that, except four keys, all the other keys are constant with only one unique value. Therefore, we should ignore the features which are not usefull in rest of the process. If feature is misrelated, or it contains lot of \"NaN\" values it should be discarded. We select the ***[\"browser\",\"operatingSystem\",\"deviceCategory\",\"isMobile\"]*** for doing the analyzing. The rest of the device features are ignored and will be removed.\n\nWe can redistribute the dataframe and contain only the useful columns."},{"metadata":{"_uuid":"ab7b9382-050d-4d49-aa37-5b95cff24f7a","_cell_guid":"316a9f74-7fb1-4ab0-a2f0-127bcaee980c","trusted":true},"cell_type":"code","source":"tmp_device_df = pd.DataFrame(train_data.device.apply(json.loads).tolist())[[\"browser\",\"operatingSystem\",\"deviceCategory\",\"isMobile\"]]\ntmp_device_df = tmp_device_df.replace([\"not available in demo dataset\",\"(not set)\"], pd.NaT)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9e899246-835a-42b0-97ed-2b56a7300b05","_cell_guid":"d9ba2f0e-b51b-4aec-8886-161bd56f296c","trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(2,2,figsize=(16,16))\ntmp_device_df[\"isMobile\"].value_counts().plot(kind=\"barh\",ax=axes[0][0],legend=\"isMobile\",color='tan').invert_yaxis()\ntmp_device_df[\"browser\"].value_counts().head(10).plot(kind=\"barh\",ax=axes[0][1],legend=\"browser\",color='teal').invert_yaxis()\ntmp_device_df[\"deviceCategory\"].value_counts().head(10).plot(kind=\"barh\",ax=axes[1][0],legend=\"deviceCategory\",color='green').invert_yaxis()\ntmp_device_df[\"operatingSystem\"].value_counts().head(10).plot(kind=\"barh\",ax=axes[1][1],legend=\"operatingSystem\",color='c').invert_yaxis()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8c6657f-874d-43d4-80f2-2b69f4136008","_cell_guid":"970ebb35-ab0e-4e3e-ad96-d9647e78e588","trusted":true},"cell_type":"markdown","source":"** - D4. geoNetwork **\n\nThe geoNetwork variable is stored in json format. We can use the same way to decompose the variable and pick the useful variables."},{"metadata":{"_uuid":"1ec77e68-d1d2-409e-8ec5-569aa4c6e74b","_cell_guid":"40b2a337-5cf0-4403-8aab-d5d12bcfae0f","trusted":true},"cell_type":"code","source":"tmp_geo_df = pd.DataFrame(train_data.geoNetwork.apply(json.loads).tolist())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60dfb5c3-b362-4995-971c-e16d2c1a1a29","_cell_guid":"d500f827-e74f-40ed-a8a1-414f1c913c29","trusted":true},"cell_type":"code","source":"# replace the JSON's \"not available in demo dataset\" and \"(not set)\" to NULL\ntmp_geo_df = tmp_geo_df.replace([\"not available in demo dataset\",\"(not set)\"], pd.NaT)\ntmp_geo_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31c7c673-07d5-4858-b7f2-bf10bf37b566","_cell_guid":"f58153d4-e3ce-4d9f-a9d1-5f8587ab92cb","trusted":true},"cell_type":"code","source":"tmp_geo_df.nunique()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1822c13e-2cd5-4f96-b51f-4b34a6a06363","_cell_guid":"8e54a53f-64f1-40e0-aab7-0d14903996d2","trusted":true},"cell_type":"code","source":"tmp_geo_df.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b5d60bae-2cf0-4032-bd98-201703ba100e","_cell_guid":"42551fe9-f2a5-4c58-9889-11f1ac549467","trusted":true},"cell_type":"markdown","source":"From the above analysis, we could conclude that: 1. \"cityId\", \"latitude\", \"longitude\" and \"networkLocation\" contain no value; 2. \"networkDomain\", \"region\" have too many null value. So we'll just drop these variables and keep \"continent\", \"subContinent\", \"country\", \"city\", \"metro\"."},{"metadata":{"_uuid":"9201645b-c557-4c93-a718-1a36df8f2bdd","_cell_guid":"7cbb9071-2ebc-4020-a081-bdcd696f0353","trusted":true},"cell_type":"code","source":"tmp_geo_df = pd.DataFrame(train_data.geoNetwork.apply(json.loads).tolist())[[\"continent\", \"subContinent\", \"country\", \"city\", \"metro\"]]\ntmp_geo_df = tmp_geo_df.replace([\"not available in demo dataset\",\"(not set)\"], pd.NaT)  \ntmp_geo_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66c3dc28-28dc-4572-84c8-28d46d9cc8e6","_cell_guid":"0d63f53e-f65e-4718-96bc-dd75a36fac7b","trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(3,2, figsize=(15,15))\ntmp_geo_df[\"continent\"].value_counts().plot(kind=\"bar\",ax=axes[0][0],title=\"Global Distributions\",rot=0,color=\"c\")\ntmp_geo_df[tmp_geo_df[\"continent\"] == \"Americas\"][\"subContinent\"].value_counts().plot(kind=\"bar\",ax=axes[1][0], title=\"America Distributions\",rot=0,color=\"tan\")\ntmp_geo_df[tmp_geo_df[\"continent\"] == \"Asia\"][\"subContinent\"].value_counts().plot(kind=\"bar\",ax=axes[0][1], title=\"Asia Distributions\",rot=0,color=\"r\")\ntmp_geo_df[tmp_geo_df[\"continent\"] == \"Europe\"][\"subContinent\"].value_counts().plot(kind=\"bar\",ax=axes[1][1],  title=\"Europe Distributions\",rot=0,color=\"green\")\ntmp_geo_df[tmp_geo_df[\"continent\"] == \"Oceania\"][\"subContinent\"].value_counts().plot(kind=\"bar\",ax = axes[2][0], title=\"Oceania Distributions\",rot=0,color=\"teal\")\ntmp_geo_df[tmp_geo_df[\"continent\"] == \"Africa\"][\"subContinent\"].value_counts().plot(kind=\"bar\" , ax=axes[2][1], title=\"Africa Distributions\",rot=0,color=\"silver\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8bde2bd-534d-4736-921d-abdb92478f41","_cell_guid":"c5047a52-78e1-403f-b51e-e9e607a17940","trusted":true},"cell_type":"markdown","source":"** - D5. socialEngagementType **\n\nFrom the summary, we could see that there's only one unique data, thus, we should drop this column."},{"metadata":{"_uuid":"fa0c0508-8d85-4cf1-8789-3299b2581803","_cell_guid":"6fe2fff1-596a-4be2-9028-94cce756e0b4","trusted":true},"cell_type":"code","source":"train_data[\"socialEngagementType\"].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1be36ee8-7666-4f67-b250-819fbb03ed4a","_cell_guid":"f99e107d-d04b-4cf7-a1a9-0f6e19299776","trusted":true},"cell_type":"markdown","source":"** - D6. totals **\n\nThe totals variable is stored in json format. We could use the same method to clean the data.\n\nFirst of all, we need to understand the meaning of these features by reading the <a href= \"https://support.google.com/analytics/answer/2731565?hl=en\"> Google Analytics definition</a>:\n\n1. **hits** : hit is an interaction that results in data being sent to Analytics. Common hit types include page tracking hits, event tracking hits, and ecommerce hits.\n1. **bounces**: Google Analytics defines a bounce as a session that has a single interaction hit. Interaction hits are pageviews and other interaction events you may have configured on your page. If you have no other interaction events configured, a bounce is simply a session that has a single pageview. The bounce rate is simply the percent of all sessions that are bounces. If out of 100 sessions, 75 of them had a single pageview and no other interaction hit, then the bounce rate is 75%.\n1. **pageview** : pageview is an instance of a page being loaded (or reloaded) in a browser. Pageviews is a metric defined as the total number of pages viewed.\n1. **newVisits** : newVisits is a binary feature to mark if the user is a new visitor.\n1. **transactionRevenue** : The actual revenue generated in the paticular session. In this dataset, the data is in USD * 1000000 (to maintain precision)."},{"metadata":{"_uuid":"66a84e1e-4cd4-42ba-9558-08e78ede563a","_cell_guid":"b23355d6-27f9-488f-aea3-5358ef0476a6","trusted":true},"cell_type":"code","source":"tmp_totals_df = pd.DataFrame(train_data.totals.apply(json.loads).tolist())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca9610e5-3ada-4052-957d-30ac72696b1e","_cell_guid":"ffde5efc-be94-4e57-8ddb-952cdf8d3f7d","trusted":true},"cell_type":"code","source":"# replace the JSON's \"not available in demo dataset\" and \"(not set)\" to NULL\ntmp_totals_df = tmp_totals_df.replace([\"not available in demo dataset\",\"(not set)\"], pd.NaT)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9622bac1-453d-4aaa-8ce2-cf9a7f4c15e3","_cell_guid":"8cc77e8d-ea50-487e-b0ca-ad6bc3eb957c","trusted":true},"cell_type":"code","source":"tmp_totals_df.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"55992c3e-4692-4ce4-84f6-d9c3635e8e4a","_cell_guid":"f6226a04-747e-4956-bd73-c94c102b71b3","trusted":true},"cell_type":"markdown","source":"From the above analysis, we could conclude that:\"visits\", \"bounces\", and \"newVisits\" contain only one unique value. But by digging deeper in these three parameters, we could see that these features are binary features, so that NULL means 0 in these features. So we could impute these NULL data with 0."},{"metadata":{"_uuid":"ccf22e5c-cadc-48ae-b3ee-46612b0245c0","_cell_guid":"379ab4ae-b2dc-4966-b4de-88fc02db0a8a","trusted":true},"cell_type":"code","source":"tmp_totals_df = tmp_totals_df.fillna(0)\ntmp_totals_df.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"27dc86be-1308-443e-8069-b32c4e92c7de","_cell_guid":"5c25950e-0669-4267-b814-2b04962357e8","trusted":true},"cell_type":"markdown","source":"So that we could see only one feature \"visits\" still contain only one value, we should drop this feature and use the other five features."},{"metadata":{"_uuid":"38a028d9-5090-4fd8-bd19-c4db33c68955","_cell_guid":"3b3aa1cf-4179-4a35-97f8-daf6cd17da53","trusted":true},"cell_type":"code","source":"tmp_totals_df = tmp_totals_df.drop(\"visits\", axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c5f5f75a-09c7-4ad5-a403-58ee99ef0d67","_cell_guid":"589ad92c-fc4a-4ddf-b8e9-7b38a6ff01ee","trusted":true},"cell_type":"markdown","source":"Since transcationRevenue is the value we're going to predict, we're going to dig a little deeper to get a more comprehensive understanding."},{"metadata":{"_uuid":"09b85c20-04bf-4b7c-9651-201c5cbb1f05","_cell_guid":"61e8d004-91d8-403a-bcf0-7b2a07f913d4","trusted":true},"cell_type":"code","source":"# Printing some statistics of our data\n# Convert the data type from string to float \ntmp_totals_df = tmp_totals_df.astype(float)\n\nprint(\"Transaction Revenue Min Value: \", \n      tmp_totals_df[tmp_totals_df['transactionRevenue'] > 0][\"transactionRevenue\"].min()) # printing the min value\nprint(\"Transaction Revenue Mean Value: \", \n      tmp_totals_df[tmp_totals_df['transactionRevenue'] > 0][\"transactionRevenue\"].mean()) # mean value\nprint(\"Transaction Revenue Median Value: \", \n      tmp_totals_df[tmp_totals_df['transactionRevenue'] > 0][\"transactionRevenue\"].median()) # median value\nprint(\"Transaction Revenue Max Value: \", \n      tmp_totals_df[tmp_totals_df['transactionRevenue'] > 0][\"transactionRevenue\"].max()) # the max value","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ed9e142-a333-4930-973c-a24dfde79ff9","_cell_guid":"6786567a-3930-40d0-b3f6-68a5fc5fdabe","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(14,5))\n\nplt.subplot(1,2,1)\ntmp = [ len(tmp_totals_df[tmp_totals_df['transactionRevenue'] == 0]),len(tmp_totals_df[tmp_totals_df['transactionRevenue'] > 0])]\nlebels = [\"Sessions w/o revenue\",\"Sessions w/ revenue\"]\n\nplt.pie(tmp, autopct='%1.1f%%', startangle=90, textprops=dict(color=\"w\",fontsize= 14))\nplt.legend(lebels,loc=\"up right\",bbox_to_anchor=(0.8, 0, 0.5, 1))\n\n# seting the distribuition of our data and normalizing using np.log on values highest than 0 and + \n# also, we will set the number of bins and if we want or not kde on our histogram\nplt.subplot(1,2,2)\nax = sns.distplot(np.log(tmp_totals_df[tmp_totals_df['transactionRevenue'] > 0][\"transactionRevenue\"]), bins=60, kde=True)\nax.set_xlabel('Transaction RevenueLog', fontsize=15) #seting the xlabel and size of font\nax.set_ylabel('Distribuition', fontsize=15) #seting the ylabel and size of font\nax.set_title(\"Distribuition of Revenue Log\", fontsize=20) #seting the title and size of font\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af7cab1e-474e-4120-a962-91d387ce5f32","_cell_guid":"4a594362-ce98-4475-aee6-befbf4639353","trusted":true},"cell_type":"markdown","source":"** - D7. trafficSource **\n\ntrafficSource attribute can be used to resolve the question of what's the most conventional manner for visitor who visit to the website and do their shopping. Like a previous Json elements existed in the dataset, this attribute is also Json file. so, we use the similar way to deserialize it. We have select \"keyword\", \"source\" and the \"medium\" as a features which can bring more useful infromation."},{"metadata":{"_uuid":"0497803d-5d86-4703-9e34-fcffb0756946","_cell_guid":"0b9c60a2-2a3b-4e9d-81bd-3dcf2f86b8b3","trusted":true},"cell_type":"code","source":"traffic_source_df = pd.DataFrame(train_data.trafficSource.apply(json.loads).tolist())[[\"keyword\",\"medium\" , \"source\"]]\ntraffic_source_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce689bda-e8a5-46ba-ad31-f89d01e8907e","_cell_guid":"81089089-d8e5-4677-b9cd-b07e57a3dc43","trusted":true},"cell_type":"code","source":"fig,axes = plt.subplots(1,2,figsize=(15,10))\ntraffic_source_df[\"medium\"].value_counts().plot(kind=\"barh\",ax = axes[0],title=\"Medium\",color=\"tan\").invert_yaxis()\ntraffic_source_df[\"source\"].value_counts().head(10).plot(kind=\"barh\",ax=axes[1],title=\"source\",color=\"teal\").invert_yaxis()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"108a1c71-9aa3-4ca4-8105-6de1f9ac4c1e","_cell_guid":"260f0bea-27de-4aa1-8d10-addc5abcd7e4","trusted":true},"cell_type":"markdown","source":"As it is completely obvious in source diagram, google is the most repetitive source. It would be interesting if we replace all google subdomains with exact 'google' and do the same analyze again."},{"metadata":{"_uuid":"f9f530bc-ea27-439b-911e-7bee0a48b3d0","_cell_guid":"444a841a-dbc1-4bd8-942d-b398bf9c8514","trusted":true},"cell_type":"code","source":"traffic_source_df.loc[traffic_source_df[\"source\"].str.contains(\"google\") ,\"source\"] = \"google\"\nfig,axes = plt.subplots(1,1,figsize=(8,8))\ntraffic_source_df[\"source\"].value_counts().head(15).plot(kind=\"barh\",ax=axes,title=\"source\",color=\"teal\").invert_yaxis()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"586e4a4a-17ef-4841-8a1d-b68231995845","_cell_guid":"37e0ba72-f2ef-4b62-a63d-ef6644f7e7ac","trusted":true},"cell_type":"markdown","source":"Google dependent redirects are more than twice the youtube sources. Combination of this feature with revenue and visits may have important result. We will do it in next step (when we are analyzing feature correlations). Now let's move on keywords feature. A glance to keyword featre represnets lot of missing values '(not provided)'. Drawing a bar chart for both of them."},{"metadata":{"_uuid":"13e0228a-c6bb-45ed-9c4e-7bc50e040ea6","_cell_guid":"42002647-2293-4fe0-b223-5889841a6edf","trusted":true},"cell_type":"code","source":"fig,axes = plt.subplots(1,2,figsize=(16,8))\ntraffic_source_df[\"keyword\"].value_counts().head(5).plot(kind=\"barh\",ax=axes[0], title=\"keywords (total)\",color=\"orange\").invert_yaxis()\ntraffic_source_df[traffic_source_df[\"keyword\"] != \"(not provided)\"][\"keyword\"].value_counts().head(10).plot(kind=\"barh\",ax=axes[1],title=\"keywords (dropping NA)\",color=\"c\").invert_yaxis()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a8b4d74d-ca59-4762-b97d-3d7706c662ad","_cell_guid":"5905088c-e88a-4214-ad72-b43b3abad21e","trusted":true},"cell_type":"markdown","source":"** -D8. visitId & sessionId**\n\nThese two features are acting the same function to index the dataset. We could ignore them since we're going to predict the customer's behavior instead of what's the revenue will be in a single session."},{"metadata":{"_uuid":"00628707-6d51-40e4-8d19-05cebfc59b5b","_cell_guid":"07f0412e-391a-4df9-afc4-27f5fa3d1de4","trusted":true},"cell_type":"markdown","source":"** -D9. visitNumber **\n\nThe feature indicates how many times a specific user visits the store. By intuition, this feature has a potential to be an important factor in regression progress."},{"metadata":{"_uuid":"121dab52-f905-4e7a-b932-ceadc455711f","_cell_guid":"137ba857-2e8e-4d88-b843-9b5c7da29ac3","trusted":true},"cell_type":"code","source":"train_data.visitNumber.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2fead9d0-652a-47ac-8ff1-4745d2031a6d","_cell_guid":"6a5c4f67-65de-4c6b-8152-04ed44b0a525","trusted":true},"cell_type":"markdown","source":"We could see that 75% of the users only visit the store for one time, so we could say that maybe majority of the visitors are lured to the store by marketing campaign and are not our target customers. We could further understand the situation to see the 90% of the users' behaviors. We can do the further analysis to see the correlation between the total visits and the revenue."},{"metadata":{"_uuid":"b05d61d5-d3af-49de-924f-d2300392d8e0","_cell_guid":"1401554d-f146-47fd-9d22-be05cc3de7c3","trusted":true},"cell_type":"code","source":"print(\"90 percent of sessions have visitNumber lower than {} times.\".format(np.percentile(list(train_data.visitNumber),90)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9fa4ad33-9cf2-4317-82e9-92b8a48105b0","_cell_guid":"5409dc9e-8f91-4a0c-a300-7e39fe90259f","trusted":true},"cell_type":"markdown","source":"** -D10. fullVisitorId **\n\nAs decribed, this feature is a unique identifier for each user of the Google Merchandise Store. We could get that in our dataset, there're 742735 different users."},{"metadata":{"_uuid":"28021a83-7c95-4eec-b957-3768af15196c","_cell_guid":"4e9f431a-9e93-41aa-821b-b505e0918c6a","trusted":true},"cell_type":"code","source":"train_data.fullVisitorId.nunique()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b08dd1b-38c7-460f-9497-d6edc2f2d2f7","_cell_guid":"b9eb73af-655a-41ea-9bc9-a6e6fe36d25f","trusted":true},"cell_type":"markdown","source":"# 3. Data preparation <a id=\"3\"></a>\n\nWith the understanding of the features in the dataset, in this section, we're going dig deeper to construct a more comprehensive view by combining the features together to see the interrelationship between the features."},{"metadata":{"_uuid":"c4928a2e-9d84-430d-b158-2477197ec4a3","_cell_guid":"ad3432f1-2fe3-440d-bf3c-22d11380656d","trusted":true},"cell_type":"markdown","source":"* A. transactionRevenue and other features\n\nIn order to find out the relationship between transaction revenue with other features, we could first write a function to plot out the relationship between different columns and the available transaction revenue.\nFirst we need to merge all the useful features we talked above and get a cleaned dataset."},{"metadata":{"_uuid":"96e502e2-ee1b-443f-a82f-19dec17b6127","_cell_guid":"0eb4eac4-93b5-4997-8ac6-33df45bb577b","trusted":true},"cell_type":"code","source":"# Merge all the useful features back together after the cleaning procedures above.\ntrain_data_cleaned = train_data[[\"fullVisitorId\", \"channelGrouping\", \"date\", \"visitStartTime\",\"visitNumber\"]]\ntrain_data_cleaned = train_data_cleaned.merge(tmp_device_df,left_index=True, right_index=True)\ntrain_data_cleaned = train_data_cleaned.merge(tmp_geo_df,left_index=True, right_index=True)\ntrain_data_cleaned = train_data_cleaned.merge(tmp_totals_df,left_index=True, right_index=True)\ntrain_data_cleaned = train_data_cleaned.merge(traffic_source_df,left_index=True, right_index=True)\ntrain_data_cleaned.head(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bbe51486-975f-4cde-8982-33554d6828ed","_cell_guid":"3ec2f507-fdb4-4a32-ac6f-4a2d07fa8059","trusted":true},"cell_type":"markdown","source":"Then we could subset the dataset, and put all the rows with revenue together."},{"metadata":{"_uuid":"47f1cf8f-8769-4143-a73e-25dd44caa2f2","_cell_guid":"4ef71c83-ac29-45bb-80d5-6210b265cf69","trusted":true},"cell_type":"code","source":"train_data_rev = train_data_cleaned[train_data_cleaned['transactionRevenue'] > 0]\ntrain_data_rev.head(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b082a786-666b-478b-89d3-670106be4a8d","_cell_guid":"1e7efa42-c6f2-41d3-ad4a-2e21bd15d909","trusted":true},"cell_type":"markdown","source":"Then we can write a function to plot the relationship between the overall dataset and the rows with revenue."},{"metadata":{"_uuid":"15152b8b-f006-4f59-bb3f-e61b6aeac132","_cell_guid":"82a3d5cf-f4dd-4075-8202-ff32260ace1d","trusted":true},"cell_type":"code","source":"def plotCategoryRateBar(a, b, colName, topN=np.nan):\n    if topN == topN:\n        vals = b[colName].value_counts()[:topN]\n        subA = a.loc[a[colName].isin(vals.index.values), colName]\n        df = pd.DataFrame({'% in Rows with Revenue':subA.value_counts() / len(a) *100, '% in Overall Dataset':vals / len(b)*100})\n    else:\n        df = pd.DataFrame({'% in Rows with Revenue':a[colName].value_counts() / len(a)*100, '% in Overall Dataset':b[colName].value_counts() / len(b)*100})\n    #return the barh plot\n    df.sort_values('% in Rows with Revenue').plot.barh(colormap='bwr', title = colName +\" and Revenue relationship\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also write a function to plot the relationship between the total/mean/median revenue.\n\nThrough this analysis, we could understand the total revenue and the unit economic from a certain group of customers."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plotCategoryAvgBar(a, colName,topN=np.nan):\n    df = pd.DataFrame()\n    \n    for item in a[colName].unique():\n        mean = np.mean(a[a[colName] == item].transactionRevenue)/100000\n        median = np.median(a[a[colName] == item].transactionRevenue)/100000\n        total = np.sum(a[a[colName] == item].transactionRevenue)/100000\n        temp_df = pd.DataFrame([[mean,median,total]],index = [item],columns=[\"mean\",\"median\",\"total\"])\n        df = pd.concat([df,temp_df])\n    \n    if topN == topN:\n        fig,axes=plt.subplots(1,2,figsize=(14,5))\n        df.loc[:,\"total\"].sort_values().tail(topN).plot.barh(ax=axes[0],color=\"b\",title = colName +\" and total revenue relationship\")\n        df.loc[:,[\"mean\",\"median\"]].sort_values(by = \"median\").tail(topN).plot.barh(ax=axes[1],colormap='PiYG', title = colName +\" and Revenue mean/median relationship\")\n    else:\n        fig,axes=plt.subplots(1,2,figsize=(14,5))\n        df.loc[:,\"total\"].sort_values().plot.barh(ax=axes[0],color=\"b\",title = colName +\" and total revenue relationship\")\n        df.loc[:,[\"mean\",\"median\"]].sort_values(by = \"median\").plot.barh(ax=axes[1],colormap='PiYG', title = colName +\" and Revenue mean/median relationship\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"948b8062-83aa-49de-923f-83043fe37dc6","_cell_guid":"d9e75427-5226-4616-98db-39650fdfa40b","trusted":true},"cell_type":"markdown","source":"A1. Channel Grouping"},{"metadata":{"_uuid":"ab837934-f5da-489a-80cf-3b5f99588789","_cell_guid":"b688eb0b-d3f9-4c34-8cca-bb1095516637","trusted":true},"cell_type":"code","source":"plotCategoryRateBar(train_data_rev, train_data_cleaned, \"channelGrouping\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotCategoryAvgBar(train_data_rev, \"channelGrouping\",5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a85db50-1acc-4d42-bcfa-9e1e2a76ecb3","_cell_guid":"4ccd662e-a21f-45d0-9b36-47625bfe484d","trusted":true},"cell_type":"markdown","source":"A2. Device"},{"metadata":{"_uuid":"6e513c57-f1a8-449d-8552-4a7c89dc918f","_cell_guid":"83c97493-2a45-42be-bf41-598818a6bacb","trusted":true},"cell_type":"code","source":"plotCategoryRateBar(train_data_rev, train_data_cleaned, \"browser\",5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25720761-1ed6-4221-b92b-3629882db4f4","_cell_guid":"dcce1887-a670-436e-bc98-bb26366b8179","trusted":true},"cell_type":"code","source":"plotCategoryRateBar(train_data_rev, train_data_cleaned, \"operatingSystem\",5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotCategoryAvgBar(train_data_rev, \"operatingSystem\",5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a7a00e43-2151-44b7-9e75-eac1c4ad35cf","_cell_guid":"378bc10e-58c8-4196-96f9-975a780ed35a","trusted":true},"cell_type":"code","source":"plotCategoryRateBar(train_data_rev, train_data_cleaned, \"deviceCategory\",5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotCategoryAvgBar(train_data_rev, \"deviceCategory\",5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed5ff554-bbc2-495f-b503-a213495f04b1","_cell_guid":"53d39edd-eef3-44fb-aa3c-7c8e3aa49887","trusted":true},"cell_type":"markdown","source":"A3. geoNetwork"},{"metadata":{"_uuid":"b92231be-f175-4118-b82b-90c38918026b","_cell_guid":"ee0c8998-029b-4027-ae3c-6a72ff5aa28e","trusted":true},"cell_type":"code","source":"plotCategoryRateBar(train_data_rev, train_data_cleaned, \"continent\",5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"021041c8-5dcb-4751-a186-82fb8da8ac15","_cell_guid":"169b18eb-0e65-462b-92e7-342888c9bd29","trusted":true},"cell_type":"code","source":"plotCategoryRateBar(train_data_rev, train_data_cleaned, \"metro\",5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4216ef75-c23e-4cc4-a42c-322a6a52cb20","_cell_guid":"4a14843c-9d18-456e-a876-9a10842eee09","trusted":true},"cell_type":"code","source":"plotCategoryRateBar(train_data_rev, train_data_cleaned, \"city\",5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38b9e831-b467-4a79-ae09-6eda3151b587","_cell_guid":"5ee46ef2-27dd-4948-9933-16b55e50bf62","trusted":true},"cell_type":"markdown","source":"A4. trafficSource"},{"metadata":{"_uuid":"191d7c7c-f947-44be-9aca-f9f124d9efaf","_cell_guid":"640e0ec5-524d-464d-b61a-7fc343fd7fa0","trusted":true},"cell_type":"code","source":"plotCategoryRateBar(train_data_rev, train_data_cleaned, \"medium\",5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a4f339d-98d6-4e40-8862-f8bf21c82d3c","_cell_guid":"ad17a887-b9c7-4576-8679-c8edd8c50b63","trusted":true},"cell_type":"code","source":"plotCategoryRateBar(train_data_rev, train_data_cleaned, \"source\",5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14ee6ab8-edfa-45db-9837-2a4b38f60d53","_cell_guid":"03dfa7b7-8010-4f27-b63e-6b9552bf5c7f","trusted":true},"cell_type":"markdown","source":"* B. Churn & Conversion\n\n> The ChurnRate is The percentage rate at which customers stop subscribing to a service. Churn rate period can be various from day to a year correspoding to business type. In this section, we will compute and visualize the monthly churn rate. Lets do more investigation on features date and fullVisitorId for more detail mining."},{"metadata":{"_uuid":"58e8ff21-ee05-4079-9525-7ed03a11f3b0","_cell_guid":"a2705bfb-c8b0-445c-9ef9-b2bd832d70ec","trusted":true},"cell_type":"code","source":"# list how many days of observations in our dataset\ndate_list = np.sort(list(set(list(train_data[\"date\"]))))\n\"first_day:'{}' and last_day:'{}' and toal number of data we have is: '{}' days.\".format(date_list[0], date_list[-1],len(set(list(train_data[\"date\"]))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6035bbe-bb09-4113-9dc3-4448a66cc1ca","_cell_guid":"c1ea4da7-88a0-4bc0-9386-5598ec9f5ad2","trusted":true},"cell_type":"markdown","source":"We could see that we have 366 days (12 month = 1 year) from August 2016 to August 2017 data for churn rate calculations. The best period for churn maybe is the monthly churn rate.\n\nWe'll define new empty dataframe and do this calculations on it and copy the our requirements on it. For monthly churn rate calculations, we need to check which users visited have visited the website monthly. This information is located in fullVisitorId. We will copy it to new df."},{"metadata":{"_uuid":"9275e2eb-7ddd-47a6-aa0d-3fc3c12fdb73","_cell_guid":"c5e67d07-22d8-4be3-a104-8a9a9787edd4","trusted":true},"cell_type":"code","source":"tmp_churn_df = pd.DataFrame()\ntmp_churn_df[\"date\"] = train_data[\"date\"]\ntmp_churn_df[\"year\"] = pd.DatetimeIndex(tmp_churn_df[\"date\"]).year\ntmp_churn_df[\"month\"] =pd.DatetimeIndex(tmp_churn_df[\"date\"]).month\ntmp_churn_df[\"fullVisitoId\"] = train_data[\"fullVisitorId\"]\ntmp_churn_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d8ddf15d-7bce-43b8-8050-350cbf0d2217","_cell_guid":"63688320-d379-4c6a-8ada-c163a24d9654","trusted":true},"cell_type":"code","source":"\"distinct users who visited the website on 2016-08 are:'{}'persons\".format(len(set(tmp_churn_df[(tmp_churn_df.year == 2016) & (tmp_churn_df.month == 8) ][\"fullVisitoId\"])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f374742-ed73-4ab1-817c-8ffb36d82924","_cell_guid":"b9fa5907-d42d-4e19-9fb8-2301d90d5675","trusted":true},"cell_type":"code","source":"# so that we could use the same method to save the interval visitors in different months\ntarget_intervals_list = [(2016,8),(2016,9),(2016,10),(2016,11),(2016,12),(2017,1),(2017,2),(2017,3),(2017,4),(2017,5),(2017,6),(2017,7)]\nintervals_visitors = []\nfor tmp_tuple in target_intervals_list: \n    intervals_visitors.append(tmp_churn_df[(tmp_churn_df.year == tmp_tuple[0]) & (tmp_churn_df.month == tmp_tuple[1]) ][\"fullVisitoId\"])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f74faa5f-c733-4ca6-b21d-72d6f002a65d","_cell_guid":"ad2d9729-bf93-4232-be75-0a62d832ab5d","trusted":true},"cell_type":"code","source":"tmp_matrix = np.zeros((11,11))\n\nfor i in range(0,11):\n    k = False\n    tmp_set = []\n    for j in range(i,11): \n        if k:\n            tmp_set = tmp_set & set(intervals_visitors[j])\n        else:\n            tmp_set = set(intervals_visitors[i]) & set(intervals_visitors[j])\n        tmp_matrix[i][j] = len(list(tmp_set))\n        k = True","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1dfdf73a-4f52-4c2e-9447-8c513a6f43f9","_cell_guid":"1a7a1440-95f5-4a50-a8fb-1d02bf819a67","trusted":true},"cell_type":"code","source":"xticklabels = [\"interval 1\",\"interval 2\",\"interval 3\",\"interval 4\",\"interval 5\",\"interval 6\",\"interval 7\",\"interval 8\",\n              \"interval 9\",\"interval 10\",\"interval 11\"]\nyticklabels = [(2016,8),(2016,9),(2016,10),(2016,11),(2016,12),(2017,1),(2017,2),(2017,3),(2017,4),(2017,5),(2017,6),(2017,7)]\nfig, ax = plt.subplots(figsize=(11,11))\nax = sns.heatmap(np.array(tmp_matrix,dtype=int), annot=True, cmap=\"coolwarm\",xticklabels=xticklabels,fmt=\"d\",yticklabels=yticklabels)\nax.set_title(\"Churn-rate heatmap\")\nax.set_xlabel(\"intervals\")\nax.set_ylabel(\"months\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2201c9dc-76cf-46b8-b9ac-63e88f981e14","_cell_guid":"10effcd3-e9bd-4bd2-a84d-19a6e2369539","trusted":true},"cell_type":"markdown","source":"We could see that our monthly churn rate is extremely high. We need to further analyze the customer behaviors behind the high churn rate."},{"metadata":{"_uuid":"f72d92c9-bea7-4622-bb35-e991741f2e5e","_cell_guid":"ec65a2e3-b159-4487-95b2-34f1dc756fa5","trusted":true},"cell_type":"markdown","source":"# 4. Modeling <a id=\"4\"></a>"},{"metadata":{"_uuid":"31207ea7-9fba-4691-9fd6-763d4235792e","_cell_guid":"020262de-1014-4949-83b7-8c011b810a7e","trusted":true},"cell_type":"markdown","source":"**Model 1. Predict if there's transaction(The possibility of the transaction)**"},{"metadata":{"_uuid":"fbd85bde-96d1-4f13-b4a1-a4a80691a15b","_cell_guid":"53446cb7-1d63-48c6-ae95-bbd1fe0dde73","trusted":true},"cell_type":"code","source":"train_data_cleaned = train_data_cleaned.drop(['fullVisitorId','date','visitStartTime'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"51fbb560-cd68-4e8e-8027-35775adae1e1","_cell_guid":"7f628784-675c-47fb-b3ed-66ea7beda36b","trusted":true},"cell_type":"code","source":"train_data_binaryprediction= train_data_cleaned\ntrain_data_binaryprediction[\"ifrevenue\"] = 0\nfor i in range(len(train_data_binaryprediction.transactionRevenue)):\n    if train_data_binaryprediction.transactionRevenue[i] != 0: train_data_binaryprediction[\"ifrevenue\"][i] = 1\n\ntrain_data_binaryprediction = train_data_binaryprediction.drop(\"transactionRevenue\", axis = 1)\ntrain_data_binaryprediction.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc9248ad-b883-4597-9b9f-f29b11519f8a","_cell_guid":"87b24c52-ee3c-4af0-ac26-06bf18b2003c","trusted":true},"cell_type":"code","source":"df_train_model1, df_test_model1 = train_test_split(train_data_binaryprediction, test_size=0.2, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3be90cff-ce57-4fdb-b852-dfc2f02fd56a","_cell_guid":"cf4f5a8c-d5da-4d90-862b-457c3dd482b0","trusted":true},"cell_type":"code","source":"categorical_features = ['channelGrouping', 'browser', 'operatingSystem', 'deviceCategory', 'isMobile',\n                        'continent', 'subContinent', 'country', 'city','metro', 'keyword', 'medium', 'source']\n\nnumerical_features = ['visitNumber', 'newVisits', 'bounces', 'pageviews', 'hits']\n\nfor column_iter in categorical_features:\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(df_train_model1[column_iter].values.astype('str')) + list(df_test_model1[column_iter].values.astype('str')))\n    df_train_model1[column_iter] = lbl.transform(list(df_train_model1[column_iter].values.astype('str')))\n    df_test_model1[column_iter] = lbl.transform(list(df_test_model1[column_iter].values.astype('str')))\n\nfor column_iter in numerical_features:\n    df_train_model1[column_iter] = df_train_model1[column_iter].astype(np.float)\n    df_test_model1[column_iter] = df_test_model1[column_iter].astype(np.float)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cf225e74-d927-482b-9790-81885b18c8e4","_cell_guid":"57447b0e-d8c4-493e-bcc0-e7e0a1aa2bfc","trusted":true},"cell_type":"markdown","source":"We will compare models on the validation set using the **log loss** metric. As a reminder, log loss is used to measure the performance of models whose output is a probability prediction between 0 and 1. It is defined as follows:\n\n$$LogLoss = -\\frac{1}{n} \\sum^{n}_{i=1}[y_{i}log(\\hat{p_{i}}) + (1 - y_{i})log(1-\\hat{p_{i}})]$$\n\nwhere\n\n+ $n$ is the number of observations (*i.e.* the number of cars) in the data set\n+ $y_{i}$ is the observed realization of car $i$; it equals 1 if the car has a major defect and 0 if not\n+ $\\hat{p_{i}}$ is the predicted probability of car $i$ having a major defect according to the model[](http://)"},{"metadata":{"_uuid":"eb917708-17dd-41f9-a0ff-9eb476bcac1c","_cell_guid":"dd9cb7cc-26a2-4a1a-b1b2-42d7e4af3018","trusted":true},"cell_type":"code","source":"params_model1 = {\n    \"objective\": \"binary\",\n    \"metric\": \"binary_logloss\",\n    \"num_leaves\": 30,\n    \"min_child_samples\": 100,\n    \"learning_rate\": 0.1,\n    \"bagging_fraction\": 0.7,\n    \"feature_fraction\": 0.5,\n    \"bagging_frequency\": 5,\n    \"bagging_seed\": 2018,\n    \"verbosity\": -1\n}\nlgb_train_model1 = lgb.Dataset(df_train_model1.loc[:,df_train_model1.columns != \"ifrevenue\"], np.log1p(df_train_model1.loc[:,\"ifrevenue\"]))\nlgb_eval_model1 = lgb.Dataset(df_test_model1.loc[:,df_test_model1.columns != \"ifrevenue\"], np.log1p(df_test_model1.loc[:,\"ifrevenue\"]), reference=lgb_train_model1)\ngbm_model1 = lgb.train(params_model1, lgb_train_model1, num_boost_round=2000, valid_sets=[lgb_eval_model1], early_stopping_rounds=100,verbose_eval=100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92ae3c49-52c4-4ddf-a7a0-77a59b0a9a10","_cell_guid":"402108e8-73b0-42ce-9ea3-aa7043f62501","trusted":true},"cell_type":"markdown","source":"We could know that since we have only 1.5% of data with revenue, it's quite an unbalanced dataset. So our log loss model should have a lower dumb-LogLoss score. We can refer to the dum-LogLoss score <a href=\"https://medium.com/@fzammito/whats-considered-a-good-log-loss-in-machine-learning-a529d400632d\">here</a>, and we could see that our score is significantly lower than the dumb-logloss score, which makes our prediction valid.\n<img src=\"https://miro.medium.com/max/1400/0*eBPkZ24YYGdCSu58.png\">"},{"metadata":{"_uuid":"73d735ee-a59d-49c0-be03-09a41de21d54","_cell_guid":"cb7d095f-f18f-44f9-85cc-89bdf2239181","trusted":true},"cell_type":"code","source":"lgb.plot_importance(gbm_model1,grid=False,height=0.6)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df8f286d-2c52-4275-a81a-95a125b3a3d0","_cell_guid":"dc60c683-0bb8-4bb1-b4b3-6eeee6967964","trusted":true},"cell_type":"markdown","source":"**Model 2. Predict log transactionRevenue**"},{"metadata":{"_uuid":"a266420d-5249-42c6-a012-e539a910d39a","_cell_guid":"831489d7-699b-445a-9d64-81d7fda0472b","trusted":true},"cell_type":"code","source":"df_train = train_data_cleaned\ndf_train = df_train.drop(\"ifrevenue\",axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec708c78-b469-4a41-9788-7bf81ceb9852","_cell_guid":"0830d41b-ca8e-459e-8699-7071b66afd88","trusted":true},"cell_type":"code","source":"df_train, df_test = train_test_split(df_train, test_size=0.2, random_state=42)\n\ndf_train[\"transactionRevenue\"] = df_train[\"transactionRevenue\"].astype(np.float)\ndf_test[\"transactionRevenue\"] = df_test[\"transactionRevenue\"].astype(np.float)\nprint(\"We have these columns for our regression problems:\\n{}\".format(df_train.columns))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"edeeddac-29ad-4846-a213-0a9c931fdff9","_cell_guid":"a01f5d1d-4e7e-4d34-b983-e50a6f86f101","trusted":true},"cell_type":"code","source":"df_train.head(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31191dae-2efa-4c08-8041-b1c6e33f903a","_cell_guid":"2e4622ae-8b9c-49e1-834e-098c69394501","trusted":true},"cell_type":"code","source":"categorical_features = ['channelGrouping', 'browser', 'operatingSystem', 'deviceCategory', 'isMobile',\n                        'continent', 'subContinent', 'country', 'city','metro', 'keyword', 'medium', 'source']\n\nnumerical_features = ['visitNumber', 'newVisits', 'bounces', 'pageviews', 'hits']\n\nfor column_iter in categorical_features:\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(df_train[column_iter].values.astype('str')) + list(df_test[column_iter].values.astype('str')))\n    df_train[column_iter] = lbl.transform(list(df_train[column_iter].values.astype('str')))\n    df_test[column_iter] = lbl.transform(list(df_test[column_iter].values.astype('str')))\n\nfor column_iter in numerical_features:\n    df_train[column_iter] = df_train[column_iter].astype(np.float)\n    df_test[column_iter] = df_test[column_iter].astype(np.float)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d7111db3-b22c-4d92-bbf0-2754cb8be898","_cell_guid":"33db4bcf-3d48-4a3e-9361-2632b69c522d","trusted":true},"cell_type":"code","source":"params = {\n    \"objective\": \"regression\",\n    \"metric\": \"rmse\",\n    \"num_leaves\": 30,\n    \"min_child_samples\": 100,\n    \"learning_rate\": 0.1,\n    \"bagging_fraction\": 0.7,\n    \"feature_fraction\": 0.5,\n    \"bagging_frequency\": 5,\n    \"bagging_seed\": 2018,\n    \"verbosity\": -1\n}\nlgb_train = lgb.Dataset(df_train.loc[:,df_train.columns != \"transactionRevenue\"], np.log1p(df_train.loc[:,\"transactionRevenue\"]))\nlgb_eval = lgb.Dataset(df_test.loc[:,df_test.columns != \"transactionRevenue\"], np.log1p(df_test.loc[:,\"transactionRevenue\"]), reference=lgb_train)\ngbm = lgb.train(params, lgb_train, num_boost_round=2000, valid_sets=[lgb_eval], early_stopping_rounds=100,verbose_eval=100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c44fad2-769f-436d-99bb-20c064f1c8f7","_cell_guid":"9f11bdf4-3df5-48df-bfeb-a4b23d5dc161","trusted":true},"cell_type":"code","source":"lgb.plot_importance(gbm,grid=False,height=0.6)","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":4}