{"cells":[{"metadata":{"_uuid":"45ee2365f5c696de32206f648df3dfb2b60e84b0"},"cell_type":"markdown","source":"# Table of Contents\n- 1. Introduction\n- 2. Evaluation - Root Mean Squared Error (RMSE)\n- 3. Make Assumptions\n- 4. Retrieving the Data\n- 5. Glimpse of Data\n- 6. Exploratory Data Analysis (EDA)\n - 6.1. Univariate Distribution\n - 6.1.1. Distribution of the target variable : transactionRevenue\n - 6.1.1.1. Target variable : transactionRevenue presents NaN values \n - 6.1.2. Numerical Distribution\n - 6.1.2.1 Distribution of date\n - 6.1.2.2 Distribution of VisitId\n - 6.1.2.3 Distribution of visitNumber\n - 6.1.2.4 Distribution of visitStartTime\n - 6.1.3. Categorical Distribution\n - 6.1.3.1 Distribution of channelGrouping\n - 6.1.3.2 Distribution of fullVisitorId\n - 6.1.3.3 Distribution of sessionId\n - 6.1.3.4 Distribution of browser \n - 6.1.3.5 Distribution of deviceCategory \n - 6.1.3.6 Distribution of isMobile\n - 6.1.3.7 Distribution of operatingSystem\n - 6.1.3.8 Distribution of city\n - 6.1.3.9 Distribution of continent\n - 6.1.3.10 Distribution of country\n - 6.1.3.11 Distribution of metro\n - 6.1.3.12 Distribution of networkDomain \n - 6.1.3.13 Distribution of region\n - 6.1.3.14 Distribution of subContinent\n - 6.1.3.15 Distribution of bounces\n - 6.1.3.16 Distribution of hits \n - 6.1.3.17 Distribution of newVisits\n - 6.1.3.18 Distribution of PageViews\n - 6.1.3.19 Distribution of adContent\n - 6.1.3.20 Distribution of adwordsClickInfo.adNetworkType\n - 6.1.3.21 Distribution of adwordsClickInfo.gclId\n - 6.1.3.22 Distribution of adwordsClickInfo.isVideoAd\n - 6.1.3.23 Distribution of adwordsClickInfo.page\n - 6.1.3.24 Distribution of adwordsClickInfo.slot\n - 6.1.3.25 Distribution of campaign\n - 6.1.3.26 Distribution of campaignCode\n - 6.1.3.27 Distribution of isTrueDirect\n - 6.1.3.28 Distribution of keyword\n - 6.1.3.29 Distribution of medium\n - 6.1.3.30 Distribution of referralPath\n - 6.1.3.31 Distribution of source\n - 6.1.3.32 Distribution of new Date Features \n - **6.1.3.33 Univariate Analysis Conclusions** \n - 6.2. Bivariate Distribution \n - 6.2.1. VisitNumber vs Target \n - 6.2.2. ChannelGrouping vs Target \n - 6.2.3. Browser vs Target \n - 6.2.4. deviceCategory vs Target \n - 6.2.5. Device OS vs Target \n - 6.2.6. Pageviews vs Target \n - 6.2.7. Continent vs Target \n - **6.2.8. Bivariate Analysis Conclusions** "},{"metadata":{"_uuid":"ecf8cddc46c13856e0f8be5993b2e931cff34412"},"cell_type":"markdown","source":" ## 1. Google Analytics Customer Revenue Prediction \n\nThe 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue. As such, marketing teams are challenged to make appropriate investments in promotional strategies.\n\nRStudio, the developer of free and open tools for R and enterprise-ready products for teams to scale and share work, has partnered with Google Cloud and Kaggle to demonstrate the business impact that thorough data analysis can have.\n\nIn this competition, **you’re challenged to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer.** Hopefully, the outcome will be more actionable operational changes and a better use of marketing budgets for those companies who choose to use data analysis on top of GA data.\n\n"},{"metadata":{"_uuid":"dbf6dbdc9f8157af7a5cdc93a1995c155f61f67b"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"4cd769ca64e7b0742582c15babc1807cadd2fe85"},"cell_type":"markdown","source":"

2. Evaluation - Root Mean Squared Error (RMSE)

\n\nSubmissions are scored on the root mean squared error. \n\nFor each **fullVisitorId** in the test set, **you must predict the natural log of their total revenue in PredictedLogRevenue**. \n"},{"metadata":{"_uuid":"30deddd64c568531c227497d68c7a7066c04802b"},"cell_type":"markdown","source":"

3. Make Assumptions

\n\nBefore looking at the data made available I always like to think what possible conclusions I might take depending on the available data. By doing this brainstorm, there is a higher chance you will think about relationship that you might not have tought if you limited your logic to the known data.\n\nSo, what are the factors that influence Google Merchandise Store sales values?\n\n- Clients with ages ranging between 18 and 30, will be responsible for the majority of sales;\n- Male gender will higher sales values then Female gender;\n- Apparel, mostly shirts and sweatshirts, and Bundles will be the most sold item in the site\n- The month with highest sales will be end of the year (December)\n- Asia and North America are the regions of the world which buy more products\n- Customer that spend more on these item have interest in technology, lifestyle, Sports, Gameing (data from Google profile)\n- Most buyers are from Tier 1 and 2 cities\n- Web platform is the most used to purchase this items\n- The majority of client's pay with credit card\n- Sales will be affected by shipping price\n- Do the store ship to all countries?\n\n"},{"metadata":{"_uuid":"7b6dbf56ccbf927c0fae642c5eea0573fb4c8fdb"},"cell_type":"markdown","source":"

4. Retrieving the Data

"},{"metadata":{"trusted":true,"_uuid":"65f5cc71ee6ef895a1710b0686fb1d75bf8cb9b4"},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\nimport matplotlib.pyplot as plt\nimport seaborn as sns\ncolor = sns.color_palette()\n\n%matplotlib inline\n\nfrom plotly import tools\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\nfrom sklearn import model_selection, preprocessing, metrics\nimport lightgbm as lgb\n\npd.options.mode.chained_assignment = None\npd.options.display.max_columns = 999","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f7f77478cd5ac6d4b601b82dcda91b14a6e407b"},"cell_type":"markdown","source":"**What should I expect the data format to be?**\n\nBoth train.csv and test.csv contain the columns listed under Data Fields. Each row in the dataset is one visit to the store. Because we are predicting the log of the total revenue **per user**, be aware that not all rows in test.csv will correspond to a row in the submission, but **all unique fullVisitorIds will correspond to a row in the submission**.\n\n**IMPORTANT: Due to the formatting o\nf fullVisitorId you must load the Id's as strings in order for all Id's to be properly unique!**\n\nThere are multiple columns which contain JSON blobs of varying depth. In one of those JSON columns, totals, the sub-column transactionRevenue contains the revenue information we are trying to predict. This sub-column exists only for the training data.\n\n**What am I predicting?**\nWe are predicting the natural log of the sum of all transactions per user. \n\n\n**Take Notice**\n - Note that we're predicting the natural log of the total revenue per unique user\n - Please note that the IDs we're using in this competition must be loaded as strings to properly deduplicate / match with BigQuery. Be wary of loading this data in programs like Excel that might also truncate or alter the IDs.\n - In addition to the regular prizes, this competition will have a special prize for R users"},{"metadata":{"trusted":true,"_uuid":"a2cb5011a3d0625b495003ade2d0d9925744f902"},"cell_type":"code","source":"# Main files\ntrain = pd.read_csv(\"../input/train.csv\",\n dtype={'fullVisitorId': 'str'} # Important!!\n )\ntest = pd.read_csv(\"../input/test.csv\",\n dtype={'fullVisitorId': 'str'}, # Important!!\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"376ee8620bfa7fc4f6b8cda850c9d1f90613b0a5"},"cell_type":"code","source":"print('Size of train data', train.shape)\nprint('Size of test data', test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c6974d6f51bf34d1367048a9ffea61d9bfe9989d"},"cell_type":"markdown","source":"

5. Glimpse of Data

"},{"metadata":{"trusted":true,"_uuid":"c0349f633cb90d4d406df14a39c0363cb66369ff"},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e8eb0242d67175cd433f72799b10d9b35dcff64"},"cell_type":"markdown","source":"The training set has 903 653 observations (each one is a visit to the store) and 12 features,"},{"metadata":{"trusted":true,"_uuid":"75e965ac21d0d11af432095f489498dd8e69b1e3"},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5e108c8e2858e8fc411fe7b088527f9d73178409"},"cell_type":"markdown","source":"The test dataset is bit smaller in terms of observations"},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"251cd7df3dcd11d77f02521b2fac55ac2e76ab0e"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c9e97788ff43077b240b7d65932d32eb455d5ea"},"cell_type":"markdown","source":"We can see some features presenting some JSON values. Let's correct that."},{"metadata":{"trusted":true,"_uuid":"eb8170333a51e4bccb524cec53b72c51b77504fd"},"cell_type":"code","source":"def string_to_dict(dict_string):\n # Convert to proper json format, from a string. json.load is for dictionary\n return json.loads(dict_string)\n\njson_cols = [\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"]\n\n\ndef create_new_dataset(df):\n new = df.copy()\n for col in json_cols:\n new[col] = new[col].apply(string_to_dict)\n #the following line will convert the JSON values into series and concatenat to the train dataframe\n new = pd.concat([new,(pd.io.json.json_normalize(new[col]))], axis=1)\n\n new = new.drop([\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"], axis = 1)\n \n return new","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"c2bc7ec9a07349fd2e172d9f5cfd7ddd081cbffe"},"cell_type":"code","source":"new_train = create_new_dataset(train)\nnew_test = create_new_dataset(test)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"6132f639eeb9041bf30c3b5d61ef79d0d98755c3"},"cell_type":"code","source":"new_train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6ce178a2ea7cc245ab6d0d556e1bd9e5cf524667"},"cell_type":"code","source":"pd.value_counts(new_train.dtypes).plot(kind=\"bar\")\nplt.title(\"type of train data\")","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"480f929d5598246c4adc2da9da63cfd684afc087"},"cell_type":"code","source":"new_train.describe()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"c14845a20649186c152fef77f913305b8d361c7e"},"cell_type":"code","source":"# Check for duplicates\nidsUnique = len(set(new_train.sessionId ))\nidsTotal = new_train.shape[0]\nidsDupli = idsTotal - idsUnique\nprint(\"There are \" + str(idsDupli) + \" duplicate sessionId for \" + str(idsTotal) + \" total entries\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c60be2e6bb85674a6714fd599fba03e43d66c53c"},"cell_type":"markdown","source":"I saw there were some columns with only one value. This will have no impact on the prediction so let's find all those columns and drop them."},{"metadata":{"trusted":true,"_uuid":"212485434adb24bfd5a21631c7c6cde91e7e179d"},"cell_type":"code","source":"#Check Unique values\nconstant_column = [col for col in new_train.columns if len(new_train[col].unique()) == 1]\nprint(list(constant_column))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9c1d95489a8a18ba845b4bfc41032b35668015a"},"cell_type":"code","source":"new_train.drop(columns=constant_column,inplace=True)\nnew_test.drop(columns=constant_column,inplace=True)\n\nprint(f'Dropped {len(constant_column)} columns.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ecc00c89efddbac4c052979c75efcc8fe3fd560"},"cell_type":"code","source":"def add_time_features(df):\n df['date'] = pd.to_datetime(df['date'], format='%Y%m%d', errors='ignore')\n df['year'] = df['date'].apply(lambda x: x.year)\n df['month'] = df['date'].apply(lambda x: x.month)\n df['day'] = df['date'].apply(lambda x: x.day)\n df['weekday'] = df['date'].apply(lambda x: x.weekday())\n df['visitStartTime_'] = pd.to_datetime(df['visitStartTime'],unit=\"s\")\n df['visitStartTime_year'] = df['visitStartTime_'].apply(lambda x: x.year)\n df['visitStartTime_month'] = df['visitStartTime_'].apply(lambda x: x.month)\n df['visitStartTime_day'] = df['visitStartTime_'].apply(lambda x: x.day)\n df['visitStartTime_weekday'] = df['visitStartTime_'].apply(lambda x: x.weekday())\n return df\ndate_features = [#\"year\",\"month\",\"day\",\"weekday\",'visitStartTime_year',\n \"visitStartTime_month\",\"visitStartTime_day\",\"visitStartTime_weekday\"]","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"2519c37a063e133e30eb9efc1bce1cf85ea70f00"},"cell_type":"code","source":"add_time_features(new_train)\nadd_time_features(new_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0ddc9aebe1b7bb4d7d2e2e6d3a2337c103e0fbe"},"cell_type":"markdown","source":"

6. Exploratory Data Analysis (EDA)

\n

6.1. Univariate Distribution

\n

6.1.1. Distribution of the target variable : **transactionRevenue**

\n\ntransactionRevenue contains the revenue information we are trying to predict. Total transaction revenue, expressed as the value passed to Analytics multiplied by 10^6 (e.g., 2.40 would be given as 2400000)."},{"metadata":{"trusted":true,"_uuid":"ce8a02d6ac0710d439691aa20ae2b5d80fac6eff"},"cell_type":"code","source":"#Target data is non numerical and has NaN. Remember our target variable has to be the log\nnew_train.transactionRevenue = new_train.transactionRevenue.astype(float)\ntarget= np.log(new_train.transactionRevenue.dropna())\n\n#Histogram\n\nplt.figure(figsize=(12,7))\nsns.distplot(target);\nplt.xlabel(\"Revenue\")\nplt.ylabel(\"Number of Buyers\")\nplt.title(\"Target Distribution\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a51d496aeb7554cdd49bd462d0d9b7bf125ab908"},"cell_type":"code","source":"print (\"Skew is:\", target.skew())\nprint(\"Kurtosis: %f\" % target.kurt())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac10a7917089145865de5e437b8cb0aa7890d6f1"},"cell_type":"markdown","source":"Our target variable appears to have a normal distribution. However, our intended target is the sum of the log transaction revenue per users. Let's see that distribution."},{"metadata":{"trusted":true,"_uuid":"fcf75c6f938144112861dda156252c2390868ce1"},"cell_type":"code","source":"grouped = new_train.groupby('fullVisitorId')[\"transactionRevenue\"].sum().reset_index()\ntarget_group = np.log(grouped.loc[grouped['transactionRevenue'] > 0, 'transactionRevenue'])\n\nplt.figure(figsize=(12,7))\nsns.distplot(target_group);\nplt.xlabel(\"Revenue\")\nplt.ylabel(\"Number of Buyers\")\nplt.title(\"Target Distribution\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"851617fa93a0eacd1af05f57c44db4d61026ed38"},"cell_type":"code","source":"print (\"Skew is:\", target_group.skew())\nprint(\"Kurtosis: %f\" % target_group.kurt())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bbc82d04fd11abfb380b22be83d63b07cdbb6b2f"},"cell_type":"markdown","source":"Both distributions are quite similar."},{"metadata":{"_uuid":"149524769a075266460b6ea363eabeacd60ffc60"},"cell_type":"markdown","source":"

6.1.1.1. Target variable : **transactionRevenue** presents NaN values

\n\nDuring my analysis I noticed that the target variable had null values for the training dataset. Assuming that each row corresponds to a visit to the website and not necessary a purchase, it is normal to have rows with null values. "},{"metadata":{"trusted":true,"_uuid":"9cf7a23039a57a7a78b9d8158c15402b9d566869"},"cell_type":"code","source":"#https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue\n\nnzi = pd.notnull(new_train[\"transactionRevenue\"]).sum()\nnzr = (grouped[\"transactionRevenue\"]>0).sum()\nprint(\"Number of instances in train set with non-zero revenue : \", nzi, \" and ratio is : \", nzi / new_train.shape[0])\nprint(\"Number of unique customers with non-zero revenue : \", nzr, \"and the ratio is : \", nzr / grouped.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"458d9c58d510ed1463c9055cf57123339ecba6d8"},"cell_type":"markdown","source":"Around 1.3% of customers generated revenues."},{"metadata":{"_uuid":"2cf2b70e3ec5dc17ed0a8c32028e7f5d01b523a4"},"cell_type":"markdown","source":"

6.1.2. Numerical Distribution

"},{"metadata":{"trusted":true,"_uuid":"704e8c76980a685ea5aa7957ca09a5851b3f5eef"},"cell_type":"code","source":"numeric_features = new_train.select_dtypes(include=[np.number])\nnumeric_features.dtypes","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"21d502728601f1ee9d59fbb2077af5c7d7bb1673"},"cell_type":"markdown","source":"

6.1.2.1 Distribution of **date**

\nThe date on which the user visited the Store. The date of the session in YYYYMMDD format."},{"metadata":{"trusted":true,"_uuid":"ca5ba03df6fcc7b31d16c97cd735812500843333"},"cell_type":"code","source":"#Original code from : https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue\n\nfrom plotly import tools\n\ndef scatter_plot(cnt_srs, color):\n trace = go.Scatter(\n x=cnt_srs.index[::-1],\n y=cnt_srs.values[::-1],\n showlegend=False,\n marker=dict(\n color=color,\n ),\n )\n return trace\n\ncnt_srs = new_train.groupby('date')['transactionRevenue'].agg(['size', 'count'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\"]\ncnt_srs = cnt_srs.sort_index()\n#cnt_srs.index = cnt_srs.index.astype('str')\ntrace1 = scatter_plot(cnt_srs[\"count\"], 'red')\ntrace2 = scatter_plot(cnt_srs[\"count of non-zero revenue\"], 'blue')\n\nfig = tools.make_subplots(rows=2, cols=1, vertical_spacing=0.08,\n subplot_titles=[\"Date - Count\", \"Date - Non-zero Revenue count\"])\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 2, 1)\nfig['layout'].update(height=800, width=800, paper_bgcolor='rgb(233,233,233)', title=\"Date Plots\")\npy.iplot(fig, filename='date-plots')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c11a9738c62e5a5d34a30dd1b14e8e4793a7943"},"cell_type":"markdown","source":"

6.1.2.2 Distribution of **VisitId**

\nAn identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId."},{"metadata":{"trusted":true,"_uuid":"d61ba226075fbe1adcd4e4f5e7af39216eda110d"},"cell_type":"code","source":"new_train.visitId.value_counts()[:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"19febe146e5f4ab2a1507371ebec422ccc4ee7d8"},"cell_type":"markdown","source":"

6.1.2.3 Distribution of **visitNumber**

\nThe session number for this user. If this is the first session, then this is set to 1."},{"metadata":{"trusted":true,"_uuid":"d5562da2c239ee9125c6d8929a535f4a30a5b148"},"cell_type":"code","source":"new_train.visitNumber.value_counts()[:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f9124303f12ac0c1854e320ff394ae502cb521e"},"cell_type":"markdown","source":"

6.1.2.4 Distribution of **visitStartTime**

\nThe timestamp (expressed as POSIX time)."},{"metadata":{"trusted":true,"_uuid":"e0a850fe94e90e7eef461f157f7b4cdd32459e66"},"cell_type":"code","source":"new_train.visitStartTime.value_counts()[:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca2a2755387a1f10eae2ba6d883f6ff296d8176d"},"cell_type":"markdown","source":"

6.1.3. Categorical Distribution

"},{"metadata":{"trusted":true,"_uuid":"bf8134e21002c5b29da490549550d3a83d596255"},"cell_type":"code","source":"categorical_features = new_train.select_dtypes(exclude=[np.number])\ncategorical_features.dtypes","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6b272205a7bad8f9d52b251efd2c9ea2b0f4b301"},"cell_type":"markdown","source":"

6.1.3.1 Distribution of **channelGrouping**

\nThe channel via which the user came to the Store. The Default Channel Group associated with an end user's session for this View."},{"metadata":{"trusted":true,"_uuid":"6c954da17a901ee91c84faae390359c87accda32"},"cell_type":"code","source":"sns.countplot(new_train.channelGrouping,order=new_train.channelGrouping.value_counts().iloc[0:10].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"904e12532f4c951f3762b5f29142ae4d948e6465"},"cell_type":"markdown","source":"

6.1.3.2 Distribution of **fullVisitorId**

"},{"metadata":{"trusted":true,"_uuid":"9d4c8fdd7d7b8729ef762a254cb44629e74dc298"},"cell_type":"code","source":"new_train.fullVisitorId.value_counts()[:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"71b6629c8608cb5ae84767dc1796852f6e8d231d"},"cell_type":"markdown","source":"

6.1.3.3 Distribution of **sessionId**

\nA unique identifier for each user of the Google Merchandise Store.The unique visitor ID (also known as client ID)."},{"metadata":{"trusted":true,"_uuid":"679c987c158cbde9d59176eb2525f31ab4b1e9e3"},"cell_type":"code","source":"new_train.sessionId.value_counts()[:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d4f2e46754a98f5c23569394999de92bb259a122"},"cell_type":"markdown","source":"

6.1.3.4 Distribution of **browser**

\nThe browser used (e.g., \"Chrome\" or \"Firefox\")."},{"metadata":{"trusted":true,"_uuid":"19ec9add0afbbefbea623c7c86ea220633fe0cfc"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.browser,order=new_train.browser.value_counts().iloc[0:10].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9b42cbf991de58875edac37625138961906519cd"},"cell_type":"markdown","source":"

6.1.3.5 Distribution of **deviceCategory**

\nThe type of device (Mobile, Tablet, Desktop)"},{"metadata":{"trusted":true,"_uuid":"f93286c32819d2135ed6f1fb42f31455cec7c336"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.deviceCategory)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d07015b102d1e47a97c7b7b15c34b7fac5f7f05"},"cell_type":"markdown","source":"

6.1.3.6 Distribution of **isMobile**

\nIf the user is on a mobile device, this value is true, otherwise false."},{"metadata":{"trusted":true,"_uuid":"8884860e50b35368ccdd3f6e3b70649ead521388"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.isMobile)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7802a0218ac1d520a089e411c4056ee948cfbaa"},"cell_type":"markdown","source":"

6.1.3.7 Distribution of **operatingSystem**

\nThe operating system of the device (e.g., \"Macintosh\" or \"Windows\")."},{"metadata":{"trusted":true,"_uuid":"3602085906589a71a387c17c14f410ac772fa3c5"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.operatingSystem,order=new_train.operatingSystem.value_counts().iloc[0:10].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b4bb75ad2cadf394a83019f0e0fb0003aab07d6"},"cell_type":"markdown","source":"

6.1.3.8 Distribution of **city**

\nUsers' city, derived from their IP addresses or Geographical IDs."},{"metadata":{"trusted":true,"_uuid":"034505a64c13e85835ec8650f48e4f8f364efea3"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.city,order=new_train.city.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9fae42e6d196a67bcbfac4c96bc722b9525ef777"},"cell_type":"code","source":"plt.figure(figsize=(18,7))\nsns.countplot(new_train.city,order=new_train.city.value_counts().iloc[1:100].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"51dc8b94650b39dd78e7273dfd807741ead1efa4"},"cell_type":"markdown","source":"

6.1.3.9 Distribution of **Continent**

\nThe continent from which sessions originated, based on IP address."},{"metadata":{"trusted":true,"_uuid":"22b0d68bbc3978ece2701ba66879d51f8cdd4b59"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.continent,order=new_train.continent.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c24035618e7aeb89acfa96d656a81c0de61bd88b"},"cell_type":"markdown","source":"

6.1.3.10 Distribution of **country**

\nThe country from which sessions originated, based on IP address."},{"metadata":{"trusted":true,"_uuid":"daa452acb3c24d5f527460e3e3e305fb366005cc"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.country,order=new_train.country.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b2361aecad56bb262c20c5ad8080b956df0344aa"},"cell_type":"markdown","source":"

6.1.3.11 Distribution of **metro**

\n\nThe Designated Market Area (DMA) from which sessions originate."},{"metadata":{"trusted":true,"_uuid":"3de638e663c6c73e6096226020de8b6f8dd67f3c"},"cell_type":"code","source":"plt.figure(figsize=(18,7));\nsns.countplot(new_train.metro,order=new_train.metro.value_counts().iloc[:50].index);\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e268beaee6b8bf7f485eaa319b4d816d228d85e1"},"cell_type":"code","source":"plt.figure(figsize=(18,7));\nsns.countplot(new_train.metro,order=new_train.metro.value_counts().iloc[2:50].index);\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"246181ca4874c0c67d1e545a7c48dbfaec67e5fd"},"cell_type":"markdown","source":"

6.1.3.12 Distribution of **networkDomain**

\nThe domain name of user's ISP, derived from the domain name registered to the ISP's IP address."},{"metadata":{"trusted":true,"_uuid":"2a5274313aa8bafd8de2e47c7aff41e23f84c497"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.networkDomain,order=new_train.networkDomain.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d75cb312216c72525cc7bb7046e03fdec987ea8"},"cell_type":"markdown","source":"

6.1.3.13 Distribution of **region**

\nThe region from which sessions originate, derived from IP addresses. In the U.S., a region is a state, such as New York."},{"metadata":{"trusted":true,"_uuid":"b823cc7fffdb91ddb7b578a0ac4a70602e481518"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.region,order=new_train.region.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0268608718821ebf44d52de158ad548051cd2dfe"},"cell_type":"markdown","source":"

6.1.3.14 Distribution of **subContinent**

\nThe sub-continent from which sessions originated, based on IP address of the visitor."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"aa670b112d61f3069cd8bc49b999d6523fa3f103"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.subContinent,order=new_train.subContinent.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36d5a206e9538bb9b6dc7243f88530e5b312c60b"},"cell_type":"markdown","source":"

6.1.3.15 Distribution of **bounces**

\nTotal bounces (for convenience). For a bounced session, the value is 1, otherwise it is null."},{"metadata":{"trusted":true,"_uuid":"1e40b3ae0c6d03dba89d15ff02705a77497e86f4"},"cell_type":"code","source":"sns.countplot(new_train.bounces)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b603a8c82d384965a8079da8789092895ef5915b"},"cell_type":"code","source":"new_train.bounces.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2a6123fd0b69a0cabf9936d3291e7d62a6b13048"},"cell_type":"code","source":"new_train.bounces.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d867fbb6c5d22b87c034e1c8bef6b303e7eed47"},"cell_type":"markdown","source":"

6.1.3.16 Distribution of **hits**

\nTotal number of hits within the session.\nA hit is one download of one file from your web site. So, if your home page is comprised of three images, a style sheet (style.css, for example) and the page itself (let’s say index.html), then every time a browser downloads that page, four hits are recorded."},{"metadata":{"trusted":true,"_uuid":"abc887bfd3e8c1793a5941141318f127351fcd25"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.hits,order=new_train.hits.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be9928f5c0cc62c73226d1f1f62898e7ac218d52"},"cell_type":"markdown","source":"

6.1.3.17 Distribution of **newVisits**

\nTotal number of new users in session (for convenience). If this is the first visit, this value is 1, otherwise it is null."},{"metadata":{"trusted":true,"_uuid":"af9cdd1da550ad7abc7de022b07e0c400137a256"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.newVisits,order=new_train.newVisits.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a61a70f98b9519c14157599bdf39bf340450de97"},"cell_type":"code","source":"new_train.newVisits.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3c8f735b1d9ed085bd4a57b3c15e088333161c64"},"cell_type":"code","source":"new_train.newVisits.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a684f39d161df3dedf062be65f3a3070aeb239c9"},"cell_type":"markdown","source":"

6.1.3.18 Distribution of **pageviews**

\nTotal number of pageviews within the session."},{"metadata":{"trusted":true,"_uuid":"ee0874479f8f57c5d0dcea866f671e133edb07c1"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.pageviews,order=new_train.pageviews.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61e266898e97fd3aa3e6b7f17db0635f4729e1b0"},"cell_type":"markdown","source":"

6.1.3.19 Distribution of **adContent**

\nThe ad content of the traffic source. Can be set by the utm_content URL parameter."},{"metadata":{"trusted":true,"_uuid":"581e743600541349f70c07cd951319912cfaf86c"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.adContent,order=new_train.adContent.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0e8236a3464b1509a75547d8d72169e3716f9875"},"cell_type":"markdown","source":"

6.1.3.20 Distribution of **adwordsClickInfo.adNetworkType**

\nNetwork Type. Takes one of the following values: {“Google Search\", \"Content\", \"Search partners\", \"Ad Exchange\", \"Yahoo Japan Search\", \"Yahoo Japan AFS\", “unknown”}"},{"metadata":{"trusted":true,"_uuid":"6c900c9a64d0aa00014c9b7dacf98312c748ef20"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train[\"adwordsClickInfo.adNetworkType\"])\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f2f447b3104d36d0d2ace3ff32a314dce65b776f"},"cell_type":"code","source":"new_train[\"adwordsClickInfo.adNetworkType\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bf66aa06f777d2ebb639f1806b59b49bfe0394d9"},"cell_type":"code","source":"new_train[\"adwordsClickInfo.adNetworkType\"].isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa1975f95861bb25f43f9ef372f5440e4f25e7a8"},"cell_type":"markdown","source":"

6.1.3.21 Distribution of **adwordsClickInfo.gclId**

\nThe Google Click ID"},{"metadata":{"trusted":true,"_uuid":"7df58c89d2354c789a58c449619a8e7b3394f908"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train[\"adwordsClickInfo.gclId\"],order=new_train[\"adwordsClickInfo.gclId\"].value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0aa89ab604d374fec9239a7ddf6355fded7e9e69"},"cell_type":"markdown","source":"

6.1.3.22 Distribution of **adwordsClickInfo.isVideoAd**

\nTrue if it is a Trueview video ad."},{"metadata":{"trusted":true,"_uuid":"9dbd8260af1d4226054f5d3047593057f8b892ae"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train[\"adwordsClickInfo.isVideoAd\"],order=new_train[\"adwordsClickInfo.isVideoAd\"].value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"01dbda2917b663299e898a9d79d331fcc3857497"},"cell_type":"code","source":"new_train[\"adwordsClickInfo.isVideoAd\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"88157f00b654443187d75443ca296cc15424069f"},"cell_type":"code","source":"new_train[\"adwordsClickInfo.isVideoAd\"].isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3c6cb8cec06cc54df1c35de4ac26ed24b5fbfed4"},"cell_type":"markdown","source":"

6.1.3.23 Distribution of **adwordsClickInfo.page**

\nPage number in search results where the ad was shown"},{"metadata":{"trusted":true,"_uuid":"093fd4d3d6c28dc24d0d59e0cc510d4a1de2e345"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train[\"adwordsClickInfo.page\"],order=new_train[\"adwordsClickInfo.page\"].value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e0c95838a434fd3038fc692c714d5ca5354e067"},"cell_type":"markdown","source":"

6.1.3.24 Distribution of **adwordsClickInfo.slot**

\nPosition of the Ad. Takes one of the following values:{“RHS\", \"Top\"}"},{"metadata":{"trusted":true,"_uuid":"2b4c45444b2a167113d910e3229f418b655f98a8"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train[\"adwordsClickInfo.slot\"],order=new_train[\"adwordsClickInfo.slot\"].value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5320916234347527ab9808a30be6a83b0b815a16"},"cell_type":"markdown","source":"

6.1.3.25 Distribution of **campaign**

\nThe campaign value. Usually set by the utm_campaign URL parameter."},{"metadata":{"trusted":true,"_uuid":"7c59abf8178ef8d273fc62b3757d3ac4227b0378"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.campaign,order=new_train.campaign.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5506efb00d0a85f655f207752263e3dc76f05091"},"cell_type":"markdown","source":"

6.1.3.26 Distribution of **campaignCode**

\nValue of the utm_id campaign tracking parameter, used for manual campaign tracking."},{"metadata":{"trusted":true,"_uuid":"8580a51d93c7669932b4df9c1d281b7dd6960bdb"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.campaignCode,order=new_train.campaignCode.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"67e014469d3631b62fd586209be70ad4784774db"},"cell_type":"code","source":"new_train.campaignCode.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"27ad721ea9a92d8a17787be6ad23afcaeb121dfb"},"cell_type":"code","source":"new_train.campaignCode.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9e5581b84ae61c49c9d14d98d840515e68a82899"},"cell_type":"markdown","source":"

6.1.3.27 Distribution of **isTrueDirect **

\nTrue if the source of the session was Direct (meaning the user typed the name of your website URL into the browser or came to your site via a bookmark), This field will also be true if 2 successive but distinct sessions have exactly the same campaign details. Otherwise NULL."},{"metadata":{"trusted":true,"_uuid":"999edb67e6543ce1627e27c5183e124fca112674"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.isTrueDirect,order=new_train.isTrueDirect.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"8f83c5346573707b97762aae01e54b989a4e7de8"},"cell_type":"code","source":"new_train.isTrueDirect.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"f11b5df4949c944692329ca0cbf07ca1406aeab5"},"cell_type":"code","source":"new_train.isTrueDirect.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2ee286e3384d4907396b5b3ff1c4b973d909c8a6"},"cell_type":"markdown","source":"

6.1.3.28 Distribution of **keyword**

\nThe keyword of the traffic source, usually set when the trafficSource.medium is \"organic\" or \"cpc\". Can be set by the utm_term URL parameter."},{"metadata":{"trusted":true,"_uuid":"ca832e2509025ae445a104f4960779c70e3f9928"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.keyword,order=new_train.keyword.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a9c668d7409b2ff68121f332d455e3da53244ea7"},"cell_type":"markdown","source":"

6.1.3.29 Distribution of **medium**

\nThe medium of the traffic source. Could be \"organic\", \"cpc\", \"referral\", or the value of the utm_medium URL parameter."},{"metadata":{"trusted":true,"_uuid":"9d24e1bb8b7753c0a17c1eea8362391f4f15a30f"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.medium,order=new_train.medium.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50b41e8ba3ba80dbd43c62ac5288099e2c336f6d"},"cell_type":"markdown","source":"

6.1.3.30 Distribution of **referralPath**

\nIf trafficSource.medium is \"referral\", then this is set to the path of the referrer. (The host name of the referrer is in trafficSource.source.)"},{"metadata":{"trusted":true,"_uuid":"09651ff50f6c569ae78b88d2e74bced253022efc"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.referralPath,order=new_train.referralPath.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4fe64c20fa931c803d7b4b6d4097dcedea92a09"},"cell_type":"markdown","source":"

6.1.3.31 Distribution of **source**

\nThe source of the traffic source. Could be the name of the search engine, the referring hostname, or a value of the utm_source URL parameter."},{"metadata":{"trusted":true,"_uuid":"ad20b9a47a4dfdd85027f88d8b989fa63d6993fd"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.source,order=new_train.source.value_counts().iloc[:50].index)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b3330a6574016281f3082f01a772ab9860ba6ad"},"cell_type":"markdown","source":"

6.1.3.32 Distribution of new Date Features

"},{"metadata":{"trusted":true,"_uuid":"57f295bd95f3bba734377601bd8fd0e67f3a2445"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.year)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ae3d6000def84120a3b766975cf9e30ff747c6c2"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.month)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8f55a121937ec1ca390e4ae262d8d7a6ff1f48fa"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.day)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"52e0a3307fd6ba380c7282df27e97368271a0334"},"cell_type":"code","source":"plt.figure(figsize=(12,7))\nsns.countplot(new_train.weekday)\nplt.xticks(rotation=90);\n#{0: 'Monday', 1: 'Tuesday', 2: 'Wednesday', 3: 'Thursday', 4: 'Friday', 5: 'Saturday', 6: 'Sunday'}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8201b614de754bf26dbce6cc6d471a9d9741f13"},"cell_type":"markdown","source":"

6.1.3.33 Univariate Analysis Conclusions

\n\n### General Overview\n\nWe have 50 types of Object data that must be converted into another type.\nThere is 1 BOOL type of data feature that needs to be converted.\nThere was 19 features which only presented a single values. These were dropped.\nCreated new time_features (year, month, day, weekday, etc)\n\n\n### Exploratory Data Analysis\n\n- **Target Variable** : log value of the target presents a normal distribution. Skewness of 0.33\n\n**Numerical Distribution**\n- **Date** : it appears that the sales tend to increase from November to December, as expected. Around the year sales are constant\n- **VisitId** : For a completely unique ID, we should use a combination of fullVisitorId and visitId.\n- **visitNumber** : Most users only have one session, the first session. I believe this means that most users visit once the website and don't return back. At least, as if they do not clean their cookies. **How many session does a user need to buy something?**\n- **visitStartTime** : This feature is expressed as POSIX time and need to be converted. However, is this feature worth having?\n\n#### Categorical Distribution\n\n#### Related to device\n- **channelGrouping** : Most users access the site through a Organic Search (Google, Yahoo) and Social Media (Facebook, Twitter). The direct access and referral also have high number. Paid Search and Affiliates does not bring much consumers. Are they worth to have? **See the origin which is bringing most spenders!**\n- **fullVisitorId** : For a completely unique ID, we should use a combination of fullVisitorId and visitId.\nIMPORTANT: Due to the formatting of fullVisitorId you must load the Id's as strings in order for all Id's to be properly unique!\n- **sessionId** : Don't believe is relevant...\n- **browser** : Most used browsers are Chrome, Safari and Firefox. **Do users using Chrome spend more?**\n- **deviceCategory** : Desktop users dominate purchases. Following is mobile and tablet. This makes sense, for me, since I also do prefer doing purchases online on the computer.\n- **isMobile** : Most users do not use Mobile as seen before. This needs to be converted to a label feature instead of Boolean.\n- **operatingSystem** : Windows, Mac, Android,iOS, Linux and Chrome OS are the only OS with impact. **Should we join all other in a single category such as \"Others\"?**\n\n\n#### Related to geoNetwork\n\n- **city** : There are a lot of Users that do not have the IP of their city or the value Not Set. There area lot of users from the area of California (Mountain View, San Francisco, San Jose, Los Angeles).\n- **continent** : Most Sales go to the Americas, Asia and Europe.\n- **country** : The USA most be the biggest delegate of purchases from the American continent. Regarding Asia, India and Vietnam and Thailand seem to be the the representatives. On the other hand, for Europe UK, Turkey and Germany.\n- **metro** : This features seems a repetition of the country and city.. However it has different values, per city.\n- **networkDomain** : Don't see how this feature might influence the purchase value\n- **region** : Seems to present the same results as metro\n- **subContinent** : Same results as Continent\n- **sessionId** :\n- **socialEngagementType** :\n\n#### Related to totals\n\n- **bounces** : There are 450 630 bounce sessions with value 1.0 and 453 023 with NaN. **Should the NaN have value zero?**\n- **hits** : Most sessions have one hit until 5. **Does anyone can take more info from these?**\n- **newVisits** 703 060 have value 1.0 whereas 200 593 have NaN. **Convert NaN to zero? Moreover, see the tendency of new vs old visits purchase**\n- **pageviews** : Most people appear to only visit one page. Does it mean they stay at the homepage or just look at the one they were redirected to? **By increasing the nr of pages surfed, so does the probability of buying?**\n\n#### Related to trafficSource\n- **adContent** The majority of purchase is related to Google Merchandise Collection and Google Online Store.\n- **adwordsClickInfo.adNetworkType** : Only 7 values are not Google Search from a total of 21453. The rest, 882193 observations have the value NaN\n- **adwordsClickInfo.gclId** : 21460 present the value False and 882193 the value NaN. Similar to adNetworkType.\n- **adwordsClickInfo.isVideoAd**\n- **adwordsClickInfo.page**\n- **adwordsClickInfo.slot** Two values considered. **Convert to label categories**\n- **campaign** most observations do to have a value set\n- **campaignCode** **only 1 value is not NaN**. So we can drop this column.\n- **isTrueDirect** 274 005 sessions accessed the website by typing the URL. The rest is NaN\n- **keyword**\n- **medium**\n- **referralPath**\n- **source**\n\n\n"},{"metadata":{"_uuid":"c73b57e4477096d7b8eaa883ca6ab6308013d228"},"cell_type":"markdown","source":"

6.2. Bivariate Distribution

\nThese analysis was done using the code of user @SRK"},{"metadata":{"trusted":true,"_uuid":"5410b4bd065b9cc1a21c7c22610bbcaa681bd037"},"cell_type":"code","source":"def horizontal_bar_chart(cnt_srs, color):\n trace = go.Bar(\n y=cnt_srs.index[::-1],\n x=cnt_srs.values[::-1],\n showlegend=False,\n orientation = 'h',\n marker=dict(\n color=color,\n ),\n )\n return trace","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"577936f1fa9ff7f1d5c65da936ae7bc2055c478d"},"cell_type":"markdown","source":"

6.2.1. VisitNumber vs Target

\nHow many session does a user need to buy something?"},{"metadata":{"trusted":true,"_uuid":"8e1ee6438b63831f0adba2f8cceb57c45efe1426"},"cell_type":"code","source":"cnt_srs = new_train.groupby('visitNumber')['transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\n\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace5 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\nfig = tools.make_subplots(rows=1, cols=3, \n vertical_spacing=0.04, \n subplot_titles=[\"VisitNumber - Count\",\n \"VisitNumber - Non-zero Revenue Count\", \n \"VisitNumber - Mean Revenue\"]);\n\nfig.append_trace(trace3, 1, 1)\nfig.append_trace(trace4, 1, 2)\nfig.append_trace(trace5, 1, 3)\nfig['layout'].update(height=600, width=1200, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a639b23853a60348f0660b1afd14d70702f6bfee"},"cell_type":"markdown","source":"

6.2.2. ChannelGrouping vs Target

\nSee the origin which is bringing most spenders!"},{"metadata":{"trusted":true,"_uuid":"49b52d8b38e818c026ba3943e22b43951faa0de1"},"cell_type":"code","source":"cnt_srs = new_train.groupby('channelGrouping')['transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\n\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace5 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\nfig = tools.make_subplots(rows=1, cols=3, \n vertical_spacing=0.04, \n subplot_titles=[\"Channel Grouping - Count\",\n \"Channel Grouping - Non-zero Revenue Count\", \n \"Channel Grouping - Mean Revenue\"]);\n\nfig.append_trace(trace3, 1, 1)\nfig.append_trace(trace4, 1, 2)\nfig.append_trace(trace5, 1, 3)\nfig['layout'].update(height=600, width=1200, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4d895b1cc4ac7c9a22275212bd8cc363a945f9a"},"cell_type":"markdown","source":"

6.2.3. Browser vs Target

\nDo users using Chrome spend more?"},{"metadata":{"trusted":true,"_uuid":"969bf26aa0ce6eb35c1420d0c95758f578a469bc"},"cell_type":"code","source":"# Device Browser\n\n#size includes NaN values, count does not:\ncnt_srs = new_train.groupby('browser')['transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\n\ntrace1 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace2 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace3 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\nfig = tools.make_subplots(rows=1, cols=3, \n vertical_spacing=0.04, \n subplot_titles=[\"Device Browser - Count\",\n \"Device Browser - Non-zero Revenue Count\", \n \"Device Browser - Mean Revenue\"]);\n\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 1, 2)\nfig.append_trace(trace3, 1, 3)\nfig['layout'].update(height=600, width=1200, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22c8b3d9b0b62d5720d0611be3ba60f57d9792da"},"cell_type":"markdown","source":"

6.2.4. deviceCategory vs Target

"},{"metadata":{"trusted":true,"_uuid":"46ba7462925ed44ec1c13920fc929b1e44dd8bff"},"cell_type":"code","source":"cnt_srs = new_train.groupby('deviceCategory')['transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\n\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace5 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\nfig = tools.make_subplots(rows=1, cols=3, \n vertical_spacing=0.04, \n subplot_titles=[\"Device Category - Count\",\n \"Device Category - Non-zero Revenue Count\", \n \"Device Category - Mean Revenue\"]);\n\nfig.append_trace(trace3, 1, 1)\nfig.append_trace(trace4, 1, 2)\nfig.append_trace(trace5, 1, 3)\nfig['layout'].update(height=600, width=1200, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots');\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"83ade65f7e0769d82ca8f8f03f3347f4c17ec156"},"cell_type":"markdown","source":"

6.2.5. Device OS vs Target

"},{"metadata":{"trusted":true,"_uuid":"bb49f041e4233dc880d6dd4efd4fc18d811dac07"},"cell_type":"code","source":"cnt_srs = new_train.groupby('operatingSystem')['transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\n\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace5 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\nfig = tools.make_subplots(rows=1, cols=3, \n vertical_spacing=0.04, \n subplot_titles=[\"Device OS - Count\",\n \"Device OS - Non-zero Revenue Count\", \n \"Device OS - Mean Revenue\"]);\n\nfig.append_trace(trace3, 1, 1)\nfig.append_trace(trace4, 1, 2)\nfig.append_trace(trace5, 1, 3)\nfig['layout'].update(height=600, width=1200, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7177ad10488e871948a176516713846c2e9d2b9e"},"cell_type":"markdown","source":"

6.2.6. Pageviews vs Target

"},{"metadata":{"trusted":true,"_uuid":"9f467893e3ce5834f72f5bd6f0e4d8e5401dc417"},"cell_type":"code","source":"cnt_srs = new_train.groupby('pageviews')['transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\n\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace5 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\nfig = tools.make_subplots(rows=1, cols=3, \n vertical_spacing=0.04, \n subplot_titles=[\"Page Views - Count\",\n \"Page Views - Non-zero Revenue Count\", \n \"Page Views - Mean Revenue\"]);\n\nfig.append_trace(trace3, 1, 1)\nfig.append_trace(trace4, 1, 2)\nfig.append_trace(trace5, 1, 3)\nfig['layout'].update(height=600, width=1200, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"06ead5135fa7b9053a88e174e3b4cd4d1678f9a4"},"cell_type":"markdown","source":"

6.2.7. Continent vs Target

"},{"metadata":{"trusted":true,"_uuid":"65bb84c8fe4851c8deacd0c039fa615079bdf777"},"cell_type":"code","source":"cnt_srs = new_train.groupby('continent')['transactionRevenue'].agg(['size', 'count', 'mean'])\ncnt_srs.columns = [\"count\", \"count of non-zero revenue\", \"mean\"]\ncnt_srs = cnt_srs.sort_values(by=\"count\", ascending=False)\n\ntrace3 = horizontal_bar_chart(cnt_srs[\"count\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace4 = horizontal_bar_chart(cnt_srs[\"count of non-zero revenue\"].head(10), 'rgba(50, 171, 96, 0.6)')\ntrace5 = horizontal_bar_chart(cnt_srs[\"mean\"].head(10), 'rgba(50, 171, 96, 0.6)')\n\nfig = tools.make_subplots(rows=1, cols=3, \n vertical_spacing=0.04, \n subplot_titles=[\"Continent - Count\",\n \"Continent - Non-zero Revenue Count\", \n \"Continent - Mean Revenue\"]);\n\nfig.append_trace(trace3, 1, 1)\nfig.append_trace(trace4, 1, 2)\nfig.append_trace(trace5, 1, 3)\nfig['layout'].update(height=600, width=1200, paper_bgcolor='rgb(233,233,233)', title=\"Device Plots\")\npy.iplot(fig, filename='device-plots');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5bbf087e1f65b864609378c2b5c016affb989b36"},"cell_type":"markdown","source":"

6.2.8. **Bivariate Analysis Conclusions**

\n\n**VisitNumber vs Target**\nHow many session does a user need to buy something?\n\n- Most users visited one time the website;\n- From those who purchased at least an item, the visit number count distribution is the same. There is a high number of users that only visited once the website and purchased an item;\n- Nevertheless, the more visits a user makes to website the more he spends. For example, from a single visit to seven visits the number of revenue increased. However 8 visits present a higher mean revenue than 9 or 10.\n- **In the end, users need at least one visit to purchase something and the biggest spenders are the one that on average visit the website 8 times.**\n\n\n**ChannelGrouping vs Target**\nSee the origin which is bringing most spenders!\n\n- Organic Search, Social, Direct and Referral bring most of the clients to the store\n- When looking only to users that purchased something, Social has small impact on revenue. Moreover, although Organic Search and Direct still bring buyers it is Referral that brings most of the buyers.\n- **Amazingly, although Display does not brin many people to the website it does bring those who spend more on average! Following is Direct as expected, Referral, Paid Search and Organic Search.**\n\n\n**Browser vs Target**\nDo users using Chrome spend more?\n\n- Most users use Chrome, Safari and Firefox\n- The majority of users that made purchase came from the same three Browsers with equal distribution. More purchases from Chrome, Safari and Firefox respectively.\n- **Biggest spenders use Firefox, followed by Chrome, Safari, Edge and IE. Therefore chrome users do not spend more.**\n\n**Browser vs Target**\n- Most people use the computer to buy articles and the majority of revenues comes from this device category.\n\n\n**Device OS vs Target**\n- The rank of the majority of users goes as follows: Windows, Mac, Android, iOS, Linux and Chrome OS.\n- The rank for people who purchase something is : Mac, Windows, ChromeOS, Linux, iOS, Android\n- **Biggest buyers are ChromeOS users! The rest goes as follow: Windows,Mac, Android, iOS, Linux, Windows Phone.**\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}