{"cells":[{"metadata":{"_uuid":"a297ba3cb007f49b8c3f2c7202c6a098ee90b654"},"cell_type":"markdown","source":"# Google Analytics Customer Revenue Prediction\n\nThis notebook is made to document the steps made to create an algorithm able to predict the natural log of the transaction revenue of some customers of a GStore. The whole description can be found at https://www.kaggle.com/c/ga-customer-revenue-prediction. "},{"metadata":{"_uuid":"9e71c3f7972ec054f28ccdd7d9aab563f4571e7f"},"cell_type":"markdown","source":"First of all, we have to include the whole set of libraries we need to achieve our result."},{"metadata":{"trusted":true,"_uuid":"0d554cb5575d1b9a3a8eaccb9690df648cb14e88"},"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn import preprocessing\nfrom sklearn import neighbors\nfrom pandas.io.json import json_normalize\n%matplotlib notebook\nimport matplotlib.pyplot as plot","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bfe1679734fb63c90825e8c988db3b2ca109fe97"},"cell_type":"markdown","source":"Then, we need to import the files where our data are collected. It is important to notice that some columns of the dataset are in a JSON format. These columns contain several informations that can't be read properly. In order to make them available to our machine learning algorithm, we need to append to the dataset an additional column for each information contained in each JSON column. In this notebook the function used to make it possible is taken from a Kaggle kernel made my Julian Peller (https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook)."},{"metadata":{"trusted":true,"_uuid":"b47f3b0023f316ed9008f6c04dc2151207a31203"},"cell_type":"code","source":"def load_df(csv_path='../input/train_v2.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5ca9617d88b98a1dc21ed0aaced4c306fcfccfb7"},"cell_type":"markdown","source":"Now we are ready to retrieve the needed information in a proper way."},{"metadata":{"trusted":true,"_uuid":"5d686457c439d4937cd8a9ea8a91498516543d18"},"cell_type":"code","source":"ga_customers_train = load_df(nrows=200000)\nga_customers_test = load_df(csv_path='../input/test_v2.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33ae703dc269140fe340f9377c2b4e25c08ddf3a"},"cell_type":"markdown","source":"Let's have a look at our data: how they look like?"},{"metadata":{"trusted":true,"_uuid":"d3f62ad2b887a1a2b5ebd47b6822a70b9d340465"},"cell_type":"code","source":"ga_customers_train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6cc0f9f818294ff0c3b49a08b3b5c19aa9b6febd"},"cell_type":"markdown","source":"Since we don't need information contained in columns named 'customDimensions' and 'hits' (I don't even know what they represent) we can drop them from the dataframe. Also, in the train set there are some fields that contain constant values that aren't useful for our purpose because they don't give us any additional information. So, in order to speed up the algorithm execution and to save time and memory, it is better to just drop them from the train dataframe (and also from the test one, because we can't train our model on those features, so those are unuseful). "},{"metadata":{"trusted":true,"_uuid":"5fe1389508ca02c66ecdfbdb0828890ecc63bdc5"},"cell_type":"code","source":"ga_customers_train = ga_customers_train.drop(columns=['customDimensions', 'hits'])\nga_customers_test = ga_customers_test.drop(columns=['customDimensions', 'hits'])\nconst_cols = [c for c in ga_customers_train.columns if ga_customers_train[c].nunique(dropna=False)==1 ]\nga_customers_train = ga_customers_train.drop(columns=const_cols)\nga_customers_test = ga_customers_test.drop(columns=const_cols)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e549e68fe45263ec940d30ed67bcdb6089a07d03"},"cell_type":"markdown","source":"Since we are predicting the natural log of the sum of all transactions per user, let's sum up all transactions per user and the compute their natural log."},{"metadata":{"trusted":true,"_uuid":"f4771502fd72dcdcb5375caba6d0aeb121fff02d"},"cell_type":"code","source":"ga_customers_train['totals.transactionRevenue'] = ga_customers_train['totals.transactionRevenue'].astype('float')\ntotal_transactions_per_user = ga_customers_train.groupby('fullVisitorId')['totals.transactionRevenue'].sum().reset_index()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eba32619fdb4e7e92a3b17d75e8e505f9d17392e"},"cell_type":"markdown","source":"Now let's plot the result."},{"metadata":{"trusted":true,"_uuid":"971ee2dc0f71efcaca38e969939accf53a8da849"},"cell_type":"code","source":"plot.figure(figsize=(8,6))\nplot.scatter(range(total_transactions_per_user.shape[0]), np.sort(np.log1p(total_transactions_per_user[\"totals.transactionRevenue\"].values)))\nplot.xlabel('index', fontsize=12)\nplot.ylabel('Transaction Revenue', fontsize=12)\nplot.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f5e6a5e035cf6a742ecd84bdf726ef726e439ca3"},"cell_type":"markdown","source":"It is really intresting to see that just a small part of the customers of the online shop has bought something. This according to the part of the problem description that says: \n> The 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue. As such, marketing teams are challenged to make appropriate investments in promotional strategies.\n\nIn fact, the percentage of real customers in this case is even smaller:"},{"metadata":{"trusted":true,"_uuid":"ea98c132f38532bd764a04935bbb21a9d3d4c899"},"cell_type":"code","source":"real_customers = pd.notnull(ga_customers_train[\"totals.transactionRevenue\"]).sum()\nprint(\"Percentage of real customers (people who bought something):\")\nprint(real_customers/ga_customers_train.size*100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"08cf72f9cda63acfd3c6a4e96316ac5b5416de61"},"cell_type":"markdown","source":"Now that we have a properly formatted dataframe and an idea of how the data are distributed, let's make some experiments in order to predict our result. What we have to do is to build an object able to predict the transaction revenue value of a user online-store session. For example, it would be great if we succeed in making a decision tree algorithm working with these data."},{"metadata":{"_uuid":"8c895cf85e9e40c6e3c24e0ca689f58673fe6750"},"cell_type":"markdown","source":"There is just one problem: decision trees don't work with categorical data. So we need to use sklearn to encode our categorical data using `LabelEncoder`.  Note that encoding is useful only categorical features, in numerical ones it could only leads to data misunderstanding and loss of mean.  So we have to check what fields are strings and what fields are already numerical, in order to properly encode the columns containing strings only."},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"5f87a69d00d83b228ff5b128dd1af75fc04e44b2"},"cell_type":"code","source":"ga_customers_train.dtypes","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0178300d12c6719564ad277d742f75b5103af226"},"cell_type":"markdown","source":"Here we can see that there are a lot of features that are marked as objects even if we certainly know that some of them are numbers and some other are strings. To fix this problem we have to manually cast those fields."},{"metadata":{"_uuid":"37b3a5bb7936f672e0f581fe3660b28918affe2c"},"cell_type":"markdown","source":"Before casting these features, however, it is better to merge test and train dataframes, in order to have the same features set and features encoding in both of them. We can then split them back in training set and test set by taking respectly the first tot  and last tot other items."},{"metadata":{"trusted":true,"_uuid":"60c1efea5695eb1f2d87e44b1ae505087a30f0cd"},"cell_type":"code","source":"mixed_dataset = ga_customers_train.append(ga_customers_test, sort=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fa16602b65a5f5d32c3c45f653c9abafc7ac25b"},"cell_type":"markdown","source":"The below snippet of code casts every feature in a proper way.\n\n**NB** Some features can be simply cast to a numerical value or to a string. Others need to be cast to a unicode type of string, in order to handle some specific records containing unicode characters."},{"metadata":{"trusted":true,"_uuid":"340cf8ae5df775bd56a6ff25c410a9f41712f644"},"cell_type":"code","source":"mixed_dataset['channelGrouping'] = mixed_dataset['channelGrouping'].fillna('none').astype('category')\n#we don't train our model on Ids features, so we can avoid to handle their types\nmixed_dataset['device.browser'] = mixed_dataset['device.browser'].fillna('none').astype('category')\nmixed_dataset['device.deviceCategory'] = mixed_dataset['device.deviceCategory'].fillna('none').astype('category')\nmixed_dataset['device.operatingSystem'] = mixed_dataset['device.operatingSystem'].fillna('none').astype('category')\nmixed_dataset['geoNetwork.city'] = mixed_dataset['geoNetwork.city'].fillna('none').astype('|S2048')\nmixed_dataset['geoNetwork.country'] = mixed_dataset['geoNetwork.country'].fillna('none').astype('unicode')\nmixed_dataset['geoNetwork.continent'] = mixed_dataset['geoNetwork.continent'].fillna('none').astype('|S2048')\nmixed_dataset['geoNetwork.metro'] = mixed_dataset['geoNetwork.metro'].fillna('none').astype('|S2048')\nmixed_dataset['geoNetwork.networkDomain'] = mixed_dataset['geoNetwork.networkDomain'].fillna('none').astype('|S2048')\nmixed_dataset['geoNetwork.region'] = mixed_dataset['geoNetwork.region'].fillna('none').astype('|S2048')\nmixed_dataset['geoNetwork.subContinent'] = mixed_dataset['geoNetwork.subContinent'].fillna('none').astype('|S2048')\nmixed_dataset['totals.bounces'] = pd.to_numeric(mixed_dataset['totals.bounces'].fillna(0))\nmixed_dataset['totals.hits'] = pd.to_numeric(mixed_dataset['totals.hits'].fillna(0))\nmixed_dataset['totals.newVisits'] = pd.to_numeric(mixed_dataset['totals.newVisits'].fillna(0))\nmixed_dataset['totals.pageviews'] = pd.to_numeric(mixed_dataset['totals.pageviews'].fillna(0))\nmixed_dataset['totals.sessionQualityDim'] = pd.to_numeric(mixed_dataset['totals.sessionQualityDim'].fillna(0))\nmixed_dataset['totals.timeOnSite'] = pd.to_numeric(mixed_dataset['totals.timeOnSite'].fillna(0))\nmixed_dataset['totals.transactionRevenue'] = mixed_dataset['totals.transactionRevenue'].fillna(0)\nmixed_dataset['totals.totalTransactionRevenue'] = pd.to_numeric(mixed_dataset['totals.totalTransactionRevenue'].fillna(0))\nmixed_dataset['totals.transactions'] = pd.to_numeric(mixed_dataset['totals.transactions'].fillna(0))\nmixed_dataset['trafficSource.adContent'] = mixed_dataset['totals.transactions'].fillna('none').astype('|S2048')\nmixed_dataset['trafficSource.adwordsClickInfo.adNetworkType'] = mixed_dataset['trafficSource.adwordsClickInfo.adNetworkType'].fillna('none').astype('|S2048')\nmixed_dataset['trafficSource.adwordsClickInfo.gclId'] = mixed_dataset['trafficSource.adwordsClickInfo.gclId'].fillna('none').astype('|S2048')\nmixed_dataset['trafficSource.adwordsClickInfo.isVideoAd'] = mixed_dataset['trafficSource.adwordsClickInfo.isVideoAd'].fillna('none').astype('|S2048')\nmixed_dataset['trafficSource.adwordsClickInfo.page'] = pd.to_numeric(mixed_dataset['trafficSource.adwordsClickInfo.page'].fillna(0))\nmixed_dataset['trafficSource.adwordsClickInfo.slot'] = mixed_dataset['trafficSource.adwordsClickInfo.slot'].fillna('none').astype('|S2048')\nmixed_dataset['trafficSource.campaign'] = mixed_dataset['trafficSource.campaign'].fillna('none').astype('|S2048')\nmixed_dataset['trafficSource.campaignCode'] = mixed_dataset['trafficSource.campaignCode'].fillna('none').astype('|S2048')\nmixed_dataset['trafficSource.isTrueDirect'] = mixed_dataset['trafficSource.isTrueDirect'].fillna('none').astype('|S2048')\nmixed_dataset['trafficSource.keyword'] = mixed_dataset['trafficSource.keyword'].fillna('none').astype('unicode')\nmixed_dataset['trafficSource.medium'] = mixed_dataset['trafficSource.medium'].fillna('none').astype('|S2048')\nmixed_dataset['trafficSource.referralPath'] = mixed_dataset['trafficSource.referralPath'].fillna('none').astype('unicode')\nmixed_dataset['trafficSource.source'] = mixed_dataset['trafficSource.source'].fillna('none').astype('|S2048')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab5fc70e0c70d2fadc6f43617cbe22c5a06ab9fd"},"cell_type":"markdown","source":"We have casted all object features. Let's check real types now."},{"metadata":{"trusted":true,"_uuid":"c52485b2fddd151c7b2ce2f5f0fe2b2220e0ac1b"},"cell_type":"code","source":"mixed_dataset.dtypes","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f80f5aea0245c72f586a666fac9436a91d3e22b7"},"cell_type":"markdown","source":"We can see that while some features have been cast correctly, other has remained marked as objects. Indeed, those features will have the same behavior of a string, so there is no real problem in this.\n\nNow we can proceed with the real encoding."},{"metadata":{"trusted":true,"_uuid":"ee88fb92d8009f5b698a0f8472569cc043def21b"},"cell_type":"code","source":"les = []\nfeatures = ['channelGrouping', \n            'device.browser',\n            'device.deviceCategory',\n            'device.operatingSystem', \n            'geoNetwork.city', \n            'geoNetwork.country',\n            'geoNetwork.metro',\n            'geoNetwork.networkDomain', \n            'geoNetwork.region',\n            'geoNetwork.continent',\n            'geoNetwork.subContinent',\n            'trafficSource.adContent',\n            'trafficSource.adwordsClickInfo.adNetworkType',\n            'trafficSource.adwordsClickInfo.gclId',\n            'trafficSource.adwordsClickInfo.isVideoAd',\n            'trafficSource.adwordsClickInfo.slot', \n            'trafficSource.campaign',\n            'trafficSource.campaignCode',\n            'trafficSource.isTrueDirect', \n            'trafficSource.keyword',\n            'trafficSource.medium', \n            'trafficSource.referralPath',\n            'trafficSource.source']\nfor feature in features:\n    le = preprocessing.LabelEncoder()\n    try:\n        le.fit(mixed_dataset[feature])\n        mixed_dataset[feature] = le.transform(mixed_dataset[feature])\n        les.append(le)\n    except (SystemError, UnicodeEncodeError):\n       print(\"Error: can't handle \" + feature + \" data type (maybe it is a unicode string).\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7778706604c09a232fd04ff6e5b4edc8c0b2a49a"},"cell_type":"markdown","source":"Let's check the result."},{"metadata":{"trusted":true,"_uuid":"a5e3e226db16fc324ea957d2f871493a9c3b2fc0"},"cell_type":"code","source":"print(mixed_dataset.values[0,:])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"75a27b12fcb33ae58dd7c82a4d26150d23941666"},"cell_type":"markdown","source":"Ok, we have formatted properly the entire merged dataset. Now we can divide the result into two objects (`X_train` and `X_test`), both having the same number of features."},{"metadata":{"trusted":true,"_uuid":"90e2f68150c2747da9ba814f14a549d213a209d1"},"cell_type":"code","source":"X_train = mixed_dataset.drop(columns=['totals.transactionRevenue','fullVisitorId', 'visitId']).values[0:ga_customers_train.shape[0],:]\nX_test = mixed_dataset.drop(columns=['totals.transactionRevenue','fullVisitorId', 'visitId']).values[ga_customers_train.shape[0]:mixed_dataset.shape[0],:]\ny = mixed_dataset['totals.transactionRevenue'].values[0:ga_customers_train.shape[0]]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc1fcb69b967b984711d2eca6fbdcfdec2fa8984"},"cell_type":"markdown","source":"Let's create and train the regressor."},{"metadata":{"trusted":true,"_uuid":"f7469772256f8871d7565504c57a931c218c8bae"},"cell_type":"code","source":"regressor = DecisionTreeRegressor()\nregressor.fit(X_train, y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd2b4cd7fbec4ec8f14a031c3a95f814ff107340"},"cell_type":"markdown","source":"Now that we have our regressor, let's try to apply it to the test set."},{"metadata":{"trusted":true,"_uuid":"ccdfd2f185b8c48c9cdbaf2732d5afba7a05fe1b"},"cell_type":"code","source":"tree_result = regressor.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4a0093677ab65fbfb99e509813f720d9974dedbd"},"cell_type":"markdown","source":"Well, we have a result.  However we don't know how good our predictions are. Let's make some considerations:\n* our regressor is a simple decision tree regressor, trained on a single dataset\n* it would be better if we have different trees trained on different datasets, in order to obtain different regressors\n* once obtained different regressors, we can combine the predictions of those regressors to get better predictions\n* `RandomForestRegressor` fom klearn makes exactly this thing: takes a dataset, creates different subsets on that dataset, creates different decision tree regressors and makes predictions based on the combination of those. \nSo, let's try to apply a random forest regressor to our data."},{"metadata":{"trusted":true,"_uuid":"abea24f514ad6ad77be59332ec50d3cc903918db"},"cell_type":"code","source":"random_forest = RandomForestRegressor(n_estimators=100, bootstrap=True)\nrandom_forest.fit(X_train, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"725f6fdfd1c5e28949a57c52889a92e80ecaec5a"},"cell_type":"code","source":"ra_predictions = random_forest.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e3ac8e4760a2258bdfd647ee51599f2fb44767f6"},"cell_type":"markdown","source":"`ra_predictions` seems to be a better predictor for our purpose. At least, it is more trained. So it's not a bas idea to create the submission file with those data."},{"metadata":{"trusted":true,"_uuid":"d0c075b7d4ea3ea606acc6683f547f6552a16166"},"cell_type":"code","source":"submission = ga_customers_test[['fullVisitorId']].copy()\nsubmission.loc[:, 'PredictedLogRevenue'] = ra_predictions\ngrouped_test = submission[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\ngrouped_test[\"PredictedLogRevenue\"] = np.log1p(grouped_test[\"PredictedLogRevenue\"])\ngrouped_test.to_csv('submit.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}