{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This R environment comes with many helpful analytics packages installed\n# It is defined by the kaggle/rstats Docker image: https://github.com/kaggle/docker-rstats\n# For example, here's a helpful package to load\n\nlibrary(tidyverse) # metapackage of all tidyverse packages\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nlist.files(path = \"../input\")\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Apprentissage\n\n

Théo DEMESSANCE and Valentin TASSEL - A5 ESILV IBO

\n
\n

Task

\n
\nIn this project, we’re challenged to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer.\n
\n
\n

Initial Data

\nUser transactions from August 1st 2016 to April 30th 2018.\n
\nGoogle Merchandise Store is a type of stores which sell Google, YouTube and Android derivatives."},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nOur task is to build an algorithm that predicts the natural log of the sum of all transactions per user. Thus, \nfor every user in the test set, the target is:\n\n$$y_{user} = \\sum_{i=1}^n transaction_{user_i}$$\n\n$$target_{user} = log(y_{user}+1).$$\n\nSubmissions are scored on the root mean squared error, which is defined as:\n\n$$RMSE = \\sqrt{\\frac{1}{n} \\sum_{i=1}^n (y_i - \\widehat{y_i})^2},$$\n\nwhere $\\widehat{y}$ is the predicted revenue for a customer and $y$ is the natural log of the actual revenue value.\n\nLet's prepare and have a look at the dataset."},{"metadata":{},"cell_type":"markdown","source":"## Load the librairies"},{"metadata":{"trusted":true},"cell_type":"code","source":"library(jsonlite)\nlibrary(data.table)\nlibrary(dplyr)\nlibrary(magrittr)\nlibrary(lubridate)\nlibrary(cowplot)\nlibrary(scales)\nlibrary(highcharter)\nlibrary(ggplot2)\nlibrary(tidyverse)\nlibrary(vcd)\nlibrary(highcharter)\nlibrary(countrycode)\nlibrary(stringr)\nlibrary(corrplot)\nlibrary(pracma)\nlibrary(dummies)\nlibrary(naniar)\nlibrary(Metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df <- read.csv(\"../input/ga-customer-revenue-prediction/train.csv\", stringsAsFactors = FALSE, colClasses = c(\"character\",\"integer\",\"character\",\"character\",\"character\",\"character\",\"character\",\"character\",\"character\",\"integer\",\"integer\",\"integer\"))\nhead(df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a total of 12 features: \n\n* **fullVisitorId** - an unique identifier for each user of the Google Merchandise Store\n* **channelGrouping** - the channel via which the user came to the Store\n* **date** - the date on which the user visited the Store\n* **device** - the specifications for the device used to access the Store\n* **geoNetwork** - this section contains information about the geography of the user\n* **sessionId** - an unique identifier for this visit to the store\n* **socialEngagementType** - engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\"\n* **totals** - this section contains aggregate values across the session\n* **trafficSource** - this section contains information about the Traffic Source from which the session originated\n* **visitId** - an identifier for this session\n* **visitNumber** - the session number for this user\n* **visitStartTime** - the timestamp (POSIX).\n\nActually the columns **device**, **geoNetwork**, **trafficSource**, **totals** contain data in JSON format. \nTo parse it we can use jsonlite library."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Opening the Data\ntrain <- read.csv(\"../input/ga-customer-revenue-prediction/train.csv\", stringsAsFactors = FALSE, colClasses = c(\"character\",\"integer\",\"character\",\"character\",\"character\",\"character\",\"character\",\"character\",\"character\",\"integer\",\"integer\",\"integer\")) ; \ntest <- read.csv(\"../input/ga-customer-revenue-prediction/test.csv\", stringsAsFactors = FALSE, colClasses = c(\"character\",\"integer\",\"character\",\"character\",\"character\",\"character\",\"character\",\"character\",\"character\",\"integer\",\"integer\",\"integer\")) ; \n\n# Creation of a test train indicator column before assembling the two tables\ntrain$datasplit <- \"train\" ; test$datasplit <- \"test\"\n\n# Removal of a visibly useless column\ntrain$campaignCode <- NULL ; test$campaignCode <- NULL\n\n# Identification of the 4 columns in json format\njson <- c(\"trafficSource\",\"totals\",\"geoNetwork\",\"device\")\ntables <- c(\"train\",\"test\")\nglob <- data.table() #empty table that will retrieve the transformed tables\n\n# Reading and successive transformation train and test (deletion of unnecessary columns when passing) \nfor (t in tables) {\n partiel <- get(t)[,setdiff(names(get(t)),json)] # non json columns\n for (j in json) partiel <- cbind(partiel,fromJSON(paste(\"[\", paste(get(t)[[j]], collapse = \",\"), \"]\")))\n temp <- partiel$adwordsClickInfo\n partiel$adwordsClickInfo <- NULL\n temp$targetingCriteria <- NULL\n result <- as.data.table(cbind(partiel,temp))\n if(t == \"train\") result$campaignCode <- NULL else result$transactionRevenue <- NA\n glob <- rbind(glob,result)\n}\nrm(partiel, train, test) ; gc()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"head(glob)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our JSON columns have been flattened, and so we are able to work with this dataset as a classic one."},{"metadata":{},"cell_type":"markdown","source":"## Constant columns\n\nLet's find constant columns: "},{"metadata":{"trusted":true},"cell_type":"code","source":"#we find the constant columns (i.e 1 distinct value in it):\nfea_uniq_values <- sapply(glob, n_distinct)\nfea_del <- names(fea_uniq_values[fea_uniq_values == 1])\n\n#and we can remove safely the columns:\nglob %<>% select(-one_of(fea_del))\nhead(glob)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"19 columns were removed as they contained only 1 distinct value. "},{"metadata":{"trusted":true},"cell_type":"code","source":"fea_del #features with 1 distinct value","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Variable type modification\nLet's observe each variable for the variable type modifications in order to make a exploratory data analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Convert date column from character to Date class\nglob$date <- as.Date(as.character(glob$date), format='%Y%m%d')\n\n#Convert chr variable type in int type\nglob$hits <- as.numeric(glob$hits)\nglob$pageviews <- as.numeric(glob$pageviews)\nglob$bounces <- as.numeric(glob$bounces)\nglob$newVisits <- as.numeric(glob$newVisits)\nglob$transactionRevenue <- as.numeric(glob$transactionRevenue)\nglob$page <- as.numeric(glob$page)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Columns type\n\nWe can see that some data are not in their natural representations. We'll need further modification to be able to analyze the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Get the type of each column\nsapply(glob, class)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"head(glob)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we do have a correct and clean dataset to begin our analysis."},{"metadata":{},"cell_type":"markdown","source":"## Missing values"},{"metadata":{"trusted":true},"cell_type":"code","source":"gg_miss_var(glob, show_pct = TRUE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's observe if the features containing missing values are quantitatives or qualitatives :"},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_value_var <- glob %>% select_if(function(col) sum(is.na(col)) > 0)\ncolnames(missing_value_var) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Qualitative missing values\n\nThe roadmap for handling the qualititive missing values :\n\n* Add an unknown class\n* Verify if it's a numerical index it can bring temporal information\n* Exclue variable with different qualitative value on all lines\n* Limit the possible class number in a variable to reduce the calculation time\n\nWe don't touch the missing data in transactionRevenue as its our response variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_value_var_qual <- missing_value_var %>% select_if(is.character)\ncolnames(missing_value_var_qual)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gg_miss_var(missing_value_var_qual, show_pct = TRUE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Putting an unknown class inside qualitative features to replace NA."},{"metadata":{"trusted":true},"cell_type":"code","source":"glob$adContent <- glob$adContent %>% replace_na(\"unknown\")\nglob$slot <- glob$slot %>% replace_na(\"unknown\")\nglob$adNetworkType <- glob$adNetworkType %>% replace_na(\"unknown\")\nglob$referralPath <- glob$referralPath %>% replace_na(\"unknown\")\nglob$keyword <- glob$keyword %>% replace_na(\"unknown\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we have a non-numerical ID, it doesn't bring information to our problem so we decide to remove this feature."},{"metadata":{"trusted":true},"cell_type":"code","source":"glob$gclId <- NULL","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Verification that all qualitative missing values has been processed"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"\"\"\"\nmissing_value_var <- glob %>% select_if(function(col) sum(is.na(col)) > 0)\nmissing_value_var_qual <- missing_value_var %>% select_if(is.character)\ncolnames(missing_value_var_qual)\ngg_miss_var(missing_value_var_qual, show_pct = TRUE)\n\"\"\""},{"metadata":{},"cell_type":"markdown","source":"This error is normal since we do not have any NA values left."},{"metadata":{},"cell_type":"markdown","source":"### Quantitative missing values"},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_value_var <- glob %>% select_if(function(col) sum(is.na(col)) > 0)\nmissing_value_var_num <- missing_value_var %>% select_if(is.numeric)\ngg_miss_var(missing_value_var_num, show_pct = TRUE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By flattening JSON, when attributes are not mentionned, the code puts the missing value as NA. So, we can easily replace it by 0."},{"metadata":{"trusted":true},"cell_type":"code","source":"glob$newVisits<- glob$newVisits %>% replace_na(0)\nglob$bounces <- glob$bounces %>% replace_na(0)\nglob$page <- glob$page %>% replace_na(0)\nglob$pageviews <- glob$pageviews %>% replace_na(0)\nglob$transactionRevenue <- glob$transactionRevenue %>% replace_na(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Logical missing values "},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_value_var <- glob %>% select_if(function(col) sum(is.na(col)) > 0)\nmissing_value_var_lgl <- missing_value_var %>% select_if(is.logical)\ngg_miss_var(missing_value_var_lgl, show_pct = TRUE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"glob$isVideoAd <- NULL \nglob$isTrueDirect <- NULL","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Verification of the missing values has been processed"},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_value_var <- glob %>% select_if(function(col) sum(is.na(col)) > 0)\ncolnames(missing_value_var)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Blank space means that all values has been processed, and we don't have any NA values left."},{"metadata":{},"cell_type":"markdown","source":"# Exploratory data analysis"},{"metadata":{},"cell_type":"markdown","source":"## Split the global dataset into train and test "},{"metadata":{"trusted":true},"cell_type":"code","source":"#We can split glob into train and test according to the datasplit column:\nglob_train <- glob[which(glob$datasplit == 'train')]\nglob_test <- glob[which(glob$datasplit == 'test')]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Target variable analysis\n\nAs a target variable, we use **transactionRevenue**."},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_train %>% filter(transactionRevenue > 0) %>% summarize('number of transactions'= n(), 'total revenues train set'= sum(transactionRevenue))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have 11 515 transactions, for a total of 1000 billions of $US"},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_train$bought = ifelse(glob_train$transactionRevenue > 0, 1, 0)\nglob_train$bought[is.na(glob_train$bought)] <- 0\nglob_train[, bought := as.factor(bought)]\ntable(glob_train$bought) / nrow(glob_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With this table, we can see that 99% of the dataset leads to NULL transactions revenue. But that does not mean that only 1% of people generates 100% of the transactions since we can come several times to the shop."},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train %>% \nggplot() + geom_density(aes(as.numeric(transactionRevenue)), na.rm = TRUE) +\nlabs(x = \"transactionRevenue\",\n title = \"Frequency Distribution of transactionRevenue\") +\ntheme_classic()\n\nplot(p1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"p2 <- glob_train %>% \n ggplot(aes(x=log(transactionRevenue), y=..density..)) + \n geom_histogram(fill='steelblue', na.rm = TRUE, bins=40) + \n geom_density(aes(log(transactionRevenue)), na.rm = TRUE) +\n labs(x = \"transactionRevenue\",\n title = \"Frequency Distribution of Natural Log transactionRevenue\") +\n theme_classic()\n\nplot(p2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The mean of the natural log of transaction revenue appears to be around 16 and is approximately normally distributed. It confirms the 80/20 rule saying that for many businesses–only a small percentage of customers produce most of the revenue. \n\nNow let’s take a look at daily revenue over the time period of the data set."},{"metadata":{},"cell_type":"markdown","source":"## Distribution of transaction dates"},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train %>% mutate(year_month = make_date(year(date), month(date))) %>% \n group_by(year_month) %>% count() %>% \n ggplot(aes(x = year_month, y = n)) +\n geom_bar(stat=\"identity\", fill=\"steelblue\") +\n labs(x = \"\", y = \"transactions\", title = \"Train\") +\n scale_x_date(labels = date_format(\"%Y - %m\"))\n\nplot(p1)\n\np2 <- glob_test %>% mutate(year_month = make_date(year(date), month(date))) %>% \n group_by(year_month) %>% count() %>% \n ggplot(aes(x = year_month, y = n)) +\n geom_bar(stat=\"identity\", fill=\"steelblue\") +\n labs(x = \"\", y = \"transactions\", title = \"Test\") +\n scale_x_date(labels = date_format(\"%Y - %m\"))\n\nplot(p2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As shown in the figure, there are only a few of the transactions after Jul 2017 in the train set, because the rest is in the test set. It makes sense to create time-based splits for train/validation sets."},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train[, .(n = .N), by = date] %>%\n ggplot(aes(x = date, y = n)) + \n geom_line(color='steelblue') +\n geom_smooth(color='green') + \n labs(\n x='',\n y='Visits (000s)',\n title='Daily visits'\n )\n\nplot(p1)\n\np2 <- glob_train[, .(revenue = sum(transactionRevenue, na.rm=TRUE)), by=date] %>%\n ggplot(aes(x=date, y=revenue)) + \n geom_line(color='steelblue') +\n geom_smooth(color='green') + \n labs(\n x='',\n y='Revenue (unit dollars)',\n title='Daily transaction revenue'\n )\n\nplot(p2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can make several inferences according to the graph: \n\n* A big high for visits and transactions before january can be related to Xmas\n* The smoothing line indicates that daily revenue has remained pretty constant over the year\n\nLet's have a look on the distribution during the day : "},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_train$visitStartTime <- as_datetime(glob_train$visitStartTime)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train[, .(visitHour = hour(visitStartTime))][\n , .(visits = .N), by = visitHour] %>%\n ggplot(aes(x = visitHour, y = visits)) + \n geom_line(color='steelblue') +\n geom_point(color = 'steelblue') + \n labs(\n x='Hour of the day',\n y='Visits (000s)',\n title='Aggregate visits by hour of the day'\n )\n\nplot(p1)\n\np2 <- glob_train[, .(transactionRevenue, visitHour = hour(visitStartTime))][\n , .(revenue = sum(transactionRevenue, na.rm = T)), by = visitHour] %>%\n ggplot(aes(x = visitHour, y = revenue)) + \n geom_line(color='steelblue') +\n geom_point(color = 'steelblue') + \n labs(\n x='Hour of the day',\n y='Transaction revenue (000s)',\n title='Aggregate revenue by hour of the day'\n )\n\nplot(p2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hours of the day are quite correlated between Revenue and Visits. At the end of the day we have an increase of visits and transactions."},{"metadata":{},"cell_type":"markdown","source":"## Categorical features analysis"},{"metadata":{},"cell_type":"markdown","source":"### Channel Grouping\n\nThe channel via which the user came to the Store : \n- Organic Search : the traffic of natural results on a search engine\n- Social : all the traffic that comes from social networks\n- Direct : a channel that brings together many unidentified traffic sources\n- Referral : traffic that comes from another website\n- Paid Search : the traffic generated by your SEA campaigns (google ads, ...)\n- Display : the traffic of the campaigns carried out on the Google display network"},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train[, .(n = .N), by = channelGrouping] %>%\n ggplot(aes(x = reorder(channelGrouping, -n), y = n)) + \n geom_bar(stat = 'identity', fill = 'steelblue') + \n labs(x = 'Channel Grouping',\n y = 'Visits',\n title = 'Visits by channel grouping')\n\nplot(p1)\n\np2 <- glob_train[, .(revenue = sum(transactionRevenue, na.rm = T)), by = channelGrouping] %>%\n ggplot(aes(x = reorder(channelGrouping, -revenue), y = revenue)) + \n geom_bar(stat = 'identity', fill = 'steelblue') + \n labs(x = 'Channel Grouping',\n y = 'Revenue',\n title = 'Revenue by channel grouping')\n\nplot(p2)\n\np3 <- glob_train %>%\n ggplot(mapping = aes(x = channelGrouping,y = log(transactionRevenue))) + \n geom_boxplot()\n\nplot(p3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that in terms of total revenue, “Referral” accounts for the largest share. “Display” had the highest average revenue, but be aware that this may be due to a particularly large transaction, as the number of visits from “Display” are very small."},{"metadata":{},"cell_type":"markdown","source":"### Operating Systems"},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train[, .(n = .N), by = operatingSystem] %>%\n ggplot(aes(x = reorder(operatingSystem, -n), y = n)) + \n geom_bar(stat = 'identity', fill = 'steelblue') +\n labs(x = 'Operating System', \n y = '# of visits in data set',\n title = 'Distribution of visits by device operating system') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1))\n\nplot(p1)\n\np2 <- glob_train[, .(revenue = sum(transactionRevenue, na.rm = TRUE)), by = operatingSystem][revenue > 0] %>%\n ggplot(aes(x = reorder(operatingSystem, -revenue), y = revenue)) + \n geom_bar(stat = 'identity', fill = 'steelblue') +\n labs(x = 'Operating System', \n y = 'Revenue',\n title = 'Distribution of revenue by device operating system') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1))\n\nplot(p2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With these plots, we are able to say that even if Windows accounted for more records in the data set than any other operating system, Macintosh accounted for more transaction revenues than other operating system."},{"metadata":{},"cell_type":"markdown","source":"### Browsers"},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train[, .(n = .N), by = browser][1:10] %>%\n ggplot(aes(x = reorder(browser, -n), y = n)) + \n geom_bar(stat = 'identity', fill = 'steelblue') +\n labs(x = 'Browser', \n y = '# of visits in data set',\n title = 'Distribution of visits by browser (Top 10 browsers)') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1))\n\nplot(p1)\n\np2 <- glob_train[, .(revenue = sum(transactionRevenue, na.rm = TRUE)), by = browser][revenue > 0] %>%\n ggplot(aes(x = reorder(browser, -revenue), y = revenue)) + \n geom_bar(stat = 'identity', fill = 'steelblue') +\n labs(x = 'Browser', \n y = 'Revenue in $',\n title = 'Distribution of revenue by browser') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1))\n\nplot(p2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Chrome seems to be the most used and the browser which generates most of the revenue"},{"metadata":{},"cell_type":"markdown","source":"### Device category"},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train[, .(n = .N), by = deviceCategory] %>%\n ggplot(aes(x = reorder(deviceCategory, -n), y = n)) + \n geom_bar(stat = 'identity', fill = 'steelblue') +\n labs(x = 'Device category', \n y = '# of visits in data set',\n title = 'Distribution of visits by device category') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1))\n\nplot(p1)\n\np2 <- glob_train[, .(revenue = sum(transactionRevenue, na.rm = TRUE)), by = deviceCategory] %>%\n ggplot(aes(x = reorder(deviceCategory, -revenue), y = revenue)) + \n geom_bar(stat = 'identity', fill = 'steelblue') +\n labs(x = 'Device category', \n y = 'Revenue in $',\n title = 'Distribution of revenue by device category') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1))\n\nplot(p2)\n\np3 <- glob_train %>%\n ggplot(mapping = aes(x = deviceCategory,y = log(transactionRevenue))) +\n geom_boxplot()\n\nplot(p3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Distribution are the same, even if tablet has a mean a bit superior than mobile, but has a lower count of visits. Now we can try to see if there is a difference between mobile and non-mobile devices:"},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train %>% \n ggplot(aes(x = log(transactionRevenue), y = ..density.., fill = isMobile)) + \n geom_density(alpha = 0.5) + \n scale_fill_manual(values = c('steelblue', 'green')) + \n labs(title = 'Distribution of log revenue by mobile and non-mobile devices')\n\nplot(p1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It appears that average transaction income on mobile devices is lower than on non-mobile devices."},{"metadata":{},"cell_type":"markdown","source":"### Network domain"},{"metadata":{"trusted":true},"cell_type":"code","source":"DT <- glob_train %>%\n count(networkDomain) %>%\n mutate(networkDomain = fct_reorder(networkDomain, n, .desc = TRUE))\n\nhead(DT[order(n, decreasing = TRUE)]) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that if we return the 5 network domains with the highest count, it is not very representative since .com or .net can be present in others website domain.\nTo answer this problem, we'll split and count only on the last part (after the .) of the domain : "},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_train[, domain := tstrsplit(glob_train$networkDomain, '\\\\.', keep = c(2))][!is.na(domain), domain := paste0('.', domain)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train[!is.na(networkDomain), .(n = .N), by = domain][order(-n)][!is.na(domain),][1:10] %>%\n ggplot(aes(x = reorder(domain, -n), y = n)) + \n geom_bar(stat = 'identity', fill = 'steelblue') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1)) +\n labs(title = 'Number of visits from top-level domains', \n y = 'Visits', \n x = 'Top-level domain')\n\nplot(p1)\n\np2 <- glob_train[!is.na(networkDomain), .(revenue = sum(transactionRevenue, na.rm = TRUE)), by = domain][order(-revenue)][!is.na(domain),][1:10] %>%\n ggplot(aes(x = reorder(domain, -revenue), y = revenue)) + \n geom_bar(stat = 'identity', fill = 'steelblue') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1)) +\n labs(title = 'Revenue from top-level domains', \n y = 'Revenue', \n x = 'Top-level domain')\n\nplot(p2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thus, .net and .com are not the most representative network domains of the dataset."},{"metadata":{},"cell_type":"markdown","source":"We replace the missing values by .unknown for the furture feature engineering"},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_train$domain <- glob_train$domain %>% replace_na(\".unknown\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Medium\n\nMediums are broad buckets of categories that describe the kind of traffic being driven to your website. For example, “Organic” is a Medium because it encompasses traffic coming from search engines such as Google. “Referral,” “Social,” and “Paid” are other examples of Mediums."},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train[, .(n = .N), by = medium] %>%\n ggplot(aes(x = reorder(medium, -n), y = n)) + \n geom_bar(stat = 'identity', fill = 'steelblue') +\n labs(x = 'Medium', \n y = '# of visits in data set',\n title = 'Distribution of visits by medium') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1))\n\nplot(p1)\n\np2 <- glob_train[, .(revenue = sum(transactionRevenue, na.rm = TRUE)), by = medium] %>%\n ggplot(aes(x = reorder(medium, -revenue), y = revenue)) + \n geom_bar(stat = 'identity', fill = 'steelblue') +\n labs(x = 'Medium', \n y = 'Revenue in $',\n title = 'Distribution of revenue by medium') + \n theme(axis.text.x = element_text(angle = 90, hjust = 1))\n\nplot(p2)\n\np3 <- glob_train %>%\n ggplot(mapping = aes(x = medium,y = log(transactionRevenue))) +\n geom_boxplot()\n\nplot(p3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Geographical analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"p1 <- glob_train[, .(revenue = sum(transactionRevenue, na.rm=TRUE)), by = continent][!is.na(continent),] %>%\n ggplot(aes(x=reorder(continent, revenue), y=revenue)) + \n geom_bar(stat='identity', fill='steelblue') + \n coord_flip() + \n labs(x='', y='Revenue', title='Total transaction revenue by continent')\n\nplot(p1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Revenues in the Americas exceed those of any other continent.\n\nNext, let's take a look at the distribution of total transaction revenues and total visits among countries. To do so, we will use the Highcharter library. We will use the country code package to convert the country names in the training dataset to iso3 codes, which we can then use to join the worldgeojson highcharter data.\n\nNote that in the maps below, we use the log rather than the gross value revenue so that we get a better dispersion for the choropleth palette."},{"metadata":{"trusted":true},"cell_type":"code","source":"by_country1 <- glob_train[, .(n = .N), by = country]\nby_country1$iso3 <- countrycode(by_country1$country, origin = 'country.name', destination='iso3c')\nby_country1[, visits := log(n)]\nby_country1[, visit := log(n)]\n\nhighchart() %>%\n hc_add_series_map(worldgeojson, by_country1, value = 'visits', joinBy = 'iso3') %>%\n hc_title(text = 'Total visits by country (natural log)') %>%\n hc_tooltip(useHTML = TRUE, headerFormat = \"\",\n pointFormat = \"{point.country}: {point.visit:.0f}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"by_country2 <- glob_train[, .(n = .N, revenue = log(sum(transactionRevenue, na.rm=TRUE))), by = country]\nby_country2$iso3 <- countrycode(by_country2$country, origin='country.name', destination='iso3c')\n#by_country2[, rev_per_visit := revenue / n]\nby_country2[, revenues := revenue]\n\nhighchart() %>%\n hc_add_series_map(worldgeojson, by_country2, value = 'revenue', joinBy = 'iso3') %>%\n hc_title(text = 'Total transaction revenue by country (natural log)') %>%\n hc_tooltip(pointFormat = \"{point.country}: ${point.revenues}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there is a huge difference between visiting and buying. Indeed, regions like Africa and the Middle East seem to visit the GStore a lot, but not to buy. On the contrary, countries such as Venezuela, Kenya and Nigeria are not the ones with the most visits, but the ones that generate the most income."},{"metadata":{},"cell_type":"markdown","source":"### Numerical features observation"},{"metadata":{"trusted":true},"cell_type":"code","source":"numerical_features <- colnames(glob_train %>% select(where(is.numeric)))\nnumerical_features <- glob_train %>% select(numerical_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"corrplot(cor(numerical_features), method = \"number\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Qualitative features observation"},{"metadata":{"trusted":true},"cell_type":"code","source":"library(Boruta)\n\nout.boruta <- Boruta(transactionRevenue~., data = glob_train[1:1000] %>% select(where(is.character), transactionRevenue))\nprint(out.boruta)\n\nplot(out.boruta, cex.axis = 0.8, las=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature engineering"},{"metadata":{},"cell_type":"markdown","source":"### Id without information removing"},{"metadata":{},"cell_type":"markdown","source":"We remove visitId cause it's the same variable than visitStarttime "},{"metadata":{"trusted":true},"cell_type":"code","source":"glob$visitId <- NULL ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Numerical feature engineering"},{"metadata":{},"cell_type":"markdown","source":"**VisitStartTime** transformation to Hours and Year day (0,365)"},{"metadata":{"trusted":true},"cell_type":"code","source":"typeof(glob$visitStartTime)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"glob$visitStartTime <- as_datetime(glob$visitStartTime)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"typeof(glob$visitStartTime)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"head(glob$visitStartTime)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Categorical feature engineering"},{"metadata":{},"cell_type":"markdown","source":"Let's observe the categorical variables present in the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"colnames(glob %>% select(where(is.character)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We change the **network domain** variable to a domain name only as .com .fr .net ..."},{"metadata":{"trusted":true},"cell_type":"code","source":"glob[, domain := tstrsplit(glob$networkDomain, '\\\\.', keep = c(2))][!is.na(domain), domain := paste0('.', domain)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We set a classe **.unknown** for the missing values"},{"metadata":{"trusted":true},"cell_type":"code","source":"glob$domain <- glob$domain %>% replace_na(\".unknown\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Feature selection"},{"metadata":{},"cell_type":"markdown","source":"Feature selection is the process of identifying and selecting a subset of input features that are most relevant to the target variable."},{"metadata":{},"cell_type":"markdown","source":"We select all the numerical variable because it's not too much variables, and the computation will be easy for these variables."},{"metadata":{},"cell_type":"markdown","source":"We make a feature selection on the categorical variable we decided to use our exploration to indicate us which variable to use.\nWe select as categorical variables : **continent, domain, transactionRevenue, channelGrouping**.\nThese are features that brings a lot of informations after our analysis using Boruta, and our sector domain knowledges \n"},{"metadata":{},"cell_type":"markdown","source":"We take **datasplit** as it tells us if the row is in the training set or the testing set"},{"metadata":{},"cell_type":"markdown","source":"We keep the most important classes for our target variable, we choose these classes after the analysis of the data exploration.\nWe set the other classes as 'Others'."},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_selection <- glob %>% select(where(is.numeric),continent, domain, transactionRevenue, datasplit, fullVisitorId, channelGrouping)\n\nglob_selection$continent <- ifelse(glob_selection$continent %in% c(\"Americas\",\"Asia\",\"Europe\"),glob_selection$continent,\"Continent_Others\")\nglob_selection$domain <- ifelse(glob_selection$domain %in% c(\".net\",\".unknown\",\".com\"),glob_selection$domain,\"domain_Others\")\nglob_selection$channelGrouping <- ifelse(glob_selection$channelGrouping %in% c(\"Referral\",\"Direct\",\"Organic Search\"),glob_selection$channelGrouping,\"channelGrouping_Others\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We create dummy variable from these qualitative column in order to process them in the models "},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_selection <- cbind(glob_selection, dummy(glob_selection$continent, sep=\"_\"), dummy(glob_selection$domain, sep=\"_\"), dummy(glob_selection$channelGrouping, sep=\"_\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We modify some names in the selection because it will cause problems in the future models"},{"metadata":{"trusted":true},"cell_type":"code","source":"names(glob_selection)[names(glob_selection) == \"NA_(not set)\"] <- \"NA_not_set\"\nnames(glob_selection)[names(glob_selection) == \"NA_Organic Search\"] <- \"NA_Organic_Search\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's remove the initial variable of the selection"},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_selection$continent <- NULL\nglob_selection$domain <- NULL\nglob_selection$channelGrouping <- NULL","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's transform the visitStartTime variable in order to have two columns constituted of the year date (day from 1 to 365) and the hour."},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_selection$yday <- yday(glob$visitStartTime)\nglob_selection$hour <- hour(glob$visitStartTime)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Label engineering "},{"metadata":{},"cell_type":"markdown","source":"Label engineering of transactionRevenue, we transform it to the log indeed log transformation can be used to make highly skewed distributions less skewed. This can be valuable both for making patterns in the data more interpretable and for helping to meet the assumptions of inferential statistics.\n\nMoreover the final metric is constituted of the prediction logarithm. "},{"metadata":{"trusted":true},"cell_type":"code","source":"glob_selection$transactionRevenue <- log(glob_selection$transactionRevenue)\nglob_selection$transactionRevenue[which(!is.finite(glob_selection$transactionRevenue))] <- 0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualisation of the feature engineering"},{"metadata":{},"cell_type":"markdown","source":"We see a resume of our feature engineering.\nWe have 23 variables !"},{"metadata":{"trusted":true},"cell_type":"code","source":"str(glob_selection, indent.str='\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train separation and sampling"},{"metadata":{},"cell_type":"markdown","source":"Our hardware is not performant so we have to take a sample to make our model training, below we can choose the right number of observation that we want in our sample."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Dataframe sample to take\nn_sample_rows = 1000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Final_test is the data to predict for the submission in the challenge\nfinal_test<- glob_selection[glob_selection$datasplit == 'test']\nid_final_test <- final_test$fullVisitorId\n\n#We remove the Id that will serve for the submission and the transactionRevenue values in the final testing set\nfinal_test$transactionRevenue <- NULL\nfinal_test$fullVisitorId <- NULL\nglob_selection$fullVisitorId <- NULL\n\n#Train_all is all the training set that we will divide in a subtraining and a subtestingset\ntrain_all<- glob_selection[glob_selection$datasplit =='train']\ntrain_all <- sample_n(train_all, n_sample_rows)\ntrain_all$transactionRevenue <- train_all$transactionRevenue %>% replace_na(0)\ntrain_all$datasplit <- NULL","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Partition of the train_all set in subtrain and subtest\nn = nrow(train_all)\ntrainIndex = sample(1:n, size = round(0.7*n), replace=FALSE)\ntrain = train_all[trainIndex ,]\ntest = train_all[-trainIndex ,]\ntestwithTransac <- test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"testlabel <-test$transactionRevenue\n#Remove transaction revenue in the testset\ntest$transactionRevenue <- NULL","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below a visualisation of our subtraining, subtesting sets and the label of the testing set"},{"metadata":{"trusted":true},"cell_type":"code","source":"head(train)\nhead(test)\nhead(testlabel)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model: Dummy Regressor"},{"metadata":{},"cell_type":"markdown","source":"Our first model is a dummy regressor: the regressor to only predict a 0.\nIt wull sere as a baseline for our future models"},{"metadata":{"trusted":true},"cell_type":"code","source":"zero_vector = rep(0,nrow(test))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We make the prediction using the RMSE demanded in the challenge using the log of predicted values and the log+1 of the testlabel."},{"metadata":{"trusted":true},"cell_type":"code","source":"#The target values is already log so we have to exp the testlabel before applying the log1p\ndummy_zero_final_result <- rmse(zero_vector, log1p(exp(testlabel)))\nprint(dummy_zero_final_result)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model 2: Random Forest"},{"metadata":{},"cell_type":"markdown","source":"### Fine tuning"},{"metadata":{"trusted":true},"cell_type":"code","source":"library(mlbench)\nlibrary(caret)\n\ncontrol <- trainControl(method=\"repeatedcv\", number=4, repeats=1, search=\"grid\")\nset.seed(22)\ntunegrid <- expand.grid(mtry=c(1:15))\nrf_gridsearch <- train(transactionRevenue ~., data=train, method=\"rf\", metric='rmse', tuneGrid=tunegrid, trControl=control)\nprint(rf_gridsearch)\nplot(rf_gridsearch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"library(randomForest)\n#Model\nrf.model <- randomForest(transactionRevenue ~., data = train, ntree = 500, mtry=3)\nprint(rf.model) \nplot(rf.model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Importance\n#rf.model$importance\nvarImpPlot(rf.model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Rmse and predictions\npredictions <- rf.model %>% predict(test)\npredictions[predictions < 0] <- 0\n#predictions[predictions < 100] <- 0\nrandom_forest_final_result <- rmse(predictions, log1p(exp(testlabel)))\n\nprint(random_forest_final_result)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model 3: XG Boost "},{"metadata":{"trusted":true},"cell_type":"code","source":"require(xgboost)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train <-train %>% relocate(transactionRevenue)\nhead(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"require(Matrix)\nrequire(ggplot2) ; require(ggthemes) ;\nsparse_matrix <- sparse.model.matrix(transactionRevenue ~ ., data =train)[,-1]\nsparse_matrix_test <- sparse.model.matrix(transactionRevenue ~ .,data =testwithTransac)[,-1]\noutput_vector = train[,transactionRevenue]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"head(sparse_matrix)\nhead(sparse_matrix_test)\nhead(output_vector)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"bestperf = 150\nbestparam_vector = c()\nreglXGB3<-c() \ntt<-Sys.time() ; nn<-0\nfor (eta in 10^seq(-3,-2,length.out = 4)) { #seq(-0.1,-1.5,length.out = 6)\nfor (max.depth in sort(seq(20,50,3))) {\nmtry=0.2#for (mtry in seq(0.2,0.25,by=0.05)) { #seq(0.1,1,length.out = 10)\n gamma=0#for (gamma in 10^(seq(-15,-5,length.out = 20))) { #gamma=0# \n lambda=0#for (lambda in 10^(seq(-3,3,length.out = 20))) {\n xgb.params <- list(\"objective\" = \"reg:squarederror\", \"eta\" = eta, \"subsample\" = 0.9, \"colsample_bytree\" = mtry, \"max_depth\" = max.depth, \"gamma\" = gamma,\"lambda\"=lambda)\n nn<-nn+1\n mod.xgb.cv <- xgb.cv(data = sparse_matrix, params=xgb.params, verbose = 0, nfold=3,maximize=FALSE ,nrounds = 10, early_stopping_rounds = 10, label =output_vector)\n mod.xgb <- xgb.train(data = sparse_matrix, params=xgb.params, nrounds = mod.xgb.cv$best_iteration, verbose = 0,maximize=FALSE)\n pred.xgb <-predict(mod.xgb, sparse_matrix_test,ntreelimit=mod.xgb.cv$best_iteration) # \n perf=rmse(pred.xgb, log1p(testlabel))\n if (perf% predict(final_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"head(sumbmission_predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission <- data.frame(fullVisitorId=id_final_test, PredictedLogRevenue=sumbmission_predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"head(submission)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission <- read_csv(\"../input/ga-customer-revenue-prediction/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission <- submission %>% group_by(fullVisitorId) %>% summarise(PredictedLogRevenue = sum(PredictedLogRevenue))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nrow(submission) == nrow(sample_submission)\nhead(submission)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"write.csv(submission, \"theovaloutput1.csv\",row.names=F)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"3.6.3"}},"nbformat":4,"nbformat_minor":4}