This kernels is gathering information in 4 best kernels.

- [SRK : Simple Exploration+Baseline - GA Customer Revenue](https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue)  
- [Troy Walters : A Very Extensive GStore Exploratoy Analysis](https://www.kaggle.com/captcalculator/a-very-extensive-gstore-exploratory-analysis)  
- [EDA on basic data and LGB (in progress)](https://www.kaggle.com/artgor/eda-on-basic-data-and-lgb-in-progress)  
- [kxx : R EDA for GStore + XGB](https://www.kaggle.com/kailex/r-eda-for-gstore-xgb)


# Introduction 

The 80/20 rule has proven true for many businesses–only a small percentage of customers produce most of the revenue. As such, marketing teams are challenged to make appropriate investments in promotional strategies.

RStudio, the developer of free and open tools for R and enterprise-ready products for teams to scale and share work, has partnered with Google Cloud and Kaggle to demonstrate the business impact that thorough data analysis can have.

In this competition, you’re challenged to analyze a Google Merchandise Store (also known as GStore, where Google swag is sold) customer dataset to predict revenue per customer. Hopefully, the outcome will be more actionable operational changes and a better use of marketing budgets for those companies who choose to use data analysis on top of GA data.

# Retrieving the Data

```{r message=FALSE, warning=FALSE,echo=FALSE,results=FALSE}
if (!require("pacman")) install.packages("pacman")
pacman::p_load(tidyverse, skimr, GGally, plotly, viridis, ggplot2,jsonlite, gridExtra, corrplot, plotly,prophet, rBayesianOptimization, 
               caret, DT, data.table, lightgbm, xgboost, magrittr, tictoc, highcharter, ggrepel, scales, cowplot, knitr, ggalluvial)
set.seed(0)
```

# Glimpse of Data with DT{.tabset .tabset-fade .tabset-pills}

```{r message=FALSE, warning=FALSE,echo=FALSE,results=FALSE}
tr <- read_csv("../input/train.csv") 
te <- read_csv("../input/test.csv") 
gc()
```

## train.csv

```{r message=FALSE}
cat("Dimension of Train.csv : " , dim(tr))

cat("Head of Train.csv")
tr  %>% head(2) %>% kable()

#unique value of bureau_balance
cat("Unique values of Train.csv")
kable(as.data.frame(lapply(tr, function(x) length(unique(x)))))
```

**Check point!!!**  

- You need to change the JSON format for the first time.
- Unique value is 1 like socialEngagementType
- NA values are expressed differently like "(not provided)".

## test.csv

```{r message=FALSE}
cat("Dimension of Test.csv : " , dim(te))

cat("Head of Test.csv")
te  %>% head(2) %>% kable()

#unique value of bureau_balance
cat("Unique values of Test.csv")
kable(as.data.frame(lapply(te, function(x) length(unique(x)))))
```

# Data preprocessing{.tabset .tabset-fade .tabset-pills}

## Json 

The code to modify the JSON format was taken from [EDA in kxx](https://www.kaggle.com/kailex/xgb-for-gstore-1-67/code). (thank)

```{r message=FALSE}
flatten_json <- . %>% 
  str_c(., collapse = ",") %>% 
  str_c("[", ., "]") %>% 
  fromJSON(flatten = T)

parse <- . %>% 
  bind_cols(flatten_json(.$device)) %>%
  bind_cols(flatten_json(.$geoNetwork)) %>% 
  bind_cols(flatten_json(.$trafficSource)) %>% 
  bind_cols(flatten_json(.$totals)) %>% 
  select(-device, -geoNetwork, -trafficSource, -totals)

tr <- parse(tr)
te <- parse(te)
```

## NA

The code to modify the NA was taken from [EDA in kxx](https://www.kaggle.com/kailex/xgb-for-gstore-1-67/code). (thank)

```{r message=FALSE}
is_na_val <- function(x) x %in% c("not available in demo dataset", "(not set)", 
                                  "unknown.unknown", "(not provided)")

tr %<>% mutate_all(funs(ifelse(is_na_val(.), NA, .)))
te %<>% mutate_all(funs(ifelse(is_na_val(.), NA, .)))
```

```{r message=FALSE}
apply(tr,2,function(x) sum(is.na(x)))
```

**Check point!!!**

- transactionRevenue(Target value) has missing values.
- In the case of campaignCode, it has one value. (mostly NA)
- Most of the variables are missing a lot.

## Unique value is 1

```{r message=FALSE}
tr %<>% select_if(function(.) n_distinct(.)>1)
te %<>% select_if(function(.) n_distinct(.)>1)

cat("Unique values of Train.csv")
kable(as.data.frame(lapply(tr, function(x) length(unique(x)))))
```

# Target value : transactionRevenue{.tabset .tabset-fade .tabset-pills}


For reference, transactionRevenue is chr, so it needs to be converted to numeric.
In the following, there is no transactionRevenue value of 0. So NA's can be expected to be zero enough. So I changed it to 0 and started work.

```{r message=FALSE}
tr$transactionRevenue <- as.numeric(tr$transactionRevenue)
summary(tr$transactionRevenue)
tr$transactionRevenue[is.na(tr$transactionRevenue)] <- 0
```

Prior to the analysis, we convert to log (transactionRevenue + 1). This is because the value of the evaluation criterion itself is a log value.

```{r message=FALSE}
tr$transactionRevenue <- log1p(tr$transactionRevenue)
```

```{r message=FALSE}
p1 <- ggplot(data=tr, aes(x=transactionRevenue,y=(..count..)/nrow(tr))) +
        geom_histogram(fill="blue") + xlab("") + ylab("% of log1p(transactionRevenue)")

p2 <- ggplot(data=tr[tr$transactionRevenue>0,], aes(x=transactionRevenue,y=(..count..)/nrow(tr[tr$transactionRevenue>0,]))) +
        geom_histogram(fill="blue") + xlab("") + ylab("% of log1p(transactionRevenue) with None-zero")

grid.arrange(p1,p2,nrow=1)
```

**Check point!!!**

- More than 0 and it is very crowded. RMSE should be focused on this point as it may be better to predict 15-20 segments better than to predict 0 better. 
(Is there a better way to predict this part better?)

## Time Series 

```{r message=FALSE}
library(lubridate)
tr %<>% 
  mutate(date = ymd(date),
         year = year(date),
         month = month(date),
         week = week(date),
         wday = wday(date),
         yday = yday(date),
         qday = qday(date),
         day = day(date))

te %<>% 
  mutate(date = ymd(date),
         year = year(date),
         month = month(date),
         week = week(date),
         wday = wday(date),
         yday = yday(date),
         qday = qday(date),
         day = day(date))

tr$flag <- "train"
te$flag <- "test"
tr_te <- tr  %>% bind_rows(te)
```

```{r message=FALSE}
tr_te_flag <- tr_te  %>% select(flag,date)
tr_te %>% group_by(date) %>% summarise(count = n()) %>% left_join(tr_te_flag, by='date') %>% 
  ggplot() + geom_line(aes(date,count,color=flag)) + theme(legend.title = element_blank(),legend.text = element_blank())
```

**Check point!!!**

Customer Visits Side

- The trend that seems unclear.
- It takes a peak in December and a bottom in January. This is the same phenomenon in both 17 and 18 years.
- There are suddenly increasing values in the middle, which is suspicious holidays.
- Visits increase from October to December. (but, the increase in 2017 is weak.)

```{r message=FALSE}
p1 <- tr %>% group_by(date) %>% summarise(sum_transactionRevenue = sum(transactionRevenue)) %>% 
  ggplot() + geom_line(aes(date,sum_transactionRevenue)) + theme(legend.title = element_blank(),legend.text = element_blank())

p2 <- tr %>% group_by(date) %>% summarise(sum_transactionRevenue = sum(transactionRevenue),count = n()) %>% 
  ggplot() + geom_line(aes(date,sum_transactionRevenue/count)) + ylab("Ratio of sum(Revenue)/count") +theme(legend.title = element_blank(),legend.text = element_blank())

grid.arrange(p1,p2)
```

**Check point!!!**

transactionRevenue

- Above is the sum of transactionRevenue and below is the value divided by the customer's number of visits. 
They are very similar in appearance. Is not it strange?

```{r message=FALSE}
p1 <- tr  %>% filter(transactionRevenue > 0)  %>% group_by(date) %>% summarise(count = n())  %>% ggplot() + geom_line(aes(date,count)) + ggtitle("count of transaction Revenue > 0") +xlab("")+ ylab("")
p2 <- tr  %>% filter(transactionRevenue == 0) %>% group_by(date) %>% summarise(count = n()) %>% ggplot() + geom_line(aes(date,count)) + ggtitle("count of transaction Revenue is 0") +xlab("")+ ylab("")
grid.arrange(p1,p2,nrow=2)
```

**Check point!!!**

Customer Visits Side

- The graphs similar but 2016 Nov,Dec very diff.



```{r message=FALSE}
p1 <- tr %>% filter(transactionRevenue>0) %>% ggplot() + geom_boxplot(aes(x=as.factor(year),y=transactionRevenue)) + xlab("year")
p2 <- tr %>% filter(transactionRevenue>0) %>% ggplot() + geom_boxplot(aes(x=as.factor(month),y=transactionRevenue)) + xlab("month")
p3 <- tr %>% filter(transactionRevenue>0) %>% ggplot() + geom_boxplot(aes(x=as.factor(wday),y=transactionRevenue)) + xlab("week of day")

grid.arrange(p1,p2,p3,nrow=3)
```


**Check point!!!**

- Is not the 6th day of August 2016 an abnormal date? Some value is too small.
- Except for that, there is only a slight difference and the trend is similar.

Prophet

```{r message=FALSE}
tr1 <- tr %>% group_by(date) %>% summarise(sum_transactionRevenue = sum(transactionRevenue))
te1 <- te %>% mutate(transactionRevenue = 0) %>% group_by(date) %>% summarise(sum_transactionRevenue = sum(transactionRevenue))

names(tr1) <- c('ds', 'y');names(te1) <- c('ds', 'y')
```

```{r message=FALSE}
m = prophet(tr1)
forecast <- predict(m, periods=213)
prophet_plot_components(m,forecast)
```

**Check point!!!**

- There isn't trend.
- There seems to be some WEEKLY trends.

# Others{.tabset .tabset-fade .tabset-pills}

## device 

The specifications for the device used to access the Store.

```{r message=FALSE, warning=FALSE}
p1 <- tr %>% group_by(channelGrouping) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~channelGrouping, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p2 <- tr %>% group_by(browser) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~browser, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p3 <- tr %>% group_by(operatingSystem) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~operatingSystem, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p4 <- tr %>% group_by(deviceCategory) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~deviceCategory, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))


subplot(list(p1,p2,p3,p4),nrows=2,shareY=TRUE, titleX = TRUE, titleY = TRUE)  %>% layout(showlegend = FALSE)
```

- From the first graph,  *channel, browser, operatingSystem. deviceCategory.*

## geoNetwork 

This section contains information about the geography of the user.

```{r message=FALSE, warning=FALSE}
p1 <- tr %>% group_by(continent) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~continent, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "The continent percentage.",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p2 <- tr %>% group_by(subContinent) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~subContinent, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "The subContinent percentage",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p3 <- tr %>% group_by(country) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~country, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "The country percentage",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p4 <- tr %>% group_by(region) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~region, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "The region percentage.",
         xaxis = list(title = ""),
         yaxis = list(title = ""))
```

```{r message=FALSE, warning=FALSE}
subplot(list(p1,p2,p3,p4),nrows=2,shareY=TRUE, titleX = TRUE, titleY = TRUE)  %>% layout(showlegend = FALSE)
```

- From the first graph,  *continent, subContinent, country. region.*

```{r message=FALSE, warning =FALSE}
p5 <- tr %>% group_by(metro) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~metro, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p6 <- tr %>% group_by(city) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~city, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))
         
p7 <- tr %>% group_by(networkDomain) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~networkDomain, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

subplot(list(p5,p6,p7),nrows=2,shareY=TRUE, titleX = TRUE, titleY = TRUE)  %>% layout(showlegend = FALSE)
```

- From the first graph,  *metro, city, networkDomain.*


## Totals 

This section contains aggregate values across the session.


```{r message=FALSE, warning=FALSE}
p1 <- tr %>% group_by(campaign) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~campaign, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p2 <- tr %>% group_by(source) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~source, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p3 <- tr %>% group_by(medium) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~medium, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p4 <- tr %>% group_by(keyword) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~keyword, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

subplot(list(p1,p2,p3,p4),nrows=2,shareY=TRUE, titleX = TRUE, titleY = TRUE)  %>% layout(showlegend = FALSE)
```

- From the first graph,  *campaign, source, medium, keyword*

```{r message=FALSE, warning =FALSE}
p5 <- tr %>% group_by(referralPath) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~referralPath, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p7 <- tr %>% group_by(adwordsClickInfo.slot) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~adwordsClickInfo.slot, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))
         
p6 <- tr %>% group_by(adwordsClickInfo.page) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~adwordsClickInfo.page, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))
         
p8 <- tr %>% group_by(adwordsClickInfo.adNetworkType) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~adwordsClickInfo.adNetworkType, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))
         
subplot(p5,p7,p6,p8,nrows=2,shareY=TRUE)
```

- From the first graph,  *referralPath, adwordsClickInfo.slot, adwordsClickInfo.page, adwordsClickInfo.adNetworkType*

## TrafficSource 

This section contains information about the Traffic Source from which the session originated.


```{r message=FALSE, warning=FALSE}
p1 <- tr %>% group_by(hits) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~hits, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p2 <- tr %>% group_by(pageviews) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~pageviews, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p3 <- tr %>% group_by(bounces) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~bounces, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))

p4 <- tr %>% group_by(newVisits) %>%
    count() %>% arrange(desc(n)) %>%  plot_ly( x = ~newVisits, y = ~ (n/nrow(tr))*100, type = 'bar',showlegend=TRUE,
        marker = list(color = 'rgb(000,051,255)',
                                  width = 1.5)) %>%
    layout(title = "",
         xaxis = list(title = ""),
         yaxis = list(title = ""))
         
subplot(p1,p2,p3,p4,nrows=2,shareY=TRUE)
```

- From the first graph,  *hits, pageviews, bounces, newVisits*


## campaignCode

```{r message=FALSE}
unique(tr$campaignCode)
tr %>% filter(campaignCode == "11251kjhkvahf")
```

**Check point!!!**

- there is no variable in the test file, there is only one ROW and it has no meaning (i think), it is probably ok to delete the variable itself.



# Question

How do you think it?

- More than 0 and it is very crowded. RMSE should be focused on this point as it may be better to predict 15-20 segments better than to predict 0 better. 
(Is there a better way to predict this part better?)
- Above is the sum of transactionRevenue and below is the value divided by the customer's number of visits. 
They are very similar in appearance. Is not it strange?