{"cells":[{"metadata":{"_uuid":"80e403b488bb3937c94a401eefe5af8ae4607b24","_execution_state":"idle"},"cell_type":"markdown","source":"# Fast Data Exploration and Model creation using R\n\nHi Kagglers! \nI decided to enter \"Google Analytics Customer Revenue Prediction\" competition with a team and I am sharing some of my work in this notebook! Hope you find it useful in some way and please comment for any suggestions! :)  \n\nI have done two previous notebooks in Kaggle, one for the well known Titanic competition [here](https://www.kaggle.com/rochellesilva/simple-tutorial-for-beginners) and another where I deal with an imbalanced dataset [here](https://www.kaggle.com/rochellesilva/model-creation-for-indians-diabetes-dataset). Note that these are both done in Python. \n\n        Data Exploration Analysis (EDA) is the first step to start any Data Modeling work.\n        However, depending on the dataset we have, it can be quick or quite complex and tiring process.\n\nAlthough Google Analytics Customer Revenue Prediction hasn't a quick EDA process lets see if we can make it simple, just for a start. For a very complete EDA on this dataset and to fully understand the data, I would suggest checking this [kernel](https://www.kaggle.com/diogomenezesborges/full-depth-eda-analysis) (which is in Python).\n\nThis notebook will be divided into 2 parts:\n\n **1. Take a look at the data:**\n - Check the features\n - Json in columns?! This needs a change.\n - Which features can be dropped which should be kept? (use your intuition!)\n   \n**2. Try some Modeling**\n - Divide data in Training and Validation set\n"},{"metadata":{"_uuid":"dc1cda3ae3265b7309cda05e3e05c2d6450a8a2b"},"cell_type":"markdown","source":"# 1. Take a look at the data:"},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"c7b26fcb2b66694a287bd36b6cbd4447f324c17b"},"cell_type":"code","source":"# load libraries\nlibrary(readr)\nlibrary(stringr)  \nlibrary(dplyr)\nlibrary(jsonlite)\nlibrary(lubridate)\nlibrary(gbm)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1d927f9a37616e86e24fc06db2883017659a5424","_kg_hide-output":true},"cell_type":"code","source":"#load the datasets\ntrain <- read_csv(\"../input/train.csv\")\ntest <- read_csv(\"../input/test.csv\")\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f0437db514a02a7a87fcfcb327d481c037887960"},"cell_type":"code","source":"paste0(\"Train has \", dim(train)[1], \" observations and \", dim(train)[2], \" columns.\")\npaste0(\"Test has \", dim(test)[1], \" observations and \", dim(test)[2], \" columns.\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b2f6fa83b0743e10f84bb9a5d3ef9e5f2f13e01e"},"cell_type":"markdown","source":"So we have quite a lot observations, but features are just 12. Lets see what features we have and evaluate them."},{"metadata":{"trusted":true,"_uuid":"74d79c8de0f9503bb85f0b2e812bfef77f84df35"},"cell_type":"code","source":"names(train)\nnames(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1288fc77f483805413cd9f3a0c1bd850b1458674"},"cell_type":"code","source":"head(train,2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2c7b02b690936d4b436ef417145dbdddc77458e4"},"cell_type":"markdown","source":"From the 12 columns, 4 of them are in JSON format (**device, geoNetwork, trafficSource and totals**) so lets transform all their information into columns. In R it is possible to use the jsonlite package but due to inconsistency of the data form the trafficSource column we need some addional tricks, I used the code from this [Kernel](https://www.kaggle.com/kailex/r-eda-for-gstore-glm-keras-xgb)."},{"metadata":{"trusted":true,"_uuid":"d9adc18a13e7fd122adf0447274ab5accbacea22"},"cell_type":"code","source":"flatten_json <- . %>% \n  str_c(., collapse = \",\") %>% \n  str_c(\"[\", ., \"]\") %>% \n  fromJSON(flatten = T)\n\nparse <- . %>% \n  bind_cols(flatten_json(.$device)) %>%\n  bind_cols(flatten_json(.$geoNetwork)) %>% \n  bind_cols(flatten_json(.$trafficSource)) %>% \n  bind_cols(flatten_json(.$totals)) %>% \n  select(-device, -geoNetwork, -trafficSource, -totals)\n\ntest <- parse(test)\ntrain <- parse(train)\n\nhead(train, 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aa3fcfd8a0469e0b9fdd9209c9ade2a5618da83b"},"cell_type":"code","source":"names(train)\nprint(\"Dimensions are now:\")\ndim(train)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"521bc6e597191dba0d9d9269b31352b8c4cb905f"},"cell_type":"markdown","source":"Now we can see clearly the column we want to predict, which is  **'transactionRevenue'**, and just like that we went from 12 to 54 features!!  \n\nAs I explained previously the objective here is to make it simple, so lets just select some columns we think it makes sense to use for this prediction (use your intuition). \n\nSo I gave a look at each feature and chose these: \n **\"channelGrouping\"**, \n **\"hits\"**,\n **\"pageviews\"**,\n **\"visitNumber\"**,\n **\"visitStartTime\"**, \n **\"browser\"**, \n **\"operatingSystem\"**,\n **\"deviceCategory\"**,\n **\"continent\"**,\n **\"country\"**,\n **\"campaign\"**.\n\nNow from these lets first check the levels in each categorical see if they need any change and convert them into factors. Then we will convert the ones that should be numeric into numeric.\n\n"},{"metadata":{"trusted":true,"_uuid":"3f1eb42a59d685bdffdd67504b0469e41806543e","_kg_hide-output":true},"cell_type":"code","source":"features <- c(\"channelGrouping\", \"hits\", \"pageviews\", \"visitNumber\",\n              \"visitStartTime\", \"browser\", \"operatingSystem\", \"deviceCategory\",\n              \"continent\", \"country\", \"campaign\", \"transactionRevenue\")\n\ntrain <- train[, features]\n\nprint(\"For channelGrouping:\")\ntable(train$channelGrouping)\n\nprint(\"For browser:\")\ntable(train$browser)\n\nprint(\"For operatingSystem:\")\ntable(train$operatingSystem)\n\nprint(\"For deviceCategory:\")\ntable(train$deviceCategory)\n\nprint(\"For continent:\")\ntable(train$continent)\n\nprint(\"For country:\")\ntable(train$country)\n\nprint(\"For campaign:\")\ntable(train$campaign)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"debe404816e5b343fc2ca02109c6bdeadf606544"},"cell_type":"markdown","source":"When we have **categorical features which have a lot of levels** it is not that useful for the model specially if many of those levels have little observations. Thus, for this cases we will see the levels that have more obsevations and all the remaining will go to a **new level \"Other\"**."},{"metadata":{"trusted":true,"_uuid":"0fc5b8283381fb2b0d7910e62e056d625e4f0dc8"},"cell_type":"code","source":"train[(train$browser!= \"Safari\") &\n      (train$browser!=\"Internet Explorer\") &\n      (train$browser != \"Firefox\"), \"browser\"] <- \"Other\"\n\n\ntrain[(train$operatingSystem!= \"Android\") &\n      (train$operatingSystem!=\"Chrome OS\") &\n      (train$operatingSystem != \"iOS\") &\n      (train$operatingSystem != \"Linux\") &\n      (train$operatingSystem != \"Macintosh\") &\n      (train$operatingSystem != \"Windows\"), \"operatingSystem\"] <- \"Other\"\n\n\ntrain[(train$country!= \"United States\") &\n      (train$country!=\"Vietnam\") &\n      (train$country != \"United Kingdom\") &\n      (train$country != \"Brazil\") &\n      (train$country != \"Canada\") &\n      (train$country != \"France\") &\n      (train$country != \"Germany\") &\n      (train$country != \"India\") &\n      (train$country != \"Italy\") &\n      (train$country != \"Japan\") &\n      (train$country != \"Mexico\") &\n      (train$country != \"Netherlands\") &\n      (train$country != \"Russia\") &\n      (train$country != \"Spain\") &\n      (train$country != \"Taiwan\") &\n      (train$country != \"Turkey\"), \"country\"] <- \"Other\"\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"27d9ac880b1947de2a58e465dd2bed62ca8c059f"},"cell_type":"markdown","source":""},{"metadata":{"trusted":true,"_uuid":"f2343998fe55105d9b17cfc8706971caec95ed5b"},"cell_type":"code","source":"#converting into factors\ncategorical <- c(\"channelGrouping\", \"browser\", \"operatingSystem\", \"deviceCategory\", \"continent\", \"country\", \"campaign\")\ntrain[, categorical] <- lapply(train[, categorical], as.factor)\n\n\n#converting into numeric\nnumeric <- c(\"hits\", \"pageviews\", \"visitNumber\", \"transactionRevenue\")\ntrain[, numeric] <- lapply(train[, numeric], as.numeric)\n\n# transactionRevenue has many Nas which in reality are equivalent to having 0\ntrain[is.na(train$transactionRevenue), \"transactionRevenue\"] <- 0\n\n# convert visit StartTime from Unix timestamp to POSIXct and get useful features from it:\ntrain$visitStartTime <- as.POSIXct(as.numeric(train$visitStartTime), origin = '1970-01-01', tz = 'GMT')\ntrain$Year <- year(train$visitStartTime)\ntrain$Year <- as.factor(train$Year)\ntrain$Month <- month(train$visitStartTime)\ntrain$Month <- as.factor(train$Month)\ntrain$Hour <- hour(train$visitStartTime)\ntrain$Hour <- as.factor(train$Hour)\n \ntrain$visitStartTime <- NULL","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d88dd04d9b3000d0a22db733ab8e76805e944ac"},"cell_type":"markdown","source":"# Try some modeling"},{"metadata":{"trusted":true,"_uuid":"0faaeeabf1d333f0058b6c48d5797e3a971b8ce2"},"cell_type":"code","source":"# split train into train and validation\n#applying log to revenue\nset.seed(123)\ntrain$transactionRevenue <- log(train$transactionRevenue)\ntrain$transactionRevenue[train$transactionRevenue < 0] <- 0\ntrain.validation.split <- sample(2\n                                 , nrow(train)\n                                 , replace = TRUE\n                                 , prob = c(0.7, 0.3))\ndf_train = train[train.validation.split == 1,]\ndf_validation = train[train.validation.split == 2,]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b7ea116d38f1b22a2af4b316ab2e4b8639fd11b"},"cell_type":"code","source":"#based on https://www.kaggle.com/tatianass/gbm-google-store-revenue\nset.seed(123)\n# Get the time to train the GBM model\nsystem.time(\n  gbm.model <- gbm(transactionRevenue ~ .\n                   , distribution = \"gaussian\"\n                   , data = df_train\n                   , n.trees = 700 # number of trees\n                   , interaction.depth = 3\n                   , n.minobsinnode = 100 # minimum number of obs needed in each node\n                   , shrinkage = 0.01 # learning rate\n                   , bag.fraction = 0.5 # subsampling\n                   , train.fraction = 0.5\n                   , cv.folds = 10      # do 10-fold cross-validation\n                   , verbose = FALSE  # don't print progress\n  )\n)\n\n\nbest.iter = gbm.perf(gbm.model, method = \"test\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33faad835e0a07df21c4764bec117ab4917967e2"},"cell_type":"code","source":"RMSE = function(m, o){\n  sqrt(mean((m - o)^2))\n}\n\n# Plot and calculate AUC on test data\ngbm.test = predict(gbm.model, newdata = df_validation, n.trees = best.iter)\n\nprint(RMSE(df_validation$transactionRevenue, gbm.test))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"977a63cb18e6ca31d06374e16a69e50685117561"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}