{"cells":[{"metadata":{"_uuid":"07d197a58170f22c4439138cb14eea54cf206cd8"},"cell_type":"markdown","source":"# Google Analytics Customer Revenue Prediction\n\nTOC\n\n- [Information](#information)\n- [Load Packages](#load-packages)\n- [Define Funciton](#define-function)\n- [Data Import](#data-import)\n\n* * *\n"},{"metadata":{"_uuid":"2946b09ce3df6923ee940a02fd9b826d80636d35"},"cell_type":"markdown","source":"## 💡 Information\n\nThis notebook is the first step of the flow we are working on for [the predict revenue per customer](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction).\n\nWe intend to read the data set prepared for the competition in R and convert it to *the [tidy format](https://tidyr.tidyverse.org/articles/tidy-data.html)* for the next phase."},{"metadata":{"_uuid":"924599e045bb129dd66eb73aa7e5f705738c18aa"},"cell_type":"markdown","source":"## 📦 Load Packages"},{"metadata":{"_uuid":"d7c33f5d664ed5242341508a7032149e85f2929a","_execution_state":"idle","trusted":true,"scrolled":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"# library(conflicted)\nlibrary(tidyverse) # General data processing packages \nlibrary(assertr)   # Data assertion\nlibrary(jsonlite, warn.conflicts = FALSE) # Read JSON format into R\n\n# library(lubridate)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8db7e3d530080cab4fb31feee4f8d4c2fcd672d9"},"cell_type":"markdown","source":"## ✨ Define Funciton\n\nThis notebooks using these functions.\n\n- `nested_json_flatten()`:  Parse JSON include column to tidy format\n- `col_types_generator()`:  Create a string declaring column sets for input datasets.\n- `read_input_csv()`:  Reading input dataset (include `col_types_generator()`)"},{"metadata":{"trusted":true,"_uuid":"3780f702a26149df2cf30bf1c744a60a48bb1619","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"nested_json_flatten <- function(.data,\n                                json_vars = NULL) {\n    \n    if (is.null(json_vars) == TRUE) json_vars = c(\"device\", \"geoNetwork\", \"totals\", \"trafficSource\")\n    \n  .data %>% \n    dplyr::select(-dplyr::one_of(json_vars)) %>% \n    dplyr::bind_cols(\n      json_vars %>% \n        purrr::map_dfc(\n          ~ paste(\"[\", paste(.data[[.x]], collapse = \",\"), \"]\") %>% \n            jsonlite::fromJSON(flatten = TRUE))      \n    )\n}\n\ncol_types_generator <- function(.data) {\n  col_vars_candidate = \n    .data %>% \n    purrr::map_chr(class)\n  col_vars = \n    dplyr::case_when(col_vars_candidate == \"character\" ~ \"c\",\n              col_vars_candidate == \"integer\" ~ \"i\",\n              col_vars_candidate == \"numeric\" ~ \"d\") %>% \n    magrittr::set_names(names(col_vars_candidate)) %>% \n    as.list() %>% \n    purrr::list_modify(date    = \"D\",\n                       visitId = \"c\") %>% \n    purrr::flatten_chr() %>% \n    stringr::str_c(collapse = \"\")\n  col_vars\n}\n\nread_input_csv <- function(file) {\n  \n  col_vars = \n    readr::read_csv(file, \n                    n_max = 1000) %>% \n    assertr::verify(ncol(.) == 12) %>% \n    col_types_generator()\n  \n  readr::read_csv(file,\n                    col_types = col_vars,\n                    locale = readr::locale(date_format = \"%Y%m%d\"))\n}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ee095697529222bfc224156a7fafca72c4a8400"},"cell_type":"markdown","source":"## 🥚 Data Import\n\nFirst of all, we check what kind of data structure the variable from part of train dataset. If R correctly recognizes the data type, use it. If not, apply the appropriate type."},{"metadata":{"_uuid":"31f4cc8dd9fe9c9b01a5a99048943007188ee431"},"cell_type":"markdown","source":"### Train dataset"},{"metadata":{"trusted":true,"_uuid":"3cc2fe21ad1e63211011c4ab7b1ad6bf4e3ea4ff","_kg_hide-output":true,"scrolled":false},"cell_type":"code","source":"# read_csv() column types imputed from the first 1000 rows on the input.\n# Therefore, even 1000 samples of sample data will be used.\ndf_sample = \n      read_csv(\"../input/train.csv\", \n               n_max = 1000) %>% \n      verify(ncol(.) == 12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b202d5f58f13d993a04832866e2172dd92017ed3","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"glimpse(df_sample)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39014ab115ddccb06b7dde46d732c6be6b1bfd85"},"cell_type":"markdown","source":"⚠️ For automatic loading, the date column, which should indicate the date, is of integer type. In addition, the unique key `visitId` is integer type. I think that it is good to keep this as a character string like `fullVisitorId` which is another [unique key](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/data). However, I do not care about the numerical value of the POSIX time in the `visitStartTime` column.\n\nTo specify an arbitrary data type for each column when reading with `read_csv()`, give the `col_types` argument a value indicating the data type of the column."},{"metadata":{"trusted":true,"_uuid":"ae13ad15fcbf26b0d1e000bc360b36314a8f1d8d"},"cell_type":"code","source":"col_vars_candidate = \n      df_sample %>% \n      map_chr(class)\n\ncol_vars = \n      case_when(col_vars_candidate == \"character\" ~ \"c\",\n                col_vars_candidate == \"integer\" ~ \"i\",\n                col_vars_candidate == \"numeric\" ~ \"d\") %>% \n      purrr::set_names(names(col_vars_candidate)) %>% \n      as.list() %>% \n      list_modify(date    = \"D\",\n                  visitId = \"c\") %>% \n      flatten_chr() %>% \n      str_c(collapse = \"\")\n\n# - c: Character\n# - D: date\n# - d: double\ncol_vars","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66e52c080702ca3acba4fc4621c961ff0db590cb"},"cell_type":"markdown","source":"This is not the end. You also need to specify `locale` to parse format the date column.\n"},{"metadata":{"trusted":true,"_uuid":"793e7b6a41fcc54c65abf7f883081161f9bbf8ec"},"cell_type":"code","source":"df_train_raw = \n      read_csv(\"../input/train.csv\",\n               col_types = col_vars,\n             locale = readr::locale(date_format = \"%Y%m%d\")) %>%\n      verify(dim(.) == c(903653, 12))\n\nglimpse(df_train_raw)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"630cbbef0725a1123d24542947d6a7311bcf4188"},"cell_type":"markdown","source":"This dataset has a column with values stored in JSON format which 4 columns starting with brackets.\n\n- `device`\n- `geoNetwork`\n- `totals`\n- `trafficSource`\n\nIt parses the value stored here and handles it as a column as follows."},{"metadata":{"trusted":true,"_uuid":"19e858197c06b6f71f6263465d50ab0479c8e708","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"# The same approach is also written in other scripts and notes.\n# paste(\"[\", paste(df_train_raw[[\"device\"]], collapse = \",\"), \"]\") %>% \n#  jsonlite::fromJSON(flatten = TRUE)\n\njson_vars = c(\"device\", \"geoNetwork\", \"totals\", \"trafficSource\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"802c007036df3f04947d93580f60982e60430f5f","scrolled":true},"cell_type":"code","source":"df_train_flatten =\n      nested_json_flatten(df_train_raw, json_vars = json_vars) %>% \n      verify(dim(.) == c(903653, 55))\n\nglimpse(df_train_flatten)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b62555f95840f586c8905b68ac8b41954e0899b6"},"cell_type":"markdown","source":"### Test dataset\n\nExecute through all processes 🚀"},{"metadata":{"trusted":true,"_uuid":"39e79a830b9260689f66e6f7595aa7ec2681cb05","_kg_hide-output":true,"scrolled":true},"cell_type":"code","source":"df_test_flatten = \n      read_input_csv(\"../input/test.csv\") %>% \n      nested_json_flatten(json_vars = json_vars) %>% \n      verify(dim(.) == c(804684, 53))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"50c7428f10a4c44d93ff92822780a207afc01638"},"cell_type":"code","source":"glimpse(df_test_flatten)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}