{"cells":[{"metadata":{"_uuid":"c020d117781cb1848693d280568080db310223d4"},"cell_type":"markdown","source":"[Shout out to ML for the original code this is based on. ](https://www.kaggle.com/mrlong/r-flatten-json-columns-to-make-single-data-frame) This is our attempt to consolidate the code and take advantage of some of the other great functions in the `tidyverse`,  most notably the `purrr` package and `map_dfc()`function. "},{"metadata":{"_uuid":"a5666807b55ab3a3f66b3a2a6b32fc5b2bc8b7fb"},"cell_type":"markdown","source":"## Loading in Packages"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"library(tidyverse)\nlibrary(jsonlite)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ceaee02626e28e36536941ce492fc47dd24c4123"},"cell_type":"markdown","source":"## Reading in Data"},{"metadata":{"trusted":true,"_uuid":"1c05e6a5c925a6ea56a1ce777061ddc403889050","_kg_hide-output":true},"cell_type":"code","source":"training_data <- read_csv(file = \"../input/train.csv\", col_names=T)  %>% \n    mutate(DataSplit = \"Training\") \n\ntesting_data <- read_csv(file = \"../input/test.csv\", col_names=T)  %>% \n    mutate(DataSplit = \"Testing\") \n\nfull_data <- bind_rows(testing_data, training_data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fcce7ff4e65e7785eac72c6647c19e4495ca8cff"},"cell_type":"markdown","source":"## Viewing JSON Columns"},{"metadata":{"trusted":true,"_uuid":"769ba959584e3cfbae159278fa5d15b0c96ff9d8"},"cell_type":"code","source":"full_data  %>% \n    select(trafficSource, totals, geoNetwork, device)  %>% \n    head(n = 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8386fe4e53516add25898c6430724ef054049e5e"},"cell_type":"markdown","source":"Ewww Gross! Let's write a `function` to parse these and extract the information! "},{"metadata":{"_uuid":"d0438247334b3cacf413a8bec603fa48c28ae217"},"cell_type":"markdown","source":"## Writing JSON Parsing Function"},{"metadata":{"trusted":true,"_uuid":"44bf814be12e70b49f4bff2f5ca4a73c1dca813e","_kg_hide-output":true},"cell_type":"code","source":"ParseJSONColumn <- function(x)  {\n     str_c(\"[ \", str_c(x, collapse = \",\", sep=\" \"), \" ]\")  %>% \n     fromJSON(flatten = T) %>% \n     as.tibble()\n                                } ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"390469f0035940e4e99455d2e6b1a5ff325cb453"},"cell_type":"markdown","source":"## Mapping `ParseJSONColumn` Into a Consolidated Data Object"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"JSONcolumn_data <- full_data  %>% \n    select(trafficSource, totals, geoNetwork, device)  %>% \n    map_dfc(.f = ParseJSONColumn) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b953bde5f013ad8023a15d25710ed97cf0133840"},"cell_type":"markdown","source":"Now it's nice and *tidy*!"},{"metadata":{"trusted":true,"_uuid":"4473a28f27f6c0d77fd0c47fc4c452746c14df70"},"cell_type":"code","source":"JSONcolumn_data  %>% \n    head(n = 5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c6fc821618eb9d731cc5f46e64d1ebf540cf08ec"},"cell_type":"markdown","source":"##  Binding JSON Column Data Back Onto Full Data Set"},{"metadata":{"trusted":true,"_uuid":"40d09267fc4ca3da868405b208e4c78cb11052f7"},"cell_type":"code","source":"full_data_wJSON <- bind_cols(full_data,JSONcolumn_data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0bfd79aadfb0055911fb73133763bfc2155c8e90"},"cell_type":"markdown","source":"## There are some empty columns that contain no information in the data set after parsing; let's look!"},{"metadata":{"trusted":true,"_uuid":"4053ed7cb0446921043031db2d28e87f4b1cea24"},"cell_type":"code","source":"full_data_wJSON  %>% \ngroup_by_at(vars(starts_with(\"mobile\"),\n                 ends_with(\"titude\"),\n                 starts_with(\"screen\"),\n                 starts_with(regex(\"browser[A-Z]\")),\n                 \"operatingSystemVersion\",\n                 \"flashVersion\",\n                 \"language\",\n                ))  %>% \ntally()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"83fbb4ac1bb0f4e48a48eba87b656fcf49dfa8f0"},"cell_type":"markdown","source":"## Dropping empty columns!"},{"metadata":{"trusted":true,"_uuid":"27193853dc2d87d7635c866376dbee34fc6d8ad9"},"cell_type":"code","source":"full_data_wJSON <- full_data_wJSON  %>% \nselect(-device,\n       -geoNetwork,\n       -totals,\n       -trafficSource,\n       -networkLocation,\n       -operatingSystemVersion,\n       -flashVersion,\n       -language,\n       -starts_with(\"mobile\"),\n       -ends_with(\"itude\"),\n       -starts_with(\"screen\"),\n       -browserVersion,\n       -contains(\".CriteriaParameters\"),\n       -cityId,\n       -browserVersion,\n       -browserSize\n    )","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e53c9489becf1c4ac86bb9aa28e85b02cc1ce9b"},"cell_type":"markdown","source":"No more empty columns !"},{"metadata":{"trusted":true,"_uuid":"9b846424fafc3a7dd18b2a6e6bd397280b7d9b1d"},"cell_type":"code","source":"full_data_wJSON  %>% \n    head(n=5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38d39805915d019298e48fb33012ee6f224189ff"},"cell_type":"markdown","source":" ## Writing Output File"},{"metadata":{"trusted":true,"_uuid":"79aed8d6032b424e301187937e73739655b4edb5"},"cell_type":"code","source":"\nfull_data_wJSON  %>% \n    write_csv(\"full_data_parsed.csv\", col_names = TRUE)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9ad792281c74f271a43af322ac6ae3a0b06f75dc"},"cell_type":"markdown","source":"Hope this helps contribute to your submissions ! :D"}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}