---
title: "Using R tidyeval to expand JSON into columns"
author: "Mark Edmondson"
date: "9/14/2018"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

## Import

Import the data and create a sample dataset

```{r message=FALSE, warning=FALSE}
library(readr)
library(tidyverse)
library(jsonlite)

train <- read_csv("../input/train.csv")

train_s <- head(train)
```

## Process

Get rid of the json columns using jsonlite and tidyeval.

JSON parsing taken from this [kaggle](https://www.kaggle.com/mrlong/r-flatten-json-columns-to-make-single-data-frame) by Mr Long which had a much quicker way to parse the JSON.

```{r}
json_to_columns <- function(df, column){
  column <- enquo(column)
  
  json_df <- df %>% 
    pull(!!column) %>% 
    paste(collapse = ",") %>% 
    paste("[",.,"]") %>% 
    fromJSON(flatten = TRUE)
  
  df %>% 
    select(-!!column) %>% 
    bind_cols(json_df)
}

all_data <- train_s %>% 
  json_to_columns(device) %>% 
  json_to_columns(geoNetwork) %>% 
  json_to_columns(totals) %>% 
  json_to_columns(trafficSource)
  
names(all_data)
```

On this system for all data:

```{r}
n <- Sys.time()
all_data <- train %>% 
  json_to_columns(device) %>% 
  json_to_columns(geoNetwork) %>% 
  json_to_columns(totals) %>% 
  json_to_columns(trafficSource)
  
cat("\n# Time taken: ", format(Sys.time() - n))
nrow(all_data)
names(all_data)
```