{"cells":[{"metadata":{"_uuid":"12a43efd7731f53297a823bf72f999589f10f982"},"cell_type":"markdown","source":"**Load libraries**"},{"metadata":{"trusted":true,"_uuid":"12c3569fcce47dc73cec7fcfbd10bcc28d8e0de3"},"cell_type":"code","source":"library(data.table)\nlibrary(dplyr)\nlibrary(ggplot2)\nlibrary(plotly)\nlibrary(gridExtra)\n\noptions(warn=-1)\nrm(list = ls())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2a3c2352f889bff7a65aa52e5eba008a5725174b"},"cell_type":"markdown","source":"**Reading Processed Dataeset**\n \n I used ** Julien's** kernal to process my dataset. [https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields/notebook] \n \n **Goals :**\n    1. We will look at the structute of data.\n    2. Missing values and the variables which we can drop from our dataset\n    3  Time Series plots of the Dependent variable and visits.\n    4. Distribution of certain categorical variables   "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"cat('Reading data \\n')\n\ntrain = data.frame(fread(\"../input/google-analytics/train_json.csv\"))\ntest = data.frame(fread(\"../input/google-analytics/test_json.csv\"))\n\ntrain$date <- as.Date(as.character(train$date), format='%Y%m%d')\ntest$date <- as.Date(as.character(test$date), format='%Y%m%d')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"90cc3b23c6c18377f3011c9956b13edb8bb3a5d1"},"cell_type":"code","source":"head(train)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"## Peek through the data\n## Distinct count of Visitors\n\n## Train\nsprintf('Unique IDs in train dataset : %s',length(unique(train$fullVisitorId))) \n\n## Test\nsprintf('Unique IDs in test dataset : %s',length(unique(test$fullVisitorId)))\n\n## Intersection of visitors in both train and test\nsprintf('Intersection of train and test : %s',length(intersect(unique(train$fullVisitorId),unique(test$fullVisitorId))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a02ab19d59eacf448e6b6d63d45206ee5698c5ad"},"cell_type":"code","source":"cat('Train \\n')\nsprintf('Train Start Date -- %s',min(train$date))\nsprintf('Train End Date -- %s',max(train$date))\n\ncat('Test \\n')\nsprintf('Test Start Date -- %s',min(test$date))\nsprintf('Test End Date -- %s',max(test$date))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff50e4290ad5caf4d8dcaa4afd9bdbf5e5de77cf"},"cell_type":"code","source":"## Missing values and distinct elements in each variable\nvariable = data.frame()\n\nfor(i in names(train))\n{\n  x = data.frame('Variable' = i)\n  k = data.frame(table(train[,i]))\n  if(nrow(k)==1)\n  {\n    x$Distinct = 1\n  }\n  \n  if(nrow(k)>1)\n  {\n    x$Distinct = length(unique(train[,i]))\n  }\n  \n  x$Missing  = sum(is.na(train[,i]))/nrow(train)\n  \n  variable = rbind(variable,x)  \n}\n\nvariable","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f7a7e704e489cff04a4d6229407ae3f6ece31a6"},"cell_type":"markdown","source":"Please NOTE that variables with just **one unique** element can be removed from our training sample as it does help the model to capture any variance in the target variable "},{"metadata":{"trusted":true,"_uuid":"dfedbe5866834ae218fef11bc84688de9c2f23d4"},"cell_type":"code","source":"x = as.character(variable$Variable[variable$Distinct==1])\n\ntrain = train[!names(train) %in% x]\ntest  = test[!names(test) %in% x]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee879198ccdb36f5f0a372a46bf2cb1a8652056c"},"cell_type":"code","source":"date_df = train %>% group_by(date) %>% summarise(Visits = n())\n\ndate = ggplot(data = date_df, aes(x=date_df$date, y=date_df$Visits)) + \n       geom_line(color='tomato3') +\n       geom_smooth(color='slateblue3') + \n       xlab(\"Time Period\") +\n       ylab(\"Visit Count in Thousands\") +\n       ggtitle(paste(\"Time-Series plot of visits\"))\n\nplot(date)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2aea1445d6dd0e64fea7d7168b558e19746a8dc5"},"cell_type":"code","source":"dv_variation = train %>% group_by(fullVisitorId) %>% summarise(Revenue = log(sum(totals.transactionRevenue,na.rm = T)))\n\nx = ggplot(data=dv_variation, aes(dv_variation$Revenue)) +\n  geom_histogram(colour=\"black\", fill=\"green4\") + \n  scale_size_area() + \n  xlab(\"Revenue (logged)\") +\n  ylab(\"Distribution\") +\n  ggtitle(paste(\"Histogram of the Dependent Variable\"))\n\ndv_variation_date = train %>% group_by(date) %>% summarise(Revenue = log(sum(totals.transactionRevenue,na.rm = T)))\n\nx2 = ggplot(data = dv_variation_date, aes(x=dv_variation_date$date, y=dv_variation_date$Revenue)) + \n       geom_line(color='tomato3') +\n       geom_smooth(color='slateblue3') + \n       xlab(\"Time Period\") +\n       ylab(\"Revenue (logged)\") +\n       ggtitle(paste(\"Time-Series plot of Revenue\"))\n\ngrid.arrange(x, x2, nrow=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"03580e5ed62d8ba8ab9f514c0922e1009cd518b7"},"cell_type":"code","source":"## Basic variable Distributions\n\n## Channel Grouping\nk = train %>% group_by(channelGrouping) %>% summarise(Count = n())\nk$Percentage = round((k$Count/sum(k$Count))*100,2)\n\np = ggplot(data=k, aes(x=k$channelGrouping, y=k$Percentage)) +\n    geom_bar(stat=\"identity\", fill=\"gold2\")+\n    theme_minimal() + \n    scale_size_area() + \n    xlab(\"Channel Grouping\") +\n    ylab(\"Count Percentage\") +\n    ggtitle(paste(\"Distribution of Channel Groups\")) + \n    geom_text(aes(label=paste0(k$Percentage,\"%\")), position=position_dodge(width=0.8), vjust=-0.5) + \n\ttheme(axis.text.x = element_text(angle=90/2, vjust=0.5)) + \n\tscale_y_continuous(labels = function(x) paste0(x, \"%\"))\n\n## Device Browser\nk1 = train %>% group_by(device.browser) %>% summarise(Count = n())\nk1$Percentage = round((k1$Count/sum(k1$Count))*100,2)\n\nk1 = k1 %>% arrange(desc(Percentage)) %>% head(10)\n\np2 = ggplot(data=k1, aes(x=k1$device.browser, y=k1$Percentage)) +\n    geom_bar(stat=\"identity\", fill=\"gold2\")+\n    theme_minimal() + \n    scale_size_area() + \n    xlab(\"Device Browser\") +\n    ylab(\"Count Percentage\") +\n    ggtitle(paste(\"Distribution of Device Browser based of Frequency Distribution\")) + \n    geom_text(aes(label=paste0(k1$Percentage,\"%\")), position=position_dodge(width=0.8), vjust=-0.5) + \n\ttheme(axis.text.x = element_text(angle=90/2, vjust=0.5)) + \n\tscale_y_continuous(labels = function(x) paste0(x, \"%\"))\n\n## Device Category\nk2 = train %>% group_by(device.deviceCategory) %>% summarise(Count = n())\nk2$Percentage = round((k2$Count/sum(k2$Count))*100,2)\n\np3 = ggplot(data=k2, aes(x=k2$device.deviceCategory, y=k2$Percentage)) +\n    geom_bar(stat=\"identity\", fill=\"slateblue3\")+\n    theme_minimal() + \n    scale_size_area() + \n    xlab(\"Device Category\") +\n    ylab(\"Count Percentage\") +\n    ggtitle(paste(\"Distribution of Device Category\")) + \n    geom_text(aes(label=paste0(k2$Percentage,\"%\")), position=position_dodge(width=0.8), vjust=-0.5) + \n\ttheme(axis.text.x = element_text(angle=90/2, vjust=0.5)) + \n\tscale_y_continuous(labels = function(x) paste0(x, \"%\"))\n  \n## Device OS\nk3 = train %>% group_by(device.operatingSystem) %>% summarise(Count = n())\nk3$Percentage = round((k3$Count/sum(k3$Count))*100,2)\n\nk3 = k3 %>% arrange(desc(Percentage)) %>% head(10)\n\np4 = ggplot(data=k3, aes(x=k3$device.operatingSystem, y=k3$Percentage)) +\n  geom_bar(stat=\"identity\", fill=\"slateblue3\")+\n  theme_minimal() + \n  scale_size_area() + \n  xlab(\"Device OS\") +\n  ylab(\"Count Percentage\") +\n  ggtitle(paste(\"Top 10 Device OS based of Frequency Distribution\")) + \n  geom_text(aes(label=paste0(k3$Percentage,\"%\")), position=position_dodge(width=0.8), vjust=-0.5) + \n  theme(axis.text.x = element_text(angle=90/2, vjust=0.5)) + \n  scale_y_continuous(labels = function(x) paste0(x, \"%\"))\n   \n## Continent\nk4 = train %>% group_by(geoNetwork.continent) %>% summarise(Count = n())\nk4$Percentage = round((k4$Count/sum(k4$Count))*100,2)\n\np5 = ggplot(data=k4, aes(x=k4$geoNetwork.continent, y=k4$Percentage)) +\n  geom_bar(stat=\"identity\", fill=\"darkolivegreen3\")+\n  theme_minimal() + \n  scale_size_area() + \n  xlab(\"Geographic Continents\") +\n  ylab(\"Count Percentage\") +\n  ggtitle(paste(\"Distribution of Geographic Continents\")) + \n  geom_text(aes(label=paste0(k4$Percentage,\"%\")), position=position_dodge(width=0.8), vjust=-0.5) + \n  theme(axis.text.x = element_text(angle=90/2, vjust=0.5)) + \n  scale_y_continuous(labels = function(x) paste0(x, \"%\"))\n   \n## Sub-Continent\nk5 = train %>% group_by(geoNetwork.subContinent) %>% summarise(Count = n())\nk5$Percentage = round((k5$Count/sum(k5$Count))*100,2)\n\nk5 = k5 %>% arrange(desc(Percentage)) %>% head(10)\n\np6 = ggplot(data=k5, aes(x=k5$geoNetwork.subContinent, y=k5$Percentage)) +\n  geom_bar(stat=\"identity\", fill=\"darkolivegreen3\")+\n  theme_minimal() + \n  scale_size_area() + \n  xlab(\"Geographic Sub-Continents\") +\n  ylab(\"Count Percentage\") +\n  ggtitle(paste(\"Top 10 Geographic Sub-Continents based of Frequency Distribution\")) + \n  geom_text(aes(label=paste0(k5$Percentage,\"%\")), position=position_dodge(width=0.8), vjust=-0.5) + \n  theme(axis.text.x = element_text(angle=90/2, vjust=0.5)) + \n  scale_y_continuous(labels = function(x) paste0(x, \"%\"))\n   \n\n## Traffic Source\nk6 = train %>% group_by(trafficSource.source) %>% summarise(Count = n())\nk6$Percentage = round((k6$Count/sum(k6$Count))*100,2)\n\nk6 = k6 %>% arrange(desc(Percentage)) %>% head(10)\n\np7 = ggplot(data=k6, aes(x=k6$trafficSource.source, y=k6$Percentage)) +\n  geom_bar(stat=\"identity\", fill=\"lightsalmon1\")+\n  theme_minimal() + \n  scale_size_area() + \n  xlab(\"Traffic Source\") +\n  ylab(\"Count Percentage\") +\n  ggtitle(paste(\"Top 10 Traffic Source based of Frequency Distribution\")) + \n  geom_text(aes(label=paste0(k6$Percentage,\"%\")), position=position_dodge(width=0.8), vjust=-0.5) + \n  theme(axis.text.x = element_text(angle=90/2, vjust=0.5)) + \n  scale_y_continuous(labels = function(x) paste0(x, \"%\"))\n   \n\n## Traffic Source\nk7 = train %>% group_by(geoNetwork.networkDomain) %>% summarise(Count = n())\nk7$Percentage = round((k7$Count/sum(k7$Count))*100,2)\n\nk7 = k7 %>% arrange(desc(Percentage)) %>% head(10)\n\np8 = ggplot(data=k7, aes(x=k7$geoNetwork.networkDomain, y=k7$Percentage)) +\n  geom_bar(stat=\"identity\", fill=\"lightsalmon1\")+\n  theme_minimal() + \n  scale_size_area() + \n  xlab(\"Geographic Network Domain\") +\n  ylab(\"Count Percentage\") +\n  ggtitle(paste(\"Top 10 Network based of Frequency Distribution\")) + \n  geom_text(aes(label=paste0(k7$Percentage,\"%\")), position=position_dodge(width=0.8), vjust=-0.5) + \n  theme(axis.text.x = element_text(angle=90/2, vjust=0.5)) + \n  scale_y_continuous(labels = function(x) paste0(x, \"%\"))\n\nplot(p)  \nplot(p2)\nplot(p3)\nplot(p4)\nplot(p5)\nplot(p6)\nplot(p7)\nplot(p8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41a66671d4c54de620fd6cbad62dd435af92e26b"},"cell_type":"markdown","source":"Please share your thoughts about this kernal and upvote if you like it. I will post another kernal with a base line model. ** Stay Tuned ! **"}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}