{"cells":[{"metadata":{"_uuid":"f2ffb85d74a794cb6a72a217e911b3da910a6316","trusted":true},"cell_type":"markdown","source":"**Introduction:**\n\nI am just curous to analyze. Successfull data Science projects should be capable of adding business value. I feel this dataset will give and develop my mindset in understanding the usefulness of Data Science from a marketing point of view. I am not a marketting guy but I know a few fundamentals. Hopping to understand whats behind this huge dataset. I have used the preprocessed dataset which Juian has published. Here is the link to it. https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields\n\n"},{"metadata":{"_uuid":"8003731db53dfd65d5829716d72869aea8919065"},"cell_type":"markdown","source":"**Loading Library**"},{"metadata":{"_uuid":"1cc335dfc533017c3ceecebf5fddfefbc297bdfa","_execution_state":"idle","trusted":true,"_kg_hide-output":true,"_kg_hide-input":false},"cell_type":"code","source":"library(tidyverse) # metapackage with lots of helpful functions\nlist.files(path = \"../input/1-quick-start-read-csv-and-flatten-json-fields\")\nlibrary(base)\nlibrary(data.table)\nlibrary(ggplot2)\nlibrary(magrittr)\nlibrary(gridExtra)\nlibrary(plotly)\nlibrary(IRdisplay)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"09f13334768080b9cf33e01913e7dc02f16b3f58"},"cell_type":"markdown","source":"**What do we have in the dataset?**\n\nEvery instance in the dataset represent an users activity. So it has the userid, sessionid, the channel, his date of visit,  the visit start time, information about the device, the geographical location information. We also have the totals like the total \n\n"},{"metadata":{"trusted":true,"_uuid":"e5935d7a705752a86463952a30ce32037b4770ff"},"cell_type":"code","source":"train_data<-  fread(\"../input/1-quick-start-read-csv-and-flatten-json-fields/train-flattened.csv\")\ntest_data<-  fread(\"../input/1-quick-start-read-csv-and-flatten-json-fields/test-flattened.csv\",nrows=50000)\ntrain_data[,date:=as.character(date)]\ntrain_data[,date:=as.Date(date,\"%Y%m%d\" )]\ntrain_data$totals.transactionRevenue[is.na(train_data$totals.transactionRevenue)] = 0\ntrain_data$totals.transactionRevenue<-log(train_data$totals.transactionRevenue+1)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"508f1e0a1ab6c4e7d39be2a20e40b9c2bf69d592"},"cell_type":"markdown","source":"**Repeat users**\n\nOnly 13% of the users move from visits for the second. Almost 87% of the users visits just once. \nAfter their second visit we can expect to retain more than 40% to third and it goes on. The Table and the graph gives the numbers and the retention percentages."},{"metadata":{"trusted":true,"_uuid":"88831b0c5fafd4fc6922b87288311716e30ae346"},"cell_type":"code","source":"train_data%>%\ngroup_by(visitNumber)%>%\nsummarize(count=n())%>%\nggplot(aes(x=visitNumber, y=count, group=1))+geom_line(colour=\"blue\")+\ngeom_point()+\nlabs(x=\"Visit Number\", y=\"count of users\", title=\"Users count by visit number\")+\ntheme_minimal()\n\ntrain_data%>%\ngroup_by(visitNumber)%>%\nsummarize(Current_visit=n())%>%filter(visitNumber<=10)%>%\nmutate( Next_visit= lead(Current_visit, 1), next_visit_= lead(visitNumber, 1))%>%\nmutate(Visit_to_Visit= paste(\"Visit\", visitNumber, \"to\", next_visit_))%>%\nmutate( retention= round( Next_visit/Current_visit,2))%>%\nselect(Visit_to_Visit,Current_visit,Next_visit, retention)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36c652153dbcc3f8ac6ec7acd4e0e267d9b8e5e8"},"cell_type":"markdown","source":"**How do the customer's behaviour changes over time?**\n\nWe can do a simple time based cohort analysis and track their behaviour over time.\n\nLets take one cohort of customers\n\n"},{"metadata":{"trusted":true,"_uuid":"e210ff4fb0ae25c0ad36400c5f26805dda555f36"},"cell_type":"code","source":"print(paste(\"The date range is between\", min(train_data$date), \"and\", max(train_data$date)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fa6a29e3b7e4665c6e0a050661c56dc91b341247"},"cell_type":"code","source":"train_data$purchased<-NA\ntrain_data$purchased<-ifelse(train_data$totals.transactionRevenue>0, \"Purchased\", \"Not_Purchased\")\ntable(train_data$purchased)\n\n#names(train_data)\ntrain_data<-train_data%>%arrange(visitNumber)\ntrain_data<-data.table(train_data)\ntrain_data[  , purchase_no:= 1:.N, by=list(purchased,fullVisitorId )]\n\ntrain_data$no_of_visits_before <-NA\ntrain_data$no_of_visits_before[ train_data$purchased==\"Purchased\"]<-train_data$purchase_no[ train_data$purchased==\"Purchased\"]\n\ntrain_data[  , min_date:= min(date), by=list(fullVisitorId )]\ntrain_data[  , max_date:= max(date), by=list(fullVisitorId )]\n\ntrain_data[  , total_days:=max_date-min_date ]\ntrain_data[  , no_of_sessions:= max(visitNumber) , by=list(fullVisitorId)]\ntrain_data[  , Total_revenue:= sum(totals.transactionRevenue) , by=list(fullVisitorId)]\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d0f6af7344659033c51c8dadb7d794e424f6276f"},"cell_type":"markdown","source":"**** Kmeans Clustering ****\n\nI wanted to know if users can be clustered based on the the pageviews, total visits and total revenue. And so we can identify the channel/device which brings such users.\nAnd also the 80:20 rule can be verified.\n"},{"metadata":{"trusted":true,"_uuid":"3cf989aa89afc3dadbe70dc2057aca9b8cfbee28"},"cell_type":"code","source":"df<-train_data%>%\ngroup_by(fullVisitorId)%>%\nsummarize(total_Views_per_user=sum(totals.pageviews), total_rev_per_user=sum(totals.transactionRevenue), total_visits_per_user= max(visitNumber))\n\ndf[is.na(df)]<-0\n\nmodel<- kmeans(df[,-1], centers = 3, nstart = 25)\nmodel$centers\ndf$cluster<-as.factor( model$cluster)\n\ndf%>%ggplot(aes(x= cluster, y=total_rev_per_user))+geom_boxplot(fill=\"light blue\", colour=\"black\" )+\nlabs(x=\"Cluster no\", y= \"Total_rev_user\", title =\"Clustering based on pageviews, totalvisits,total_revenue \")+\ntheme_minimal()\n\ng1<- df%>%group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%\nggplot(aes(x=cluster, y=revenue, fill=cluster))+geom_bar(stat=\"identity\")+labs(x=\"Cluster\", y=\"Revenue\" ,\"Total Revenue in clusters\")+coord_flip()+theme_minimal()\n\ng2<-df%>%group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%\nggplot(aes(x=cluster, y=no_of_users, fill=cluster))+geom_bar(stat=\"identity\")+labs(x=\"Cluster\", y=\"Total USers in Clusters\")+coord_flip()+theme_minimal()\n\ngrid.arrange(g1,g2, ncol=2)\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f02b56bc0a972b0014101a20cca4e23bf544e9fc"},"cell_type":"markdown","source":"**80:20 Principle**\n\nAlmost 75% of the revenue is generated through less than 3 % of the users.\n"},{"metadata":{"trusted":true,"_uuid":"ffa099940a02cd3850132a7e513c068e149499c5","_kg_hide-input":false},"cell_type":"code","source":"colors <- c('rgb(211,94,96)', 'rgb(128,133,133)', 'rgb(144,103,167)')\n\ng1<- df%>%group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=c(1,2,3)))%>%\nplot_ly( labels = ~cluster, values = ~revenue,  marker = list(colors = colors)) %>% add_pie(hole = 0.6) %>%\n  layout(title = 'Total revenues by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\n\ng2<- df%>%group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=cluster))%>%\nplot_ly( labels = ~cluster, values = ~no_of_users,   marker = list(colors = colors)) %>% add_pie(hole = 0.6)%>%\n  layout(title = 'Total no of users by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\nhtmlwidgets::saveWidget(g1, \"d.html\")\ndisplay_html('<iframe src=\"d.html\" width=100% height=450></iframe>')\n\nhtmlwidgets::saveWidget(g2, \"d1.html\")\ndisplay_html('<iframe src=\"d1.html\" width=100% height=450></iframe>')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3c7ee6dbaf0996536f7ddaf90f4d03216db6c9d"},"cell_type":"markdown","source":"**American users -- Cluster view**"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4f3d9cf70bdc712ef8162913865ad7ca0dd1a896"},"cell_type":"code","source":"colors <- c('rgb(211,94,96)', 'rgb(128,133,133)', 'rgb(144,103,167)')\ng1<- df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Americas\")%>% group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=c(1,2,3)))%>%\nplot_ly( labels = ~cluster, values = ~revenue,  marker = list(colors = colors)) %>% add_pie(hole = 0.6) %>%\n  layout(title = 'Americas Total revenues by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\n\ng2<-df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Americas\")%>%group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=cluster))%>%\nplot_ly( labels = ~cluster, values = ~no_of_users,   marker = list(colors = colors)) %>% add_pie(hole = 0.6)%>%\n  layout(title = ' Americas  Total no of users by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\nhtmlwidgets::saveWidget(g1, \"d2.html\")\ndisplay_html('<iframe src=\"d2.html\" width=100% height=450></iframe>')\n\nhtmlwidgets::saveWidget(g2, \"d3.html\")\ndisplay_html('<iframe src=\"d3.html\" width=100% height=450></iframe>')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9fe22e865f1970a594141b42a5b304224f34f6af"},"cell_type":"markdown","source":"**African users cluster view**"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"688da82a7f74500862fe42076c91d100111a8df5"},"cell_type":"code","source":"colors <- c('rgb(211,94,96)', 'rgb(128,133,133)', 'rgb(144,103,167)')\n\ng1<- df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Africa\")%>% group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=c(1,2,3)))%>%\nplot_ly( labels = ~cluster, values = ~revenue,  marker = list(colors = colors)) %>% add_pie(hole = 0.6) %>%\n  layout(title = 'Africas Total revenues by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\n\ng2<-df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Africa\")%>%group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=cluster))%>%\nplot_ly( labels = ~cluster, values = ~no_of_users,   marker = list(colors = colors)) %>% add_pie(hole = 0.6)%>%\n  layout(title = ' Africas  Total no of users by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\nhtmlwidgets::saveWidget(g1, \"d4.html\")\ndisplay_html('<iframe src=\"d4.html\" width=100% height=450></iframe>')\n\nhtmlwidgets::saveWidget(g2, \"d5.html\")\ndisplay_html('<iframe src=\"d5.html\" width=100% height=450></iframe>')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4800e96779d0bc0233b70b2862e46fc70b7f694"},"cell_type":"markdown","source":"**Asian Users Custer view**"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"11273dd589cc1efcc2c4090921cf3e458f9cd68d"},"cell_type":"code","source":"colors <- c('rgb(211,94,96)', 'rgb(128,133,133)', 'rgb(144,103,167)')\n\ng1<- df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Asia\")%>% group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=c(1,2,3)))%>%\nplot_ly( labels = ~cluster, values = ~revenue,  marker = list(colors = colors)) %>% add_pie(hole = 0.6) %>%\n  layout(title = 'Asia Total revenues by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\n\ng2<-df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Asia\")%>%group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=cluster))%>%\nplot_ly( labels = ~cluster, values = ~no_of_users,   marker = list(colors = colors)) %>% add_pie(hole = 0.6)%>%\n  layout(title = ' Asia  Total no of users by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\nhtmlwidgets::saveWidget(g1, \"d6.html\")\ndisplay_html('<iframe src=\"d6.html\" width=100% height=450></iframe>')\n\nhtmlwidgets::saveWidget(g2, \"d7.html\")\ndisplay_html('<iframe src=\"d7.html\" width=100% height=450></iframe>')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7d4b1e873c86b15a4e636a3b858ef57008c63f3"},"cell_type":"markdown","source":"**European users cluster view**"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"74c9308787e20edf6a34d6258c9c1e16b4f599ae"},"cell_type":"code","source":"colors <- c('rgb(211,94,96)', 'rgb(128,133,133)', 'rgb(144,103,167)')\n\ng1<- df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Europe\")%>% group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=c(1,2,3)))%>%\nplot_ly( labels = ~cluster, values = ~revenue,  marker = list(colors = colors)) %>% add_pie(hole = 0.6) %>%\n  layout(title = 'Europe Total revenues by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\n\ng2<-df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Europe\")%>%group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=cluster))%>%\nplot_ly( labels = ~cluster, values = ~no_of_users,   marker = list(colors = colors)) %>% add_pie(hole = 0.6)%>%\n  layout(title = ' Europe Total no of users by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\nhtmlwidgets::saveWidget(g1, \"d8.html\")\ndisplay_html('<iframe src=\"d8.html\" width=100% height=450></iframe>')\n\nhtmlwidgets::saveWidget(g2, \"d9.html\")\ndisplay_html('<iframe src=\"d9.html\" width=100% height=450></iframe>')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f2f8b1c9a127bf0f9049c94736e5bafb10cf8cc8"},"cell_type":"code","source":"colors <- c('rgb(211,94,96)', 'rgb(128,133,133)', 'rgb(144,103,167)')\n\ng1<- df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Oceania\")%>% group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=c(1,2,3)))%>%\nplot_ly( labels = ~cluster, values = ~revenue,  marker = list(colors = colors)) %>% add_pie(hole = 0.6) %>%\n  layout(title = 'Oceania Total revenues by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\n\ng2<-df%>%left_join(train_data, by='fullVisitorId') %>%filter(geoNetwork.continent==\"Oceania\")%>%group_by(cluster)%>%summarise(revenue= sum(total_rev_per_user), no_of_users=n())%>%mutate(cluster=factor(cluster, levels=cluster))%>%\nplot_ly( labels = ~cluster, values = ~no_of_users,   marker = list(colors = colors)) %>% add_pie(hole = 0.6)%>%\n  layout(title = ' Oceania Total no of users by cluster',\n         xaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE),\n         yaxis = list(showgrid = FALSE, zeroline = FALSE, showticklabels = FALSE))\n\n\nhtmlwidgets::saveWidget(g1, \"d10.html\")\ndisplay_html('<iframe src=\"d10.html\" width=100% height=450></iframe>')\n\nhtmlwidgets::saveWidget(g2, \"d11.html\")\ndisplay_html('<iframe src=\"d12.html\" width=100% height=450></iframe>')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7eaf08059f75cd7741994f14b6810879f2214c2f"},"cell_type":"code","source":"library(maps)\nmap.world <- map_data(\"world\")\nhead(map.world)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d0c180b3ff5e4e30baff4366a1458528c97faeb9"},"cell_type":"code","source":"train_data<- train_data%>%left_join(df, by='fullVisitorId')\n#names(train_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2a9870e6df791610a2002045e7b4f9535f941bfd","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"data<- train_data%>%filter(date<=\"2017-03-01\")\nfwrite(data, \"data.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35b4a43192ea822123df6e191750e64e211ca439"},"cell_type":"code","source":"train_data%>%ggplot(aes(x=cluster, fill=device.isMobile))+\ngeom_bar(stat=\"count\",position=\"fill\" )+\ntheme_minimal()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"45138cc6f845c0e9775bbb4a7dea61092eed5a77"},"cell_type":"code","source":"top_10<- train_data%>% group_by(trafficSource.source )%>%summarise(count=n())%>%arrange(count)%>%tail(10)%$%trafficSource.source\n\ntrain_data%>%filter(trafficSource.source%in%top_10)%>%  mutate(trafficSource.source=factor(trafficSource.source, levels=top_10))%>%\nggplot(aes( x=trafficSource.source))+geom_bar(stat=\"count\" , fill=\"light blue\", colour=\"black\")+\nfacet_wrap(~cluster)+\ncoord_flip()+\ntheme_minimal()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9aa312e44e9111708875a9cbdc8a240347f10261"},"cell_type":"markdown","source":"**Pageviews of Purchaser and non Purchaser**"},{"metadata":{"trusted":true,"_uuid":"15dfb455bda2dd76e62b160d1aad56f20efb77e5"},"cell_type":"code","source":"train_data%>%ggplot(aes(y=totals.pageviews, x=purchased))+\ngeom_boxplot(fill=\"light blue\", colour=\"black\")+\nlabs(x=\"Purchased/Not Purchased\", y=\"Page views\", title=\"\")+\ntheme_minimal()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e880f867aff2c0137f295f6f3f57ed1620cebe70"},"cell_type":"code","source":"train_data%>%ggplot(aes(y=totals.transactionRevenue, x=totals.pageviews))+\ngeom_point(col=\"blue\")+\ngeom_smooth(method = \"lm\", se=TRUE, color=\"black\", aes(group=1)) +\ntheme_minimal()+\nlabs(x = \"Total Page Views\", y = \"Total Revenues\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0c794494625c100e3ff1c40d85131f754c3a5f8"},"cell_type":"code","source":"train_data%>%ggplot(aes(y=totals.transactionRevenue, x=no_of_sessions))+\ngeom_point(col=\"blue\")+\ngeom_smooth(method = \"lm\", se=TRUE, color=\"black\", aes(group=1)) +\ntheme_minimal()+\nlabs(x = \"Sessions per user\", y = \"Total Revenues\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4f9eea5b3d799f4b9c6b11b4fca515e3a32e8ea"},"cell_type":"code","source":"train_data%>%\ngroup_by(geoNetwork.country) %>%\nsummarise(count=n())%>%\narrange(count)%>%\ntail(20)%>%\nmutate(geoNetwork.country=factor(geoNetwork.country, levels=geoNetwork.country))%>%\nggplot(aes(x=geoNetwork.country,y=count ))+\ngeom_bar(stat=\"identity\",colour=\"black\", fill=\"light blue\")+\ntheme_minimal()+\nlabs(titile=\"Visits count by country\")+\ncoord_flip()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0eade88c3f5bb127355ec6cf26c0d97afde0d0f4"},"cell_type":"code","source":"train_data%>%\ngroup_by(geoNetwork.country) %>%\nsummarise(count=sum(totals.transactionRevenue))%>%\narrange(count)%>%\ntail(20)%>%\nmutate(geoNetwork.country=factor(geoNetwork.country, levels=geoNetwork.country))%>%\nggplot(aes(x=geoNetwork.country,y=count ))+\ngeom_bar(stat=\"identity\",colour=\"black\", fill=\"light blue\")+\ntheme_minimal()+\nlabs(titile=\"Total revenues by country\")+\ncoord_flip()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8958954740935dbd551f2d91b718e34c604e7cff"},"cell_type":"code","source":"a1<- train_data%>%filter(purchased==\"Purchased\")%>%\ngroup_by(geoNetwork.country) %>%\nsummarise(purchased=n(), Rev=sum(totals.transactionRevenue))\n\nb1<- train_data%>%filter(purchased!=\"Purchased\")%>%\ngroup_by(geoNetwork.country) %>%\nsummarise(not_purchased=n())%>%\nleft_join(a1, by='geoNetwork.country')\n\nb1[is.na(b1)]<-0\n\nb1<-  b1%>%mutate(total= purchased+not_purchased) %>%  mutate( Purchase_rate=purchased/(purchased+not_purchased))\n\nb1%>%arrange(desc(Purchase_rate))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"007b7592a22577781e7a5b179fde1ce273b4c3c6"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4d77f6d01d82538d5798934b9f24c2b4eb07b91b"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ae3b13f9ffb5eea7fc4ef6ac5522cb4e2b9f0e0f"},"cell_type":"code","source":" head(train_data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30592703e14e710cab4a43780ef6a384b380a3e7"},"cell_type":"markdown","source":"Lets filter and  analyze the user behaviour for Sep 2016 cohort."},{"metadata":{"trusted":true,"_uuid":"c50724be88e87d4ce021b1230e4f2ef6a4e21f8f"},"cell_type":"code","source":"coh$purchased<-NULL\ncoh$purchased<-ifelse(coh$totals.transactionRevenue>0, \"Purchased\", \"Not_Purchased\")\n\nSep_cohort<- train_data%>%\nfilter(date<=\"2016-09-30\" & date>=\"2016-09-01\" )%>%\nfilter(visitNumber==1)%>%select(fullVisitorId)\nSep_cohort<-Sep_cohort$fullVisitorId\ncoh<-train_data%>%filter(fullVisitorId%in%Sep_cohort)\ncoh <-(coh [!duplicated(coh[c(\"fullVisitorId\",\"sessionId\")]),])\n\ncoh%>%\ngroup_by(trafficSource.referralPath,purchased)%>%\nsummarize(count=n())%>%\narrange(desc(trafficSource.referralPath))%>%\nspread(purchased,count, fill=0 )%>%\nmutate(conversion=Purchased/Not_Purchased)%>%\narrange(desc(conversion))\n\ncoh%>%ggplot(aes(x=visitNumber,fill=purchased ))+geom_bar(stat=\"count\")\n\ncoh%>%filter(visitNumber>4)%>% ggplot(aes(x=visitNumber,fill=purchased ))+geom_bar(stat=\"count\")+facet_wrap(~trafficSource.source)\ntable(coh$trafficSource.source)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0151c15b40724173f38156e0b6ceb6e38ce4845e"},"cell_type":"code","source":"coh%>%group_by(visitNumber,trafficSource.medium )%>% summarize(count=n())","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"cf6399ec5c931ebf5def97d4567aff47fe8e96a6"},"cell_type":"code","source":"print(paste(\"The date range is between\", min(coh$date), \"and\", max(coh$date)))\ncoh%>%filter(date<\"2016-09-01\")%$%fullVisitorId\ncoh%>%filter(fullVisitorId==9847307838447511552)\ncoh%>%filter(fullVisitorId==790023959302630400)\n\ncoh%>%group_by(fullVisitorId, visitNumber)%>%summarise(count=n())%>%arrange(desc(fullVisitorId))%>%filter(count>1)\n\ncoh%>%group_by(fullVisitorId, sessionId)%>%summarise(count=n())%>%arrange(desc(fullVisitorId))%>%filter(count>1)\n\nsess<- coh%>%group_by(visitNumber, sessionId)%>%summarise(count=n())%>%arrange(desc(sessionId))%>%filter(count>1)%$%sessionId\ncoh%>%filter(sessionId%in%sess)%>%arrange(fullVisitorId)%>%select(totals.transactionRevenue)\ncoh$totals.transactionRevenue\nhead(coh)\nlength(sess)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5b83ef0737b4354a285a976d3435485161444fc"},"cell_type":"markdown","source":"**Trend of visits over time  and Bounce rate**"},{"metadata":{"trusted":true,"_uuid":"8d5867a7c1965986b1fd38ab2c72160808bfa043"},"cell_type":"code","source":"paste(\"The date range is between\", min(train_data$date), \"and\", max(train_data$date))\ntrain_data$totals.bounces[is.na(train_data$totals.bounces)]<-0\ntrain_data$totals.bounces<-factor(train_data$totals.bounces, levels=c(1,0))\ntrain_data%>%\ngroup_by(date, totals.bounces)%>%\nsummarise(count=n())%>%\nggplot(aes(x=date,y=count, group=factor( totals.bounces), colour=factor( totals.bounces)))+\ngeom_line(stat=\"identity\" )+\nscale_color_manual(labels = c(\"Bounced: Yes \", \"Bounced: No\"), values = c(\"blue\", \"red\"))+\nlabs(title=\"User aquisition over time\")+\ntheme_minimal()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98cfe6cecc0b5db2b086545b699dc7e695915cb6"},"cell_type":"markdown","source":"**Bounce rate by channels**"},{"metadata":{"trusted":true,"_uuid":"ea69f7a59fbc21681f96b40384d1a0fd84a3aeb4"},"cell_type":"code","source":"paste(\"The date range is between\", min(train_data$date), \"and\", max(train_data$date))\ntrain_data$totals.bounces[is.na(train_data$totals.bounces)]<-0\n\ntrain_data%>%\ngroup_by(date, totals.bounces)%>%\nsummarise(count=n())%>%\nggplot(aes(x=date,y=count, group=factor( totals.bounces), colour= totals.bounces))+\ngeom_line(stat=\"identity\" )+\nlabs(title=\"User aquisition over time\")+\nscale_color_manual(labels = c(\"Bounced: Yes \", \"Bounced: No\"), values = c(\"blue\", \"red\"))+\ntheme_minimal()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"83ef107db1f5f7f410af7a53d58838f899f804d8"},"cell_type":"markdown","source":"> **Channel: Referral  --- Bounce rate**"},{"metadata":{"trusted":true,"_uuid":"ecc9224f02b3cfae11e08606b3ce918db1be6828"},"cell_type":"code","source":"train_data%>%filter(channelGrouping==\"Referral\")%>%\ngroup_by(date, totals.bounces)%>%\nsummarise(count=n())%>%\nggplot(aes(x=date,y=count, group=factor( totals.bounces), colour= totals.bounces))+\ngeom_line(stat=\"identity\" )+\nlabs(title=\" Channel: Referral, User aquisition over time\")+\nscale_color_manual(labels = c(\"Bounced: Yes \", \"Bounced: No\"), values = c(\"blue\", \"red\"))+\ntheme_minimal()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"111e6bc670237e011aa66501a547bf5df12a093a"},"cell_type":"markdown","source":"**Channel: Social ---Bounce rate**"},{"metadata":{"trusted":true,"_uuid":"a100733c2ba09235b85fc452958f4c2bd78d2231"},"cell_type":"code","source":"train_data%>%filter(channelGrouping==\"Social\")%>%\ngroup_by(date, totals.bounces)%>%\nsummarise(count=n())%>%\nggplot(aes(x=date,y=count, group=factor( totals.bounces), colour= totals.bounces))+\ngeom_line(stat=\"identity\" )+\nlabs(title=\"Channel: Social,  User aquisition over time\")+\nscale_color_manual(labels = c(\"Bounced: Yes \", \"Bounced: No\"), values = c(\"blue\", \"red\"))+\ntheme_minimal()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38b5ae0be553ad231075d2fe74a7e01982be02c5"},"cell_type":"markdown","source":"**Channel: Others ---Bounce rate**"},{"metadata":{"trusted":true,"_uuid":"1be84e2ecdfc764bbf6440158b6a58d00f50041a"},"cell_type":"code","source":"train_data%>%filter(channelGrouping!=\"Social\"|channelGrouping!=\"Referral\")%>%\ngroup_by(date, totals.bounces)%>%\nsummarise(count=n())%>%\nggplot(aes(x=date,y=count, group=factor( totals.bounces), colour= totals.bounces))+\ngeom_line(stat=\"identity\" )+\nscale_color_manual(labels = c(\"Bounced: Yes \", \"Bounced: No\"), values = c(\"blue\", \"red\"))+\nlabs(title=\"User aquisition over time\")+\ntheme_minimal()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dbe0bb1b170e486abaf91fa2584a5f6e5299e71a"},"cell_type":"markdown","source":"**Social media is the way to acquire new customers but it is not revenue generating channell **\nUsers who visits the website through referrals make more transcations with us.\n\n"},{"metadata":{"trusted":true,"_uuid":"949b263a86edd74beaa5ecca8a9c9275c287db32"},"cell_type":"code","source":"\ntrain_data%>%\ngroup_by(visitNumber, channelGrouping)%>%\nsummarize(count=n())%>%\nfilter(visitNumber<4)%>%\nggplot(aes( y= count, x= channelGrouping, fill=channelGrouping))+\ngeom_bar(stat=\"identity\")+\nfacet_wrap(~visitNumber)+\ncoord_flip()+\nlabs(title=\"Bar chart: Visit number 1 to 3 and the Channel groupings\", y= \"count of users\", x=\"Visit Number\" )+\ntheme_minimal()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c56ddcd3c01dde82932c6cde23a8c2fb9a965175"},"cell_type":"code","source":"train_data%>%\ngroup_by(visitNumber, channelGrouping)%>%\nsummarize(count=sum(totals.transactionRevenue), no=n() )%>% \nmutate(average_Revenue_per_user=count/no)%>%\nfilter(visitNumber<4)%>%\nggplot(aes( y= count, x= channelGrouping, fill=channelGrouping))+\ngeom_bar(stat=\"identity\")+\nfacet_wrap(~visitNumber)+\ncoord_flip()+\nlabs(title=\"Total Revenue Earned : Visit number 1 to 10 and the Channel groupings\", y= \"Revenue Generated\", x=\"Visit Number\" )+\ntheme_minimal()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"846f954c3167705f4506239aae84b87a784c8522"},"cell_type":"code","source":"train_data%>%\ngroup_by(visitNumber, channelGrouping)%>%\nsummarize(count=sum(totals.transactionRevenue), no=n() )%>% \nmutate(average_Revenue_per_user=count/no)%>%\nfilter(visitNumber<4)%>%\nggplot(aes( y= average_Revenue_per_user, x= channelGrouping, fill=channelGrouping))+\ngeom_bar(stat=\"identity\")+\nfacet_wrap(~visitNumber)+\ncoord_flip()+\nlabs(title=\"Average Revenue Earned per user : Visit number 1 to 3 and the Channel groupings\", y= \"Revenue Generated\", x=\"Visit Number\" )+\ntheme_minimal()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2fba1c52f3177730754867494a639dd042d044dd"},"cell_type":"code","source":"\n\ntrain_data%>%\ngroup_by(visitNumber, channelGrouping)%>%\nsummarize(count=n())%>%\nfilter(visitNumber<10)%>%\nggplot(aes(x=factor( visitNumber), y= count, fill= channelGrouping))+\ngeom_bar(stat=\"identity\", position = \"fill\")+\nlabs(title=\"Bar chart: Visit number 1 to 10 and the Channel groupings\", y= \"Proportion of user\", x=\"Visit Number\" )+\ntheme_minimal()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66dfa689a5941ac47e609c0e426292c9888d824a"},"cell_type":"markdown","source":"**Missing data check**"},{"metadata":{"trusted":true,"_uuid":"9380d72a0e49ad3271a0432548ce128527721258"},"cell_type":"code","source":"missing_values<-data.frame( lapply(train_data, function (x) sum(is.na(x))/nrow(train_data) *100) )\nmissing_values <- gather(missing_values, key=\"Feature\", value=\"missing\")\nmissing_values%>%arrange(desc(missing))%>%filter(missing>0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6b1850a5eeee8e3681915d030191cb197432f341"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e186d8fccc439d6b2456b50d1b755799d8d8afa9"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a03ea1c6ac7104f6671cc2cb7d18ac480b46074"},"cell_type":"markdown","source":"**Training set **\n\nEvery users can have multiple sessions."},{"metadata":{"trusted":true,"_uuid":"ba203812b3d84a78ed4f4760599cc691a75cb331"},"cell_type":"code","source":"print(paste(\"The total no of rows\", nrow(train_data))) \nprint(paste(\"The total no of unique visitor id\", length (unique(train_data$fullVisitorId))))\nprint(paste(\"The total no of unique session id\", length (unique(train_data$sessionId))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14f8d01abc21f6101d3ba1ffa41686ab846997a0"},"cell_type":"markdown","source":"**Analyzing users with multiple sessions:** (training set)"},{"metadata":{"trusted":true,"_uuid":"b064476380998d3427d7a9bdd828d3e1511a4d2c"},"cell_type":"code","source":"train_data%>%group_by(fullVisitorId)%>%\nsummarise(count=n())%>%\narrange(desc(count))%>%\nggplot(aes(x=count))+geom_histogram(fill=\"light blue\", color=\"black\")%>%\nlabs(title=\"Distribution of No of Sessions/ user\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98d07d2560b0953028f0b49116d946f63269a9a1"},"cell_type":"code","source":"\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8fe2c7a31769181bfc17d987adf8c712effe75a"},"cell_type":"markdown","source":"**Analyzing users with multiple sessions:** (test set)"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"ba61a295a4be825b76950527fb461491157144fc"},"cell_type":"code","source":"test_data%>%\ngroup_by(fullVisitorId)%>%\nsummarise(count=n())%>%\nggplot(aes(x=count))+geom_histogram(fill=\"light blue\", color=\"black\")+\nlabs(title=\"Distribution of No of Sessions/ user\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61dcb94016b72bd2225b2ab45f98b0edf67b3eb6"},"cell_type":"markdown","source":"**Overall average revenue per user :**\n\nRevenue per user is "},{"metadata":{"trusted":true,"_uuid":"e05bc6c1cdd6490f289cc903465094d9234e7ef6"},"cell_type":"code","source":"print (paste(\"The revenue per user is \",  sum(train_data$totals.transactionRevenue)/length (unique(train_data$fullVisitorId))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"465c5e6a4c1a71d0c5544332eb35105ab3bfb3ed"},"cell_type":"markdown","source":"**Test set**"},{"metadata":{"trusted":true,"_uuid":"afc7ffa6ef9f03783610be0d0896cec35ef36fef"},"cell_type":"code","source":"print(paste(\"The total no of rows\", nrow(test_data))) \nprint(paste(\"The total no of unique full visitor id\", length (unique(test_data$fullVisitorId))))\nprint(paste(\"The total no of unique visitor id\", length (unique(test_data$visitId))))\nprint(paste(\"The total no of unique session id\", length (unique(test_data$sessionId))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15e4ab0669b72662cca1dee3d27d2c1d0b3ba60e","scrolled":true},"cell_type":"code","source":"train_data%>%ggplot(aes(x=totals.transactionRevenue))+geom_histogram(bins = 1000000)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}