{"cells":[{"metadata":{"_uuid":"a93fd068e6d5559202fc3e3391d7435e6d1668b1","_execution_state":"idle","trusted":false},"cell_type":"markdown","source":"This is a simple Exploratory Data Analysis in order to get a little grip of the variables in dataset.\n\n# Table of Contents \n\n * Prerequisites\n \n1.  Understanding the data\n \n2.  Look for NA's\n \n3.  Plot some data\n\n# Prerequisites \n\nThe first thing we need is get what variables are going to be necessary, for this the first step is to get the variables in the json columns and transform them into separate variables. This might help you:\n\n[Json Extraction](https://www.kaggle.com/mrlong/r-flatten-json-columns-to-make-single-data-frame)\n\nThe second step is to get rid of all the constant variables in both train and test datasets.\n\n\n## 1. Understanding the data \n\n**fullVisitorId**- A unique identifier for each user of the Google Merchandise Store.\n\n**channelGrouping** - The channel via which the user came to the Store.\n\n**date** - The date on which the user visited the Store.\n\n**device** - The specifications for the device used to access the Store.\n\n  * **browser**: The browser used to access the Store.\n\n  * **operatingSystem**: Operating system of the device.\n  \n  * **isMobile**: Whether the access was from a mobile or tablet.\n  \n  * **deviceCategory**: specifies if the device is a Table, a Mobile or a Desktop.\n \n\n**geoNetwork** - This section contains information about the geography of the user.\n\n  * **continent**: Self explanatory\n\n  * **subContinent**: A more specific continent information.\n  \n  * **country**: Self explanatory\n  \n  * **region**: Self explanatory\n  \n  * **metro**: mostly north american cities, includes state.\n\n  * **networkDomain**: Network provider.\n\n**sessionId** - A unique identifier for this visit to the store.\n\n**socialEngagementType** - Engagement type, either \"Socially Engaged\" or \"Not Socially Engaged\".\n\n**totals** - This section contains aggregate values across the session.\n\n  * **hits**: Each loaded item.\n  \n  * **pageviews**: The number of times a page was viewed.\n  \n  * **bounces**: Visits that consist of one page-view.\n\n  * **newVisits**: New visits to the domain\n  \n  * **transactionRevenue**: The target to predict, only in train dataset.\n\n**trafficSource** - This section contains information about the Traffic Source from which the session originated.\n\n  * **campaign**: \n  \n  * **source**: The previous source that directed to the Store.\n  \n  * **medium**: Describes how the user got there, beacause a referral, by their own (organic), ...\n\n  * **keyword**: Keywords used in a search, for example:  \"you tube tshirt\", \"youtuber water bottle\", \"www dot google dot com\" \n  \n  * **adwordsClickInfo**: Extra information provided by AdWords\n  \n  * **isTrueDirect**: If the visitor came directly to the store\n  \n    + page\n    \n    + slot: position of the add, on Top or Right-Hand Side\n    \n    + gclId: Google Click Identifier appended in the URL used when auto-taggin is enabled .\n    \n    + adNetworkType: Origin of the search, can be Google Search or Search partners\n\n**visitId** - An identifier for this session. This is part of the value usually stored as the _utmb cookie. This is only unique to the user. For a completely unique ID, you should use a combination of fullVisitorId and visitId.\n\n**visitNumber** - The session number for this user. If this is the first session, then this is set to 1.\n\n**visitStartTime** - The timestamp (expressed as POSIX time).\n\n\n"},{"metadata":{"_uuid":"251be308a6b18a1efebdbc3c1a6b96c8ac986e03"},"cell_type":"markdown","source":"## 2. Look for NA's \n\nNow let's load the dataset:"},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"87a74074e49e6d9e36881cc482937366909a02f9"},"cell_type":"code","source":"library(data.table)\n\ntrain <- fread('../input/flattrain/flattenTrain.csv')\nhead(train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3041957429ff1c7a618c4bb4ad2f54724fc66b24"},"cell_type":"markdown","source":"Let's check for NA's:"},{"metadata":{"trusted":true,"_uuid":"c7871de91bca0c7cf89db011c9adffbc33e3cbef"},"cell_type":"code","source":"naCount <- apply(train, 2, function(x) sum(is.na(x)))\nnaCount <- data.frame(names = names(naCount), count = naCount)\nprint(naCount)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true,"_uuid":"516ab88a25168473115703b0ac5f59e77ce988b9"},"cell_type":"code","source":"library(ggplot2)\nnaPlot <- ggplot(naCount, aes(names, weight = count, fill = names)) + \n                geom_bar() + \n                coord_flip() +\n                scale_y_continuous(expand = c(0, 0))\noptions(repr.plot.width=20, repr.plot.height=10)\nnaPlot","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b4178344d7ca10e9c9347b089c956c521ba26f17"},"cell_type":"markdown","source":"Now lets see the percentage of NAs:"},{"metadata":{"trusted":true,"_uuid":"5a32d588fbb5b159ca8add15230cb71b6f7a13c1"},"cell_type":"code","source":"naCount$pctNA <- naCount$count*100 / nrow(train)\nas.data.frame(t(naCount[naCount$pctNA > 0, c(2,3)]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6688a84677d6fff0cfa6979b87f382b111751d88"},"cell_type":"markdown","source":"14 variables out of our current 36 have NAs and 12 of them are over 50%.If we take a closer look, our target \"transactionRevenue\" has almost 99% of missing values which means that most of the revenue is generated by a small group. Lets explore the zero transaction value visitors against the non-zero ones."},{"metadata":{"trusted":true,"_uuid":"3be5b674cb31118fb253597c4962d4eaf60cf336"},"cell_type":"code","source":"ind <- train$transactionRevenue > 0 \nind[is.na(ind)] <- FALSE\nnonZeroTrain <- train[ind,]\nzeroTrain <- train[which(ind==FALSE),]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"208bd57333b4d43a8ca7d48798567dfbb8f1847e"},"cell_type":"markdown","source":"Lets compare the NAs between these two groups:"},{"metadata":{"trusted":true,"_uuid":"92a925d81f01930c9142a69eb660148886c86eab"},"cell_type":"code","source":"naZero <- apply(zeroTrain, 2, function(x) sum(is.na(x)))\nnaZero <- data.frame(names = names(naZero), count = naZero)\nnaZero[naZero$count>0,]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21a537ba186fa94ac38b25b2baea482d906fe59e"},"cell_type":"code","source":"naNonZero <- apply(nonZeroTrain, 2, function(x) sum(is.na(x)))\nnaNonZero <- data.frame(names = names(naNonZero), count = naNonZero)\nnaNonZero[naNonZero$count>0,]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f26db3e4b375e34703559177d457ed0f850c7580"},"cell_type":"markdown","source":"We have different number of variables in each set, these accounts for the transactionRevenue and page views which only has 100 NA across the board."},{"metadata":{"trusted":true,"_uuid":"7efedd9ba7f39193baa4a0efda50cbefdf4944ed"},"cell_type":"code","source":"naNonZeroPlot <- ggplot(naNonZero[naNonZero$count>0,], aes(names, weight = count, fill = names)) + \n                geom_bar() + \n                coord_flip() +\n                xlab('Variables') +\n                ylab('NA Count') +\n                ggtitle('Non Zero Revenue Visitors') +\n                scale_y_continuous(expand = c(0, 0))\nnaZeroPlot <- ggplot(naZero[naZero$count>0,], aes(names, weight = count, fill = names)) + \n                geom_bar() + \n                coord_flip() +\n                xlab('Variables') +\n                ylab('NA Count') +\n                ggtitle('Zero Revenue Visitors') +\n                scale_y_continuous(expand = c(0, 0))\n\ngridExtra::grid.arrange(naNonZeroPlot, naZeroPlot, nrow = 2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"250efd9d896b7cb5a09d84cfbf34a5f94f12d548"},"cell_type":"markdown","source":"Most NAs stay consistent between both groups with the exception of bounces which has less NA in the Zero side and isTrueDirect which decreases on the NonZero groups, that might imply that most revenue visitors already knew about the store. Also, the newVisits NAs increase on the the nonZero visitors which means that the visitor has already been in the store.\n\nSo it's more likely that a visitor that has already checked the store and comes directly to it will produce more revenue than first visitors (Duh!)\n\n\n## 3. Plot some data \n\nNow lets check some no NA variables:"},{"metadata":{"trusted":true,"_uuid":"b6cc4a61766641ff251eb03929ddd680e74214a9"},"cell_type":"code","source":"noNaVariablesNonZero <- naNonZero[naNonZero$count == 0, ]$names\nnoNaVariablesZero <- naZero[naZero$count == 0, ]$names\n\ncat('\\n\\nThe no NAs variables for the NonZero revenue Group are: \\n\\n')\nprint(as.character(noNaVariablesNonZero))\ncat('\\n\\nThe no NAs variables for the Zero revenue Group are: \\n\\n')\nprint(as.character(noNaVariablesZero))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"25c1fab665d6fb0d3438dd44fa552ac52005838a"},"cell_type":"code","source":"channelZero <- ggplot(zeroTrain, aes(x = channelGrouping, fill = channelGrouping)) +\n                geom_bar() +\n                coord_flip()+\n                ggtitle('channelGrouping for Zero Revenue Visitors')\nchannelNonZero <- ggplot(nonZeroTrain, aes(x = channelGrouping, fill = channelGrouping)) +\n                geom_bar() +\n                coord_flip() +\n                ggtitle('channelGrouping for Non-Zero Revenue Visitors')\nbrowserlZero <- ggplot(zeroTrain, aes(x = browser, fill = browser)) +\n                geom_bar() +\n                coord_flip()+\n                ggtitle('browser for Zero Revenue Visitors')\nbrowserlNonZero <- ggplot(nonZeroTrain, aes(x = browser, fill = browser)) +\n                geom_bar() +\n                coord_flip() +\n                ggtitle('browser for Non-Zero Revenue Visitors')\noperatingSystemZero <- ggplot(zeroTrain, aes(x = operatingSystem, fill = operatingSystem)) +\n                geom_bar() +\n                coord_flip()+\n                ggtitle('operatingSystem for Zero Revenue Visitors')\noperatingSystemNonZero <- ggplot(nonZeroTrain, aes(x = operatingSystem, fill = operatingSystem)) +\n                geom_bar() +\n                coord_flip() +\n                ggtitle('operatingSystem for Non-Zero Revenue isMobile')\nisMobileZero <- ggplot(zeroTrain, aes(x = isMobile, fill = channelGrouping)) +\n                geom_bar() +\n                coord_flip()+\n                ggtitle('isMobile for Zero Revenue Visitors')\nisMobileNonZero <- ggplot(nonZeroTrain, aes(x = isMobile, fill = isMobile)) +\n                geom_bar() +\n                coord_flip() +\n                ggtitle('isMobile for Non-Zero Revenue Visitors')\ndeviceCategoryZero <- ggplot(zeroTrain, aes(x = deviceCategory, fill = deviceCategory)) +\n                geom_bar() +\n                coord_flip()+\n                ggtitle('deviceCategory for Zero Revenue Visitors')\ndeviceCategoryNonZero <- ggplot(nonZeroTrain, aes(x = deviceCategory, fill = deviceCategory)) +\n                geom_bar() +\n                coord_flip() +\n                ggtitle('deviceCategory for Non-Zero Revenue Visitors')\nsubContinentZero <- ggplot(zeroTrain, aes(x = subContinent, fill = subContinent)) +\n                geom_bar() +\n                coord_flip()+\n                ggtitle('subContinent for Zero Revenue Visitors')\nsubContinentNonZero <- ggplot(nonZeroTrain, aes(x = subContinent, fill = subContinent)) +\n                geom_bar() +\n                coord_flip() +\n                ggtitle('subContinent for Non-Zero Revenue Visitors')\noptions(repr.plot.width=20, repr.plot.height=30)\ngridExtra::grid.arrange(channelZero, channelNonZero,\n                        operatingSystemZero, operatingSystemNonZero,\n                        isMobileZero, isMobileNonZero,\n                        deviceCategoryZero, deviceCategoryNonZero,\n                        subContinentZero, subContinentNonZero,\n                        browserlNonZero,\n                        nrow = 6)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1f394d585c9f6fa9747cdceaa139751079fe636"},"cell_type":"markdown","source":"A brief summary of the above:\n\n* Most revenue visitors come from referrals and less by social channels.\n* While the most used operating system is Windows, Mac users are more likely to produce revenue.\n* Most revenue comes from purchases in desktop devices.\n* North Americans are the primary source of revenue... by far.\n* Chrome and Safari are the primary browsers used by the Non Zero revenue visitors\n\n\n\n*More to come...*"}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}