{"cells":[{"metadata":{"_uuid":"f4ffd675415135b3de6254306057cae303a1bf32"},"cell_type":"markdown","source":"# Automated EDA"},{"metadata":{"_uuid":"4ff1b55e030b6320842c050c9d77cee6ce334294"},"cell_type":"markdown","source":"Hi, This kernel is automatically generated by the [Data Geek Bot](https://github.com/Ankur3107/Kaggle-Data-Geek-Bot) - The kaggle bot to generate automated EDA for any dataset. "},{"metadata":{"_uuid":"59a31dae6a020f094a66e773539b74b447ec9c98"},"cell_type":"markdown","source":"### Loading Packages"},{"metadata":{"trusted":true,"_uuid":"35c4a892a057f6e5880284c59d0c97793f19acf2","_kg_hide-output":true},"cell_type":"code","source":"library(dplyr)\nlibrary(reshape)\nlibrary(data.table)\nlibrary(data.table)\nlibrary(formattable)\nlibrary(gridExtra)\nlibrary(funModeling)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd127e69878a4b9cc541002383574935d2c1cad0"},"cell_type":"markdown","source":"### Loading Data"},{"metadata":{"_uuid":"3217b2645629a54d4fa8de3ce8d44c9b52b9b675","_execution_state":"idle","trusted":true},"cell_type":"code","source":"input_dir = '../input/'\ncsv_files = list.files(input_dir, recursive = T, full.names = T)\ncsv_files = csv_files[grep('.csv', csv_files)]\ncsv_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"989143ec927b409f979f1f4ef20450314daaedc8"},"cell_type":"code","source":"data = read.csv(csv_files[5], stringsAsFactors = F)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9eb7ea9bdf04308ed8e327bbab7a3dba315b567d"},"cell_type":"markdown","source":"### Helper Functions"},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true,"_uuid":"7877e394e4e4b246bf2c4eaf48bf6e069170c52e"},"cell_type":"code","source":"# Function 1 : For ploting missing value\nplot_missing <- function(data, title = NULL, ggtheme = theme_gray(), theme_config = list(\"legend.position\" = c(\"bottom\"))) {\n ## Declare variable first to pass R CMD check\n feature <- num_missing <- pct_missing <- group <- NULL\n ## Check if input is data.table\n is_data_table <- is.data.table(data)\n ## Detect input data class\n data_class <- class(data)\n ## Set data to data.table\n if (!is_data_table) data <- data.table(data)\n ## Extract missing value distribution\n missing_value <- data.table(\n \"feature\" = names(data),\n \"num_missing\" = sapply(data, function(x) {sum(is.na(x))})\n )\n missing_value[, feature := factor(feature, levels = feature[order(-rank(num_missing))])]\n missing_value[, pct_missing := num_missing / nrow(data)]\n missing_value[pct_missing < 0.05, group := \"Good\"]\n missing_value[pct_missing >= 0.05 & pct_missing < 0.4, group := \"OK\"]\n missing_value[pct_missing >= 0.4 & pct_missing < 0.8, group := \"Bad\"]\n missing_value[pct_missing >= 0.8, group := \"Remove\"][]\n ## Set data class back to original\n if (!is_data_table) class(missing_value) <- data_class\n ## Create ggplot object\n output <- ggplot(missing_value, aes_string(x = \"feature\", y = \"num_missing\", fill = \"group\")) +\n geom_bar(stat = \"identity\") +\n geom_text(aes(label = paste0(round(100 * pct_missing, 2), \"%\"))) +\n scale_fill_manual(\"Group\", values = c(\"Good\" = \"#1a9641\", \"OK\" = \"#a6d96a\", \"Bad\" = \"#fdae61\", \"Remove\" = \"#d7191c\"), breaks = c(\"Good\", \"OK\", \"Bad\", \"Remove\")) +\n scale_y_continuous(labels = comma) +\n coord_flip() +\n xlab(\"Features\") + ylab(\"Number of missing rows\") +\n ggtitle(title) +\n ggtheme + theme_linedraw()+\n do.call(theme, theme_config)\n ## Print plot\n print(output)\n ## Set return object\n return(invisible(missing_value))\n}\n\n# Function 2: For plotting histogram\nplot_histogram <- function(data, title = NULL, ggtheme = theme_gray(), theme_config = list(), ...) {\n if (!is.data.table(data)) data <- data.table(data)\n ## Stop if no continuous features\n if (split_columns(data)$num_continuous == 0) stop(\"No Continuous Features\")\n ## Get continuous features\n continuous <- split_columns(data)$continuous\n ## Get dimension\n n <- nrow(continuous)\n p <- ncol(continuous)\n ## Calculate number of pages\n pages <- ceiling(p / 16L)\n for (pg in seq.int(pages)) {\n ## Subset data by column\n subset_data <- continuous[, seq.int(16L * pg - 15L, min(p, 16L * pg)), with = FALSE]\n setnames(subset_data, make.names(names(subset_data)))\n n_col <- ifelse(ncol(subset_data) %% 4L, ncol(subset_data) %/% 4L + 1L, ncol(subset_data) %/% 4L)\n ## Create ggplot object\n plot <- lapply(\n seq_along(subset_data),\n function(j) {\n x <- na.omit(subset_data[, j, with = FALSE])\n ggplot(x, aes_string(x = names(x))) +\n geom_histogram(bins = 30L, ...,fill='#92b7ef') +\n scale_x_continuous(labels = comma) +\n scale_y_continuous(labels = comma) +\n ylab(\"Frequency\") +\n ggtheme + theme_linedraw()+\n do.call(theme, theme_config)\n }\n )\n ## Print plot object\n if (pages > 1) {\n suppressWarnings(do.call(grid.arrange, c(plot, ncol = n_col, nrow = 4L, top = title, bottom = paste(\"Page\", pg))))\n } else {\n suppressWarnings(do.call(grid.arrange, c(plot, top = title)))\n }\n }\n}\n\n# Function 3 : Getting missing values\n.getAllMissing <- function(dt) {\n if (!is.data.table(dt)) dt <- data.table(dt)\n sapply(dt, function(x) {\n sum(is.na(x)) == length(x)\n })\n}\n\n# Function 4 : Spliting columns\nsplit_columns <- function(data) {\n ## Check if input is data.table\n is_data_table <- is.data.table(data)\n ## Detect input data class\n data_class <- class(data)\n ## Set data to data.table\n if (!is_data_table) data <- data.table(data)\n ## Find indicies for continuous features\n all_missing_ind <- .getAllMissing(data)\n ind <- sapply(data[, which(!all_missing_ind), with = FALSE], is.numeric)\n ## Count number of discrete, continuous and all-missing features\n n_all_missing <- sum(all_missing_ind)\n n_continuous <- sum(ind)\n n_discrete <- ncol(data) - n_continuous - n_all_missing\n ## Create object for continuous features\n continuous <- data[, which(ind), with = FALSE]\n ## Create object for discrete features\n discrete <- data[, which(!ind), with = FALSE]\n ## Set data class back to original\n if (!is_data_table) class(discrete) <- class(continuous) <- data_class\n ## Set return object\n return(\n list(\n \"discrete\" = discrete,\n \"continuous\" = continuous,\n \"num_discrete\" = n_discrete,\n \"num_continuous\" = n_continuous,\n \"num_all_missing\" = n_all_missing\n )\n )\n}\n\n# Function 5 : plotting density plot for numerical variable \nplot_density <- function(data, title = NULL, ggtheme = theme_gray(), theme_config = list(), ...) {\n if (!is.data.table(data)) data <- data.table(data)\n ## Stop if no continuous features\n if (split_columns(data)$num_continuous == 0) stop(\"No Continuous Features\")\n ## Get continuous features\n continuous <- split_columns(data)$continuous\n ## Get dimension\n n <- nrow(continuous)\n p <- ncol(continuous)\n ## Calculate number of pages\n pages <- ceiling(p / 16L)\n for (pg in seq.int(pages)) {\n ## Subset data by column\n subset_data <- continuous[, seq.int(16L * pg - 15L, min(p, 16L * pg)), with = FALSE]\n setnames(subset_data, make.names(names(subset_data)))\n n_col <- ifelse(ncol(subset_data) %% 4L, ncol(subset_data) %/% 4L + 1L, ncol(subset_data) %/% 4L)\n ## Create ggplot object\n plot <- lapply(\n seq_along(subset_data),\n function(j) {\n x <- na.omit(subset_data[, j, with = FALSE])\n ggplot(x, aes_string(x = names(x))) +\n geom_density(...,fill=\"#e2c5e5\") +\n scale_x_continuous(labels = comma) +\n scale_y_continuous(labels = percent) +\n ylab(\"Density\") +\n ggtheme + theme_linedraw()+\n do.call(theme, theme_config)\n }\n )\n ## Print plot object\n if (pages > 1) {\n suppressWarnings(do.call(grid.arrange, c(plot, ncol = n_col, nrow = 4L, top = title, bottom = paste(\"Page\", pg))))\n } else {\n suppressWarnings(do.call(grid.arrange, c(plot, top = title)))\n }\n }\n}\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"29361b6b6c2e4ab62e6933d3221567e1ad9aa9f5"},"cell_type":"markdown","source":"# Exploratory Data Analysis"},{"metadata":{"_uuid":"34613832e1b62c1b2c350926c3a774003ad0b55e"},"cell_type":"markdown","source":"## Structure & Dimension"},{"metadata":{"trusted":true,"_uuid":"eb9bf6bf4c5aeea20a0f29613be634537201ccab"},"cell_type":"code","source":"print(paste(\"The Dataset have\",dim(data)[1],\"data points and \", dim(data)[2], \"Features.\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c5b76f4811c15724bc20771e807f38c6d7bc9ce"},"cell_type":"code","source":"print(\"Let's see the head of data.\")\nprint(head(data))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ac420347fcddbe97bc9f1eb8e835339cb718165"},"cell_type":"markdown","source":"## Profiling Data Input\n\nProbably one of the first steps, when we get a new dataset to analyze, is to know if there are missing values (NA in R) and the data type.\n\nThe **df_status** function coming in funModeling can help us by showing these numbers in relative and percentage values. It also retrieves the infinite and zeros statistics.\n\n* **q_zeros:** quantity of zeros (p_zeros: in percent)\n* **q_inf:** quantity of infinite values (p_inf: in percent)\n* **q_na:** quantity of NA (p_na: in percent)\n* **type:** factor or numeric\n* **unique:** quantity of unique values"},{"metadata":{"trusted":true,"_uuid":"5c098e6787aa5ff4b329a338d793887862f0a09b"},"cell_type":"code","source":"df_status(data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb5b40fe4e9fc75c09021d5d3ffd4a429b22cd70"},"cell_type":"markdown","source":"## Missing Value Analysis"},{"metadata":{"trusted":true,"_uuid":"9b98af53533beb748bca328d08a3524c66f0c4ae","_kg_hide-input":false},"cell_type":"code","source":"plot_missing(data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66bf5c2239b9abd8ad3b80de8b4d7f587d8e5423"},"cell_type":"markdown","source":"## Categorical Feature Analysis"},{"metadata":{"trusted":true,"_uuid":"0a7ada4656a3beab9c5a767180424e966828a2fe"},"cell_type":"code","source":"getDataFrameWith50Categories <- function(df){\n factorDF <- mutate_all(df, function(x) as.factor(x))\n features <- names(factorDF)\n for(feature in features){\n if(length(levels(factorDF[,feature]))>50){\n factorDF[feature] <- NULL\n }\n \n }\n factorDF \n}\n \ncategoricalData <- getDataFrameWith50Categories(data)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5056d853c1f7f533aea1f9603ccfef46936deb86"},"cell_type":"markdown","source":"### Describe Categorical Features"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"e7a1a081d2c48fd79d00f7dbd3fa9a32746c8b6e"},"cell_type":"code","source":"describe(categoricalData)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8fc2ac8ccb65006cbe46c58f948ad1aab1bbc40"},"cell_type":"markdown","source":"### Categorical Features Plotting"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"9e0150e90d79bc99921fae1493193beab6730ff8"},"cell_type":"code","source":"freq(categoricalData)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60ff9b412112c8252563ddc60f3f609b0b1a2a1f"},"cell_type":"markdown","source":"## Numerical Feature Analysis"},{"metadata":{"trusted":true,"_uuid":"f6a0042e3fe913536f8345bd937aeea201ce9fa4"},"cell_type":"code","source":"getNumericalDF <- function(df){\n numericDF <- df\n features <- names(numericDF)\n for(feature in features){\n if(!is.numeric(df[,feature])){\n numericDF[feature] <- NULL\n }\n }\n numericDF\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21a398a8cb159b67179dd4317e933c4cbef7ca9c","_kg_hide-output":true},"cell_type":"code","source":"numericalData <- getNumericalDF(data) # which are num/int data type\nnumericalDataFeature <- names(numericalData) \ncategoricalDataFeature <- names(categoricalData)\nnumericalData <- select(numericalData, -one_of(categoricalDataFeature)) # select that feature which are not categoricalDataFeature","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3463923f1e0585cc5fe423345c389fec4d073cc8"},"cell_type":"markdown","source":"* **variable:** variable name\n\n* **mean:** the well-known mean or average\n\n* **std_dev:** standard deviation, a measure of dispersion or spread around the mean value. A value around 0 means almost no variation (thus, it seems more like a constant); on the other side, it is harder to set what high is, but we can tell that the higher the variation the greater the spread. Chaos may look like infinite standard variation. The unit is the same as the mean so that it can be compared.\n\n* **variation_coef:** variation coefficient=std_dev/mean. Because the std_dev is an absolute number, it’s good to have an indicator that puts it in a relative number, comparing the std_dev against the mean A value of 0.22 indicates the std_dev is 22% of the mean If it were close to 0 then the variable tends to be more centered around the mean. If we compare two classifiers, then we may prefer the one with less std_dev and variation_coef on its accuracy.\n\n* **p_01, p_05, p_25, p_50, p_75, p_95, p_99:** Percentiles at 1%, 5%, 25%, and so on. Later on in this chapter is a complete review about percentiles."},{"metadata":{"trusted":true,"_uuid":"a58834b839494147ea8618253b4371cc043f0f42"},"cell_type":"code","source":"profiling_num(numericalData)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12e77867887a1d5f0dadf601805e92ead888761c"},"cell_type":"markdown","source":"### Numerical Feature Plotting (Histogram)"},{"metadata":{"trusted":true,"_uuid":"f95d13cb8f84ce3ae900082810c0ad985058a4ba"},"cell_type":"code","source":"plot_histogram(numericalData)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"347eaffbf7914c84fb520d4bc58886de38cd3d83"},"cell_type":"markdown","source":"### Numerical Feature Plotting (Density)"},{"metadata":{"trusted":true,"_uuid":"7c16fa8a82f1b75cfbf8c8d5b9d7f23097c2d123"},"cell_type":"code","source":"plot_density(numericalData)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eea9dc8dcfa1292e9038cc6ef9d9ec0be409d913"},"cell_type":"markdown","source":"\n Contact With Me
\n\n \n \n \n \n "}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}