{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this notebook I am going to clean the data and correct as well as it has been discussed [here](http://www.kaggle.com/c/ashrae-energy-prediction/discussion/119261)\nFirst we load all of our libraries. We will be using \n1. *data.table* - for reading the data quickly\n2. *tidyverse *\n3. *lubridate* - for extraction of time features such as day, month, hour \n4. *imputeTS* - for interpolating missing data"},{"metadata":{"_uuid":"051d70d956493feee0c6d64651c6a088724dca2a","_execution_state":"idle","trusted":true},"cell_type":"code","source":"suppressWarnings(suppressMessages(library(tidyverse)))\nsuppressWarnings(suppressMessages(library(data.table))); suppressWarnings(suppressMessages(library(lubridate)))\nsuppressMessages(library(imputeTS))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Importing Weather dataset. This will later be joined to the main training set.\n\nRight now I have loaded it and converted some variable to factors since they were categorical features."},{"metadata":{"trusted":true},"cell_type":"code","source":"df <- fread(\"../input/ashrae-energy-prediction/weather_train.csv\", stringsAsFactors = FALSE) %>%\nmutate(site_id = as.factor(site_id), cloud_coverage = as.factor(cloud_coverage)) %>%\nrename(wd = wind_direction, ws = wind_speed)\n\ncolSums(is.na(df))/nrow(df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Columns like air_temperature, dew_temperature, sea_level_pressure, wd and ws have small number of missing columns so you might want to interpolate these. One can also use a package called mi. But I wasn't able to understand it.\n\nI wasn't also able to interpolate *cloud_coverage* since it is a factor."},{"metadata":{"trusted":true},"cell_type":"code","source":"#df[, 3] <- na.interpolation(df[, 3])\n#df[, 7] <- na.interpolation(df[, 7])\n#df[, 8] <- na.interpolation(df[, 8])\n#df[, 9] <- na.interpolation(df[, 9])\n#df[, 5] <- na.interpolation(df[, 5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reading the train and building_metadata datasets..."},{"metadata":{"trusted":true},"cell_type":"code","source":"train <- fread(\"../input/ashrae-energy-prediction/train.csv\", stringsAsFactors = FALSE) %>%\nmutate(building_id = as.factor(building_id), meter = as.factor(meter))\n\nbuild_meta <- fread(\"../input/ashrae-energy-prediction/building_metadata.csv\", \n                    drop = c(\"year_built\", \"floor_count\")) %>%\nmutate(site_id = as.factor(site_id), building_id = as.factor(building_id), primary_use = as.factor(primary_use))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Joining the data to make a one big dataset..."},{"metadata":{"trusted":true},"cell_type":"code","source":"train <- left_join(train, build_meta, by = \"building_id\")\ntrain <- left_join(train, df, by = c(\"site_id\", \"timestamp\"))\n\ntrain <- train %>% mutate(timestamp = as.POSIXct(strptime(timestamp, \"%Y-%m-%d %H:%M:%S\")), month = as.factor(month(timestamp)), day = as.factor(day(timestamp)),\n                         hour = as.factor(hour(timestamp)), building_id = as.factor(building_id)) %>% select(-c(timestamp))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Since electric meter readings of site 0 were reported worng due to miscommunication,  we will convert them from kBTU to kWH.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train[which(train$site_id %in% \"0\" & train$meter %in% 0), ] <- train[which(train$site_id %in% \"0\" & train$meter %in% 0), ] %>% mutate(meter_reading = meter_reading*0.29307)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}