{"cells":[{"metadata":{},"cell_type":"markdown","source":"Hi Kagglers! Welcome to the basic EDA kernel for ASHRAE Great Energy Prediction competition! My focus in this kernel will be to build some basic understanding of the datasets provided and then move on to build predictive models."},{"metadata":{"_uuid":"051d70d956493feee0c6d64651c6a088724dca2a","_execution_state":"idle","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"## Importing packages\nlibrary(tidyverse)\nlibrary(data.table)\nlibrary(Hmisc)\nlibrary(lubridate)\nlibrary(caret)\nlibrary(lightgbm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Loading & Understanding the Datasets"},{"metadata":{},"cell_type":"markdown","source":"Following files are pre-loaded in our directory."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"list.files(path = \"../input/ashrae-energy-prediction\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's read the datasets."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"building <- fread(\"../input/ashrae-energy-prediction/building_metadata.csv\", key = \"building_id\")\ntrain <- fread(\"../input/ashrae-energy-prediction/train.csv\", key = \"building_id\")\nweather_train <- fread(\"../input/ashrae-energy-prediction/weather_train.csv\", key = \"site_id\")\ntest <- fread(\"../input/ashrae-energy-prediction/test.csv\", key = \"building_id\")\nweather_test <- fread(\"../input/ashrae-energy-prediction/weather_test.csv\", key = \"site_id\")\ngc()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at our datastes one by one."},{"metadata":{},"cell_type":"markdown","source":"#### Building Data"},{"metadata":{},"cell_type":"markdown","source":"This dataset contains metadata about buildings. Take a look at the variables present in the dataset."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Variables in the dataset\nnames(building)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Following description is provided to us:\n+ site_id - Foreign key for the weather files.\n+ building_id - Foreign key for training.csv\n+ primary_use - Indicator of the primary category of activities for the building based on EnergyStar property type definitions\n+ square_feet - Gross floor area of the building\n+ year_built - Year building was opened\n+ floor_count - Number of floors of the building\n\nLet's look at the structure of the dataset."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"glimpse(building)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"[](http://)It has 1449 observations and 6 variables. Take a look at the description of dataset."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"summary(building)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Following observations can be made after looking at the above outputs and data description.\n   + *site_id* is just an unique identifier, acting as foriegn key for weather data.\n   + *building_id* is just an unique identifier, acting as a foriegn key for training file.\n   + *primary_use* is an indicator variable. We will convert it to factor.\n   + *year_built* and *floor_count* has many missing values. \n   + Looking at the summary statistics of *floor_count*, it seems that this should be a factor.\n   \nLet's make the above changes before we proceed."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"building$primary_use <- as.factor(building$primary_use)\nbuilding$floor_count <- as.factor(building$floor_count)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Take a look at the summary to confirm the changes made."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# summary of building dataset\nsummary(building)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Training Data "},{"metadata":{},"cell_type":"markdown","source":"This is the dataset which contains our dependenent variable. Take a look at the variable names."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Variables in train set\nnames(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Following description is provided to us:\n + building_id - Foreign key for the building metadata.\n + meter - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, 3: hotwater}. Not every building has all meter types.\n + timestamp - When the measurement was taken\n + meter_reading - The target variable. Energy consumption in kWh (or equivalent).\n\nLet's look at the structure of the dataset."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"glimpse(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It contains 4 variables and 20,216,100 observations. Let's see the summary of the data."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"summary(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Following observations can be made after looking at the above outputs.\n   + *building_id* is the foreign key for building metadata file.\n   + *meter* has 4 unique values. It should be a factor.\n   + *timestamp* needs to be converted to date and time format.\n   + There are no missing values in training set.\n   \nLet's make the above changes before we proceed."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"train$meter <- as.factor(train$meter)\ntrain$timestamp <- ymd_hms(train$timestamp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Take a look at the summary to confirm the changes made."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"summary(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Train Weather Data"},{"metadata":{},"cell_type":"markdown","source":"This dataset contains weather data from a meteorological station as close as possible to the site. Let's take a look at the variables measured."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"names(weather_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Following data description is given:\n + site_id\n + air_temperature - Degrees Celsius\n + cloud_coverage - Portion of the sky covered in clouds, in oktas\n + dew_temperature - Degrees Celsius\n + precip_depth_1_hr - Millimeters\n + sea_level_pressure - Millibar/hectopascals\n + wind_direction - Compass direction (0-360)\n + wind_speed - Meters per second\n\nTake a look at the structure and summary of the dataset."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"glimpse(weather_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"summary(weather_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Following observations can be made after looking at the above outputs.\n   + We shall convert *timestamp* to date-time format.\n   + There are many missing values in training set.\n   \nLet's make the above changes."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"weather_train$timestamp <- ymd_hms(weather_train$timestamp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Test Data"},{"metadata":{},"cell_type":"markdown","source":"Following description is provided:\n + row_id - Row id for submission file\n + building_id - Building id code\n + meter - The meter id code\n + timestamp - Timestamps for the test data period\n \nLet's take a deeper look at our test data now."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"glimpse(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's a pretty huge dataset. It might not be a good idea to join it to the train dataset to make complete dataset. Let's make changes to the variable types, as done for other datasets."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"test$meter <- as.factor(test$meter)\ntest$timestamp <- ymd_hms(test$timestamp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see the summary now."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"summary(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Test Weather Data"},{"metadata":{},"cell_type":"markdown","source":"This dataset is similar to weather dataset for training dataset and contains weather information for test dataset. Let's see the structure of this dataset."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"glimpse(weather_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Make changes to the variables as done for training weather data."},{"metadata":{"trusted":true},"cell_type":"code","source":"# formating weather test dataset\nweather_test$timestamp <- ymd_hms(weather_test$timestamp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### EDA"},{"metadata":{},"cell_type":"markdown","source":"Let's begin our EDA by combining above datasets to get our complete training and test set."},{"metadata":{},"cell_type":"markdown","source":"#### Creating combined training data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_complete <- train %>% left_join(building) %>% left_join(weather_train)\ndim(train_complete)\ngc()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Creating combined test data"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"test_complete <- test %>% left_join(building) %>% left_join(weather_test)\ndim(test_complete)\n\n# removing datasets not required\nrm(building, train, weather_train, weather_test, test)\ngc()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Missing Values"},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at the missing values present in the dataset for each variable."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"missing_values_per_column <- colSums(sapply(train_complete, is.na))\nmissing_percent_per_column <- round(missing_values_per_column/nrow(train_complete)*100, 2)\nsort(missing_percent_per_column[missing_percent_per_column > 0], decreasing = T)\nrm(missing_values_per_column, missing_percent_per_column)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's a lot of missing values for few variables. We will deal with this later."},{"metadata":{},"cell_type":"markdown","source":"#### Sampling for visualizing our dataset"},{"metadata":{},"cell_type":"markdown","source":"Since the datasets are huge, let's sample 100000 rows from our dataset to do rest of our EDA. "},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"set.seed(1985)\ns <- sample(nrow(train_complete), 100000)\nsampled_train <- train_complete[s, ]\nrm(s)\ngc()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Take a look at few rows of our sampled train set."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"head(sampled_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### The Target Variable: meter_reading"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% ggplot(aes(meter_reading)) + geom_histogram(fill = \"blue\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is a very skewed distribution. There are many outliers. Let's apply log transformation to see whether it becomes normal."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% ggplot(aes(log(meter_reading + 1))) + geom_histogram(fill = \"blue\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now the distribution looks quite normal. Going forward, we will use this transformed variable to compared with other independent variables."},{"metadata":{},"cell_type":"markdown","source":"#### Independent Variables"},{"metadata":{},"cell_type":"markdown","source":"Now, we will visualize our independent variables one by one and also look at their relation with our target variable."},{"metadata":{},"cell_type":"markdown","source":"**Meter Type**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"table(sampled_train$meter)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see whether their is any differnce between meter readings for different meter type."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% ggplot(aes(meter, log(meter_reading + 1), fill = meter)) + geom_boxplot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Indeed, there seems to be quite a difference between meter readings for different meter types."},{"metadata":{},"cell_type":"markdown","source":"#### Site Id"},{"metadata":{},"cell_type":"markdown","source":"Let's see how many sites are present in the dataset?"},{"metadata":{"trusted":true},"cell_type":"code","source":"unique(sampled_train$site_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have 16 unique sites present in the dataset. Let's see their frequency distribution."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"table(sampled_train$site_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Site 7, 10, 11 & 12 have very low representation while site 2,3 9, 13 & 14 have very high representation in the dataset."},{"metadata":{},"cell_type":"markdown","source":"**Do energy consumption differ across different sites?**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% ggplot(aes(factor(site_id), log(meter_reading + 1)), fill = site_id) + geom_boxplot(fill = \"blue\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is quite a variation between different sites in terms of energy usage. Site 5 & 8 have lowest energy consumption while site site 7 have highest energy consumption. There are many bivariate outliers present."},{"metadata":{},"cell_type":"markdown","source":"**Primary Use**"},{"metadata":{},"cell_type":"markdown","source":"Take a look at its distribution."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"table(sampled_train$primary_use)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Education, Office, Entertainment are categories which dominates here. Let's see relation of this w.r.t. our target variable."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"options(repr.plot.width = 12, repr.plot.height = 6)\nsampled_train %>% ggplot(aes(primary_use, log(meter_reading + 1), fill = primary_use)) + geom_boxplot() +\n    coord_flip()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"   + Building used for religious worship consumes lowest energy.\n   + Helthcare buildings have highest energy usage.\n   + There are some very high energy consuming buildings (outliers) for some of the usage category."},{"metadata":{},"cell_type":"markdown","source":"#### Square Feet"},{"metadata":{},"cell_type":"markdown","source":"Let's see the distribution of this variable."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sampled_train %>% ggplot(aes(square_feet)) + geom_histogram(fill = \"blue\") + \n    geom_vline(xintercept = median(sampled_train$square_feet))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here also, the distribution is quite skewed. The median square foot of the buildings present in the dataset is 72709. "},{"metadata":{},"cell_type":"markdown","source":"#### Year Built"},{"metadata":{},"cell_type":"markdown","source":"Let's see the distribution of this variable first."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% ggplot(aes(year_built)) + geom_histogram(fill = \"blue\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a peak at around 1975. It might be due to more number of buildings built during that year. Let's visualize that as well."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% group_by(year_built) %>% \n    summarise(num_of_buildings = n()) %>% ggplot(aes(year_built, num_of_buildings)) +\n        geom_line(col = \"blue\") + ylim(0,3500)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Do older houses consumes more power?**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% group_by(year_built) %>% ggplot(aes(year_built, log(meter_reading + 1))) +\n        geom_point() + geom_smooth(method = \"lm\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No such indication here. Let's move on to our next independent variable."},{"metadata":{},"cell_type":"markdown","source":"#### Floor Count"},{"metadata":{},"cell_type":"markdown","source":"Let's see the distribution of this variable now."},{"metadata":{"trusted":true},"cell_type":"code","source":"table(sampled_train$floor_count)\nsampled_train %>% ggplot(aes(floor_count)) + geom_bar(fill = \"blue\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Buildings with more number of floors, consumes more power?**"},{"metadata":{"trusted":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% ggplot(aes(floor_count, log(meter_reading + 1))) +\n        geom_boxplot(fill = \"blue\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Indeed. Energy consumption seems to be increasing with number of floors till certain level, then it dips off."},{"metadata":{},"cell_type":"markdown","source":"#### Air Temperature"},{"metadata":{},"cell_type":"markdown","source":"Let's see the distribution of this variable present in our dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% ggplot(aes(air_temperature)) + geom_histogram(fill = \"blue\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is quite uniform looking distribution varying from around -25 deg C to around 50 deg C. I do not expect it to be directly related with our dependent variable as colder weather will increase energy consumption on heaters and hotter weather will increase energy consumption on chillers. Let's check that."},{"metadata":{"trusted":true},"cell_type":"code","source":"options(repr.plot.width = 9, repr.plot.height = 6)\nsampled_train %>% ggplot(aes(air_temperature, log(meter_reading + 1))) + geom_point()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No relation as we argued. Let's try to see the impact of air temperature on enrgy consumption of different meter types."},{"metadata":{"trusted":true},"cell_type":"code","source":"options(repr.plot.width = 12, repr.plot.height = 9)\nsampled_train %>% ggplot(aes(air_temperature, log(meter_reading))) + geom_point() + \n    geom_smooth(method = \"lm\") + facet_wrap(~meter)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Except *electricity* consumption, we do see a relation here. With rising temperature there is an increase in *chilledwater* consumption, while *steam* and *hotwater* consumption is going down, as expected."},{"metadata":{},"cell_type":"markdown","source":"TBC"},{"metadata":{},"cell_type":"markdown","source":"**Please upvote, if you like it!**"}],"metadata":{"kernelspec":{"display_name":"R","language":"R","name":"ir"},"language_info":{"mimetype":"text/x-r-source","name":"R","pygments_lexer":"r","version":"3.4.2","file_extension":".r","codemirror_mode":"r"}},"nbformat":4,"nbformat_minor":1}