{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ASHRAE - Great Energy Predictor III Competition\nThis notebook contains Exploratory Data Analysis and starter code for the Great Energy Predictor III Competition.\n\n**tl;dr**:\n- We are trying to predict energy consuption for 1449 buildings. The value we are trying to predict is the `meter_reading`\n- Each building can have multiple meters - The meter id code. Read as `{0: electricity, 1: chilledwater, 2: steam, 3: hotwater}`. Not every building has all meter types.\n- We are given:\n    1. Historic meter reading data by timestamp for the building (`train.csv`)\n    2. Building metadata including the building use, square ft area, year build(`building_meta.csv`). This data does not change between the training and test set.\n    3. Weather data with predicpitation, cloud_coverage, `air_temperature` and more (`weather_[train/test].csv`)\n- We are also provied csvs to be used for submission:\n    1. `test.csv` which contains the meter, building id and timestamp we will be predicting for\n    2. `sample_submission.csv` which contains all the future data we would like to predict\n\n**In summary from the competition description:**\n*In this competition, you’ll develop accurate predictions of metered building energy usage in the following areas: chilled water, electric, natural gas, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe.*"},{"metadata":{},"cell_type":"markdown","source":"## Reading in Data and preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pylab as plt\nimport seaborn as sns\n\ntrain = pd.read_csv('../input/ashrae-energy-prediction/train.csv')\ntest = pd.read_csv('../input/ashrae-energy-prediction/test.csv')\nweather_te = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv')\nweather_tr = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv')\nbmd = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv')\n\n# Set timestamps\ntrain['timestamp'] = pd.to_datetime(train['timestamp'])\ntest['timestamp'] = pd.to_datetime(test['timestamp'])\nweather_tr['timestamp'] = pd.to_datetime(weather_tr['timestamp'])\nweather_te['timestamp'] = pd.to_datetime(weather_te['timestamp'])\n\nsns.set(style=\"whitegrid\")\nsns.set_color_codes(\"pastel\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Time Series Competition\n\nIt's important to note that the data provided is `time series` in nature. We are given one year of data (2016) and are asked to predict 2 years of meter readings.\n\nPer the description:\n\n**This competition challenges you to build these counterfactual models across four energy types based on historic usage rates and observed weather. The dataset includes three years of hourly meter readings from over one thousand buildings at several different sites around the world.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"meter_mapping = {0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'}\ntrain['meter_type'] = train['meter'].map(meter_mapping)\ntest['meter_type'] = test['meter'].map(meter_mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(['timestamp','meter_type'])['meter_reading'] \\\n    .median() \\\n    .reset_index().set_index('timestamp') \\\n    .groupby('meter_type')['meter_reading'] \\\n    .plot(figsize=(15, 5), title='Median Meter Reading by Meter Type (Test Set)')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['train'] = 1\ntest['train'] = 0\ntt = pd.concat([train, test], axis=0, sort=True)\n\ntt.groupby(['timestamp','meter_type'])['meter_reading'] \\\n    .median() \\\n    .reset_index().set_index('timestamp') \\\n    .groupby('meter_type')['meter_reading'] \\\n    .plot(figsize=(15, 5), title='Median Meter Reading by Meter Type (train and test timeframe)')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluation Metric\n\nWe will be evaluated by the metirc `Root Mean Squared Logarithmic Error`.\n\nThe RMSLE is calculated as:\n\n$ ϵ=1n∑i= \\sqrt{ 1/n (log(pi+1)−log(ai+1))^2 } $\nWhere:\n\n- ϵ is the RMSLE value (score)\n- n is the total number of observations in the (public/private) data set,\n- pi is your prediction of target, and\n- ai is the actual target for i.\n- log(x) is the natural logarithm of x\n\nUnderstanding and optimizing your predictions for this evaluation metric is paramount for this compeition.\n\nAs mentioned in this discussion thread this can be calculated as https://www.kaggle.com/questions-and-answers/60012\n\n```\nfrom sklearn.metrics import mean_squared_log_error\nnp.sqrt(mean_squared_log_error( y_test, predictions ))\n```"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Evaluating the Target Variable\n\nAs always we will start by looking at the larget variable. As we have 1000+ buildings that we have meter data for. We will look at the meter data by meter type.\n- Electricity meters are 3x more common than the next meter type (chilledwater)\n- Steam has much larger average meter values than the rest (13882 average reading)"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(train.groupby('meter_type')['meter_reading'] \\\n                 .describe() \\\n                 .astype(int)) \\\n                 .sort_values('count',\n                              ascending=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plotting the distribution of the target.\nFirst thing we notice here is the extremely skewed distribution due to a few values that are very very large...."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['meter_reading'].plot(kind='hist',\n                        bins=50,\n                        figsize=(15, 2),\n                       title='Distribution of Target Variable (meter_reading)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Removing the high values we can get a better idea about the distribution of values. We may want to create different models for different buildings."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.query('meter_reading < 5000')['meter_reading'] \\\n    .plot(kind='hist',\n          figsize=(15, 3),\n          title='Distribution of meter_reading, excluding values greater than 5000',\n          bins=200)\nplt.show()\ntrain.query('meter_reading < 500')['meter_reading'] \\\n    .plot(kind='hist',\n          figsize=(15, 3),\n          title='Distribution of meter_reading, excluding values greater than 500',\n         bins=200)\nplt.show()\ntrain.query('meter_reading < 100')['meter_reading'] \\\n    .plot(kind='hist',\n          figsize=(15, 3),\n          title='Distribution of meter_reading, excluding values greater than 100',\n         bins=100)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Target for a single building /w Multiple Meters. Viewing over Time.\nOn inspection of the data over time, we can see that this data is very messy. There appears to be times when the values drop to zero."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.query('building_id == 0 and meter == 0') \\\n    .set_index('timestamp')['meter_reading'].plot(figsize=(15, 3),\n                                                 title='Building 0 - Meter 0')\n\nplt.show()\ntrain.query('building_id == 753').set_index('timestamp').groupby('meter')['meter_reading'].plot(figsize=(15, 3),\n                                                 title='Building 753 - Meters 0-3')\nplt.show()\ntrain.query('building_id == 1322').set_index('timestamp').groupby('meter')['meter_reading'].plot(figsize=(15, 3),\n                                                 title='Building 1322 - Meters 0-3')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using Building Metadata"},{"metadata":{"trusted":true},"cell_type":"code","source":"# First take a look at the building metadata\nbmd.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- We see that there is spike in buildings that were built in the year 1976\n- 774 buildings have no year built information"},{"metadata":{"trusted":true},"cell_type":"code","source":"bmd.groupby('year_built')['site_id'] \\\n    .count() \\\n    .plot(figsize=(15, 5),\n          style='.-',\n          title='Building Meta Data - Count by Year Built')\nplt.show()\nprint('{} Buildings have no year data.'.format(np.sum(bmd['year_built'].isna())))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Building Primary Use\n- Education is the most common type of building use, with office second.\n- There is a steep drop off in number of buildings after Lodging."},{"metadata":{"trusted":true},"cell_type":"code","source":"bmd.groupby('primary_use') \\\n    .count()['site_id'] \\\n    .sort_values() \\\n    .plot(kind='barh',\n          figsize=(15, 5),\n          title='Count of Buildings by Primary Use')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Aggregate some meter reading stats\nmeter_reading_stats = train.groupby('building_id')['meter_reading'].agg(['mean','max','min']).reset_index()\nbmd_with_stats = pd.merge(bmd, meter_reading_stats, on=['building_id']).rename(columns={'mean':'mean_meter_reading',\n                                                                       'max':'max_meter_reading',\n                                                                       'min':'min_meter_reading'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Building Type and Meter Reading "},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\", category=RuntimeWarning) \nsns.pairplot(bmd_with_stats.dropna(),\n             vars=['mean_meter_reading','min_meter_reading',\n                   'max_meter_reading','square_feet','year_built'],\n             hue='primary_use')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Time Series Impact on Energy Consumption"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['Weekday'] = train['timestamp'].dt.weekday\ntrain['Weekday_Name'] = train['timestamp'].dt.weekday_name\ntrain['Month'] = train['timestamp'].dt.month\ntrain['DayofYear'] = train['timestamp'].dt.dayofyear\ntrain['Hour'] = train['timestamp'].dt.hour","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In order to properly visualize the data, we can normalize the meter reading by type. This allows us to compare how the time series features impact each meter reading type, but on the same scale. The normalized value shows the value in relation to the meter type's average."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['normalized_meter_reading_type'] = \\\n    train.groupby('meter_type')['meter_reading'] \\\n        .transform(lambda x: (x - x.mean()) / x.std())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 5))\nsns.barplot(data=train.groupby(['Weekday_Name','meter_type']).mean().reset_index(),\n            x='Weekday_Name',\n            y='normalized_meter_reading_type',\n            hue='meter_type',\n            ax=ax)\nplt.title('Day of Week vs. Normalized Meter Reading')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 5))\nsns.barplot(data=train.groupby(['Month','meter_type']).mean().reset_index(),\n            x='Month',\n            y='normalized_meter_reading_type',\n            hue='meter_type',\n            ax=ax)\nplt.title('Month vs. Normalized Meter Reading')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 5))\nsns.barplot(data=train.groupby(['Hour','meter_type']).mean().reset_index(),\n            x='Hour',\n            y='normalized_meter_reading_type',\n            hue='meter_type',\n            ax=ax)\nplt.title('Hour within Day vs. Normalized Meter Reading')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 5))\nsns.lineplot(data=train.groupby(['DayofYear','meter_type']).mean().reset_index(),\n            x='DayofYear',\n            y='normalized_meter_reading_type',\n            hue='meter_type',\n            ax=ax)\n# plt.title('Day of Year vs. Normalized Meter Reading')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Next.... weather data .. check back soon."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}