{"cells":[{"metadata":{},"cell_type":"markdown","source":"![Save Energy](https://isoilonline.com/wp-content/uploads/energy-efficiency.jpg)"},{"metadata":{},"cell_type":"markdown","source":"# Quick Intro\nA quick intro to the Kernel. The competition seeks to retroactively measure future hourly energy consumption for 2 years based on past one year consumption. The training data for hourly energy consumption is collected from thousands of buildings across 16 sites measuring electricity, chilled water, steam and hotwater consumption. To aid in the analysis building attributes such as square feet, floor count etc. are collected as well as various ambient weather conditions at the 16 sites in which the buildings under study are located. \n\nIn this Kernel, we will do EDA on various aspects of the data such as energy consumption by building, site, pirmary use of building, type of energy used as well as the pattern of weather parameters such temperature, pressure, wind speed, cloud cover etc. and try to record what is the importance of each of these features so that we can explore options for feature selection and feature engineering as the road to prediction modeling."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Import Libraries\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Import and Preview Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata = pd.read_csv('/kaggle/input/ashrae-energy-prediction/building_metadata.csv')\nweather_train = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_train.csv')\ntrain = pd.read_csv('/kaggle/input/ashrae-energy-prediction/train.csv')\nweather_test = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_test.csv')\ntest = pd.read_csv('/kaggle/input/ashrae-energy-prediction/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preview Building_Metadata table"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('The shape of Building Metadata is :',building_metadata.shape)\nbuilding_metadata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preview Weather_train data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('The shape of weather_train data is :',weather_train.shape)\nweather_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preview train data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('The shape of train data is :',train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preview weather_test data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('The shape of weather_test data is :',weather_test.shape)\nweather_test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preview test data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('The shape of test data is :',test.shape)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Key observations\n## Key observations on building metadata \n### How many buildings are under the study?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of Buildings :', len(building_metadata.building_id.unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How many sites are under study and how many buildings under each site?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of Sites and number of Buildings in Each site')\nprint(building_metadata.site_id.unique())\nprint(building_metadata.site_id.value_counts().sort_index())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we have 1449 buildings in all spread across 16 sites."},{"metadata":{},"cell_type":"markdown","source":"### What are the buildings used for?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"building_count = building_metadata.primary_use.value_counts()\nprint('Number of Buildings by Primary Use:')\nplt.figure(figsize=(15,3))\nsns.barplot(building_count.index,building_metadata.primary_use.value_counts())\nplt.xticks(rotation = 90)\nplt.ylabel('Number of Buildings')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reduce Memory Consumption\n#### source: https://www.kaggle.com/kernels/scriptcontent/3684066/download\nThe train and test sets have millions of rows with a huge size in GBs. We will reduce the size of the dataframes for better performance."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n start_mem = df.memory_usage().sum() / 1024**2 \n for col in df.columns:\n col_type = df[col].dtypes\n if col_type in numerics:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64) \n end_mem = df.memory_usage().sum() / 1024**2\n if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"## Reducing memory of the data frames\nbuilding_metadata = reduce_mem_usage(building_metadata)\ntrain = reduce_mem_usage(train)\nweather_train = reduce_mem_usage(weather_train)\nweather_test = reduce_mem_usage(weather_test)\ntest = reduce_mem_usage(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Merge Datasets and extract Date time features"},{"metadata":{"trusted":true},"cell_type":"code","source":"# First merge train and building data\ntrain = pd.merge(train,building_metadata,how = 'left') \nprint(train.shape)\ntrain.head()\n\n# Now Merge train_building with weather_train data\ntrain = pd.merge(train,weather_train, on = ['site_id','timestamp'], how = 'left')\nprint(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# First merge test and building data\ntest = pd.merge(test,building_metadata,how = 'left') \nprint(test.shape)\ntest.head()\n\n# Now Merge test_building with weather_test data\ntest = pd.merge(test,weather_test, on = ['site_id','timestamp'], how = 'left')\nprint(test.shape)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Convert 'timestamp' to datetime object"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['timestamp'] = pd.to_datetime(train.timestamp)\ntest['timestamp'] = pd.to_datetime(test.timestamp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Missing value Report for final Merged Train Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def report_missing_data(df):\n print('Total Number of rows :', len(df))\n for column in df.columns:\n print(column,':', 'Missing rows:', sum(df[column].isnull()), '|', '% Missing: {:.2f}'.format(sum(df[column].isnull())*100/len(df)),'%')\nreport_missing_data(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Extract Date time features for Train set"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get month, day, weekday,weekday name, hour etc. from the datetime object - timestamp.\ntrain['month'] = train.timestamp.dt.month\ntrain['day'] = train.timestamp.dt.day\ntrain['weekday'] = train.timestamp.dt.weekday\ntrain['hour'] = train.timestamp.dt.hour\ntrain['weekday_name'] = train.timestamp.dt.weekday_name\n\n# Get meter names from codes for better understanding\nmeter_dict = {0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'}\ntrain['meter_name'] = train.meter.map(meter_dict)\n\n# Calculate elapsed years for each building\ntrain['meter_year'] = train.timestamp.dt.year \ntrain['elapsed_years'] = train.meter_year - train.year_built\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reduce the size of merged train data\ntrain = reduce_mem_usage(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check the distribution of the target variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(train.meter_reading, kde = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are too many meter readings with value of zero and hence confounding actual distribution. We can deal with such outliers through two ways\n\n* Outlier removal, based on outlier thresholds or setting IQR thresholds above or below which observations are treated as outliers.\n* Apply Log Transformation to bring the points closer so that the outliers are squished in, and other observations become visible.\n\nWe will use the Log transform approach and re-display the target variable - 'meter_reading' distribution. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot hist of log transformed target variable\ntrain['log_meter_reading'] = np.log(train.meter_reading + 1)\nsns.distplot(train.log_meter_reading)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we have managed to bring the target variable into a normal distribution. This will be useful when running Linear Regression models for predicting meter readings.\nNow Lets try to understand what are the factors influencing Energy consumption. We will analyze in the following angles:\n\n"},{"metadata":{},"cell_type":"markdown","source":"# Analyze Energy Consumption\nWe will focus on analyzing energy consumption across different dimensions - \n\n1. **By Primary Use** - Which types of buildings are more frequent in this study? which of them consume more energy on an average\n2. **By Meter Type** - Which meters are more installed in the buildings under this study? Which of them show more average energy consumption?\n3. **By Site Id** - Which Sites consume more energy? Some sites have more buildings and hence can contribute more to energy consumption. Some sites may use more of a particular energy meter which has its own consumption characteristics."},{"metadata":{},"cell_type":"markdown","source":"## 1.1 Analyze Energy Consumption based on Primary Use"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.subplot(1,3,2)\ntrain.groupby('primary_use')['building_id'].nunique().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'purple', title = 'Count of Buildings by Primary use')\nplt.subplot(1,3,1)\ntrain.groupby('primary_use')['meter_reading'].sum().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Total Energy consumption by Primary use')\nplt.subplot(1,3,3)\ntrain.groupby('primary_use')['meter_reading'].mean().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Mean Energy consumption by Primary use')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The ideal candidates for piloting energy consumption initiatives could be those which have more energy consumption across lesser number of buildings.\n\n**Education and Services have the highest mean energy consumption.** While Education also has the maximum total as well as mean energy consumption, Services while having few buildings have second highest mean consumption. **Service buildings while being fewer in number can contribute to more energy savings**.All other buildings have far less mean consumption.\n\nLets look deeper into the **'mean consumption by primary use'** plot and try to break it down based on **which meter types contribute to more consumption** in each of these primary use building type."},{"metadata":{},"cell_type":"markdown","source":"## 1.2 Energy Consumption by Primary Use by Meter Type"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#train.groupby(['primary_use','meter'])['meter_reading'].mean().sort_values(ascending = False).reset_index().plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Mean Energy consumption by Primary use')\npivot_df = train.groupby(['primary_use','meter_name'])['meter_reading'].mean().sort_values(ascending = False).reset_index()\npivot_df.head()\npivot_df = pivot_df.pivot(index='primary_use', columns='meter_name', values='meter_reading')\npivot_df['AllMeters'] = pivot_df.sum(axis = 1)\npivot_df.sort_values('AllMeters', ascending = False).drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'Mean Hourly Energy consumption by Primary use by meter', stacked = True)\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Steam is more used than other meter types in Education, Services, Public Services, Health care, office, parking, Food sales, lodging, manufacturing etc. Also Looks like Steam consumption is more than 90% for the highest energy consuming buildings like Education, Services etc. \n\nThough Electricity meters are installed in maximum number of buildings, they seem to show much lower hourly average consumption. hot water seems to be the least used meter.\nChilled water is more used in \n\nLets look at the overall trend for the mean consumption by meter type - we would expect steam to be the most dominating meter type."},{"metadata":{},"cell_type":"markdown","source":"## 2.1 Analyze Energy Consumption based on Meter used"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#plt.figure(figsize=(30,10))\nplt.subplot(1,3,2)\ntrain.groupby('meter_name')['building_id'].count().sort_values(ascending = False).plot(kind = 'bar', figsize = (15,6), color = 'purple', title = 'Count of Meter Type')\nplt.subplot(1,3,1)\ntrain.groupby('meter_name')['meter_reading'].sum().sort_values(ascending= False).plot(kind = 'bar', figsize = (15,6), color = 'green', title = 'Total Energy consumption by Meter Type' )\nplt.subplot(1,3,3)\ntrain.groupby('meter_name')['meter_reading'].mean().sort_values(ascending= False).plot(kind = 'bar', figsize = (15,6), color = 'green', title = 'Mean Energy consumption by Meter Type' )\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Electricity meters are** 3 times more used than chilled water **and **4 times more used than steam** and **6 times more used than hotwater**.\n\nInspite of that, **Steam and Chilled water seem to be showing highest mean energy consumption** in terms of Total as well as mean hourly energy consumption. **Electricity meter has the lowest mean consumption** while having total consumption slightly higher than hot water due to more meters being installed.\n\nGiven the huge margin between Steam and Other Meter types, **the overall trend for energy consumption will mostly likely be driven by Steam energy meters**."},{"metadata":{},"cell_type":"markdown","source":"## 3.1 Analyze mean energy consumption by time - by hours, days and months"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15, 6), dpi=100)\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Analyze mean energy consumption by time - by hours, days and months by **'meter type'**\nBy breaking down the overall mean energy consumption trend by 'meter type' we can get insight on how different forms of energy are being consumed through the year. We saw earlier that Steam meter type has the highest total and mean energy consumption, so we expect the trend for overall energy usage and that for steam meters should be pretty identical."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"meter_dict = {0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'}\n# Plot energy consumption by meter\nplt.figure(figsize = (20,10))\nfor i in range(4):\n plt.subplot(2,2,i+1)\n train[train.meter == i][['timestamp','meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(title = meter_dict[i], label = 'By Hour')\n train[train.meter == i][['timestamp','meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(title = meter_dict[i], label = 'By Day')\n train[train.meter == i][['timestamp','meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(title = meter_dict[i], label = 'By Month')\n #plt.title('No 1099: Energy consumption by time by meter type')\n plt.legend()\n plt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Analysis of Meter Types:**\n\n1. **Steam** : Closely mirrors the overall trend infact driving the overall trend - June Mid to Nov Mid - a big drop. Then a spike in Nov Mid. The scale of consumption in 10000's.\n2. **Electricity**: Constant trend from Jan to May after which the consumption peaks to a new constant average - 15% higher than before. The scale of consumption is in smaller 100's\n3. **Chilled water**: Gradual increase in consumption from Jan to Sep and then a spike in Sep/Oct before finally dropping to march levels by Dec. Probably this is end of summer and moving into winter.The scale of consumption is in 1000's\n4. **Hot water**: Peak at start of year and continuously falling to a trough (20% of peak) by June-July and then again rising to original peak by end of year. Probably this is summer time when hot water is not needed that much. Scale is in larger 100's to lower 1000's.\n\n#### Given that the trend is completely different for each meter type, should we be predicting consumption for each meter type separately?"},{"metadata":{},"cell_type":"markdown","source":"So once building 1099 was removed, the steam energy trend follows a more continuous pattern starting high at Jan, reducing gradually to a low between June to Sep and climbing up again back to jan levels by the end of the year. This is also the same trend for hotwater usage - kinda nice given steam and hotwater are close cousins:)"},{"metadata":{},"cell_type":"markdown","source":"## 3.3 Analyze Hourly Mean Energy Consumption by Site Id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.subplot(1,3,1)\ntrain.groupby('site_id')['meter_reading'].sum().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Total Energy consumption in sites 0 to 15')\nplt.subplot(1,3,2)\ntrain.groupby('site_id')['building_id'].nunique().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'purple', title = 'Count of Buildings in sites 0 to 15')\nplt.subplot(1,3,3)\ntrain.groupby('site_id')['meter_reading'].mean().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Mean Energy consumption in sites 0 to 15')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"6 Sites - 13, 9, 2, 14, 3 and 15 are having about twice the number of buildings as others. Site 13 in particular seems to be 10x as compared to the next highest energy consuming site. We would expect the overall mean energy consumption pattern to be driven by site 13."},{"metadata":{},"cell_type":"markdown","source":"## 3.4 Check Time Series Hourly and Daily energy consumption by Site Id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Plot energy consumption by site\nplt.figure(figsize = (20,20))\nfor i in range(16):\n plt.subplot(8,2,i+1)\n train[train.site_id == i][['timestamp','meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(title = 'Site {}'.format(i), label = 'By Hour')\n train[train.site_id == i][['timestamp','meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(title = 'Site {}'.format(i), label = 'By Day')\n plt.legend()\n plt.tight_layout() # Add tight layout in loop to prevent overlapping text\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Bingo! The trend for Site 13 and for overall trend seems to be identical.I bet Site 13 should have more of Steam consumption as the patterns for overall trend, Steam and Site 13 all look alike. \n\n"},{"metadata":{},"cell_type":"markdown","source":"Lets look at a few sites which are interesting\n\n1. Site 6 - Major consumption happens only in Sep and Oct. looking at the scale of consumption, it could be only because of steam. \n2. Site 8 - pretty normal trend - scale indicates electricity meter usage\n3. Site 9 - mostly constant low usage (could be electricity) with some peaks in between (could be chilled water or steam)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sites = [6,8,9]\nfor site in sites:\n print('site {}'.format(site), train[train.site_id == site].groupby('meter_name')['meter_reading'].sum())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.5 Energy consumption by Site by meter type"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"pivot_df = train.groupby(['site_id','meter_name'])['meter_reading'].mean().sort_values(ascending = False).reset_index()\npivot_df.head()\npivot_df = pivot_df.pivot(index='site_id', columns='meter_name', values='meter_reading')\npivot_df['AllMeters'] = pivot_df.sum(axis = 1)\npivot_df.sort_values('AllMeters', ascending = False).drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'Mean Energy consumption by Site Id by meter', stacked = True)\nplt.tight_layout()\n\n\n## Show values in bar plot\n#ax = pivot_df.sort_values('AllMeters', ascending = False).drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'Mean Energy consumption by Site Id by meter', stacked = True)\n#for p in ax.patches:\n # ax.annotate(str(p.get_height()), (p.get_x() * 1.005, p.get_height() * 1.005))\n#plt.tight_layout() \n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can clearly see that Except for Site 13 all other sites do not have mean steam energy usage that much. That is why other sites show a different trend from the Steam meter consumption trend. "},{"metadata":{},"cell_type":"markdown","source":"Finally lets get down to building level. Lets check the general trend of energy consumption by buildings."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby('building_id')['meter_reading'].mean().plot(figsize = (20,6), color = 'green', title = 'Mean Energy consumption in all 1449 buildings')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like there is one outlier which is dwarfing all other buildings energy consumption. lets sort by ascending order of mean energy consumption by building."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby('building_id')['meter_reading'].mean().sort_values(ascending = False)[:10]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"AS we can see building 1099 has consumption in **million units** while others are in **10,000 units**. Lets see which meters are used in building 1099."},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train.building_id == 1099].groupby('meter_name')['meter_reading'].mean().plot(kind = 'bar', figsize = (6,4), color = 'green', title = 'Mean Energy consumption for Building 1099 by meter')\nprint('% Steam Consumption for Building 1099 out of total energy consumption is :',(train[train.building_id == 1099].groupby('meter_name')['meter_reading'].mean()[1] / train[train.building_id == 1099].groupby('meter_name')['meter_reading'].mean().sum())*100,'%')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Literally all of the energy consumed by building 1099 is steam."},{"metadata":{},"cell_type":"markdown","source":"### What is the mean % consumption of Steam for Building 1099 vs Steam consumption for all buildings? \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"steam_1099 = train[(train.building_id == 1099) & (train.meter == 2)]['meter_reading'].sum()\nsteam_others = train[(train.building_id != 1099) & (train.meter == 2)]['meter_reading'].sum()\nsteam_total = train[train.meter == 2]['meter_reading'].sum()\nprint('% of Steam Consumption for Building 1099 out of total for all buildings',100*steam_1099/steam_total)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Quite clearly Building 1099 is dominating the overall steam consumption.Hence it may be better to remove just Building 1099 rather than remove entire steam consumption to understand overall energy usage trend."},{"metadata":{},"cell_type":"markdown","source":"### How does Site 13 energy consumption look after removing building 1099\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize = (20,10))\ntrain[train.building_id != 1099][train.site_id == 13][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(title = 'Site 13 - No 1099 - Energy trend', label = 'By Hour')\ntrain[train.building_id != 1099][train.site_id == 13][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(label = 'By Day')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a more continuous trend for Site 13 without the outlier building 1099."},{"metadata":{},"cell_type":"markdown","source":"Lets filter building 1099 out to get a proper trend of mean energy consumption for all other buildings."},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train.building_id != 1099].groupby('building_id')['meter_reading'].mean().plot(figsize = (20,6), color = 'green', title = 'Mean Energy consumption in all buildings except Building 1099')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we are able to capture the trend of mean hourly energy consumption across buildings in a better way.\n\nGiven Steam meter distorts the whole picture of energy consumption so much, we can exclude outliers and relook at the energy trends."},{"metadata":{},"cell_type":"markdown","source":"### Energy consumption trend just for Steam meters"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15, 6), dpi=100)\ntrain[train.meter == 2][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain[train.meter == 2][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain[train.meter == 2][['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('Only Steam consumption : Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Energy consumption trend just for Steam meters - outlier Building 1099 removed"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15, 6), dpi=100)\ntrain[(train.meter == 2) & (train.building_id != 1099)][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain[(train.meter == 2) & (train.building_id != 1099)][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain[(train.meter == 2) & (train.building_id != 1099)][['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('Only Steam Consumption for All buildings except 1099: Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Overall Mean Energy Consumption - All Meters except Steam"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15, 6), dpi=100)\ntrain[train.meter != 2][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain[train.meter != 2][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain[train.meter != 2][['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('All meters except Steam : Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there is gradual increase in mean consumption with a peak during Sep to Oct and then back to original levels."},{"metadata":{},"cell_type":"markdown","source":"### Overall Mean Energy Consumption - All Meters - remove outlier Building 1099"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15, 6), dpi=100)\ntrain[train.building_id != 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('All Meters - All buildings except 1099 : Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This looks just the same as previous plot where we filtered out entire steam meters. So we have captured the true end of energy consumption by just leaving out one outlier building - 1099 - than filtering out all steam meters. \n\nLets see if the huge steam consumption for building 1099 happens in a specific time period.\n\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15, 6), dpi=100)\ntrain[train.building_id == 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(label='By hour', alpha=0.8, title = 'Energy consumption for Building 1099').set_ylabel('Meter reading', fontsize=14);\ntrain[train.building_id == 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(label='By Day', alpha=0.8, title = 'Energy consumption for Building 1099').set_ylabel('Meter reading', fontsize=14);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a huge spike in energy consumption for building 1099 between march and June mid. It drops to zero in June mid till Nov first week and again peaks briefly for just a week and comes to zero back again. \n\nIt would be interesting to see when building 1099 was constructed."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Year Built for Building 1099 is :',building_metadata[building_metadata.building_id == 1099].year_built.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Floor count for Building 1099 is :',building_metadata[building_metadata.building_id == 1099].floor_count.values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the year was not captured, may have been a pretty old building which must be using steam energy."},{"metadata":{},"cell_type":"markdown","source":"We can now revisit earlier plots by removing building 1099 and deduce the true energy consumption trend and compare against what it was when including Outlier"},{"metadata":{},"cell_type":"markdown","source":"# Review Plots post outlier "},{"metadata":{},"cell_type":"markdown","source":"## 1.1A Energy Consumption by Primary Use - Full Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.subplot(1,3,2)\ntrain.groupby('primary_use')['building_id'].nunique().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'purple', title = 'Count of Buildings by Primary use')\nplt.subplot(1,3,1)\ntrain.groupby('primary_use')['meter_reading'].sum().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Total Energy consumption by Primary use')\nplt.subplot(1,3,3)\ntrain.groupby('primary_use')['meter_reading'].mean().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Mean Energy consumption by Primary use')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.1B Energy Consumption by Primary Use - Remove Outlier Building 1099"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.subplot(1,3,2)\ntrain[train.building_id != 1099].groupby('primary_use')['building_id'].nunique().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'purple', title = 'Count of Buildings by Primary use')\nplt.subplot(1,3,1)\ntrain[train.building_id != 1099].groupby('primary_use')['meter_reading'].sum().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Total Energy consumption by Primary use')\nplt.subplot(1,3,3)\ntrain[train.building_id != 1099].groupby('primary_use')['meter_reading'].mean().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Mean Energy consumption by Primary use')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the outlier building 1099 was of Education Primary Use - removing that pushed Education from 1st to 5th highest energy consumer.Services becomes the highest mean enrergy consumer moving from 2nd to 1st position."},{"metadata":{},"cell_type":"markdown","source":"## 1.2A Energy Consumption by Primary Use by Meter Type - Original Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#train.groupby(['primary_use','meter'])['meter_reading'].mean().sort_values(ascending = False).reset_index().plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Mean Energy consumption by Primary use')\npivot_df = train.groupby(['primary_use','meter_name'])['meter_reading'].mean().sort_values(ascending = False).reset_index()\npivot_df.head()\npivot_df = pivot_df.pivot(index='primary_use', columns='meter_name', values='meter_reading')\npivot_df['AllMeters'] = pivot_df.sum(axis = 1)\npivot_df.sort_values('AllMeters', ascending = False).drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'Mean Hourly Energy consumption by Primary use by meter', stacked = True)\nplt.tight_layout()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.2B Energy Consumption by Primary Use by Meter Type - Outlier Building 1099 removed"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#train.groupby(['primary_use','meter'])['meter_reading'].mean().sort_values(ascending = False).reset_index().plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Mean Energy consumption by Primary use')\npivot_df = train[train.building_id != 1099].groupby(['primary_use','meter_name'])['meter_reading'].mean().sort_values(ascending = False).reset_index()\npivot_df.head()\npivot_df = pivot_df.pivot(index='primary_use', columns='meter_name', values='meter_reading')\npivot_df['AllMeters'] = pivot_df.sum(axis = 1)\npivot_df.sort_values('AllMeters', ascending = False).drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'No 1099: Mean Hourly Energy consumption by Primary use by meter', stacked = True)\nplt.tight_layout()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Same trend as in previous plot. Note types of buildings with Steam as energy tend to consume more energy."},{"metadata":{},"cell_type":"markdown","source":"## 2.1A Analyze Energy Consumption based on Meter used - With Original Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#plt.figure(figsize=(30,10))\nplt.subplot(1,3,2)\ntrain.groupby('meter_name')['building_id'].count().sort_values(ascending = False).plot(kind = 'bar', figsize = (15,6), color = 'purple', title = 'Count of Meter Type')\nplt.subplot(1,3,1)\ntrain.groupby('meter_name')['meter_reading'].sum().sort_values(ascending= False).plot(kind = 'bar', figsize = (15,6), color = 'green', title = 'Total Energy consumption by Meter Type' )\nplt.subplot(1,3,3)\ntrain.groupby('meter_name')['meter_reading'].mean().sort_values(ascending= False).plot(kind = 'bar', figsize = (15,6), color = 'green', title = 'Mean Energy consumption by Meter Type' )\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.1B Analyze Energy Consumption based on Meter used - Outlier Building 1099 removed"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#plt.figure(figsize=(30,10))\nplt.subplot(1,3,2)\ntrain[train.building_id != 1099].groupby('meter_name')['building_id'].count().sort_values(ascending = False).plot(kind = 'bar', figsize = (15,6), color = 'purple', title = 'Count of Meter Type')\nplt.subplot(1,3,1)\ntrain[train.building_id != 1099].groupby('meter_name')['meter_reading'].sum().sort_values(ascending= False).plot(kind = 'bar', figsize = (15,6), color = 'green', title = 'Total Energy consumption by Meter Type' )\nplt.subplot(1,3,3)\ntrain[train.building_id != 1099].groupby('meter_name')['meter_reading'].mean().sort_values(ascending= False).plot(kind = 'bar', figsize = (15,6), color = 'green', title = 'Mean Energy consumption by Meter Type' )\nplt.tight_layout()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No change in trend. Steam energy due to its scale exceeds other meter types even after removing building 1099. This is due to huge steam consumption by service type buildings. But note that chilled water is catching up with steam once the outlier was removed."},{"metadata":{},"cell_type":"markdown","source":"# Analyze Time Series Energy consumption\n## 3.1A Analyze mean energy consumption by time - by hours, days and months - Original Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15, 6), dpi=100)\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.1B Analyze mean energy consumption by time - by hours, days and months - Outlier Building 1099 removed"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15, 6), dpi=100)\ntrain[train.building_id != 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('No 1099: Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Removal of the outlier gave a clear continuous time series trend. Consumption falls from Jan to June continuously before peaking in Sep Oct and then returning to Jan levels."},{"metadata":{},"cell_type":"markdown","source":"## 3.2A Analyze mean energy consumption by time - by hours, days and months by 'meter type' - Original Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"meter_dict = {0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'}\n# Plot energy consumption by meter\nplt.figure(figsize = (20,10))\nfor i in range(4):\n plt.subplot(2,2,i+1)\n train[train.meter == i][['timestamp','meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(title = meter_dict[i], label = 'By Hour')\n train[train.meter == i][['timestamp','meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(title = meter_dict[i], label = 'By Day')\n train[train.meter == i][['timestamp','meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(title = meter_dict[i], label = 'By Month')\n plt.legend()\n plt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2B Analyze mean energy consumption by time - by hours, days and months by **'meter type'** - Outlier Building 1099 removed"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"meter_dict = {0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'}\n# Plot energy consumption by meter\nplt.figure(figsize = (20,10))\nfor i in range(4):\n plt.subplot(2,2,i+1)\n train[train.building_id != 1099][train.meter == i][['timestamp','meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(title = meter_dict[i], label = 'By Hour')\n train[train.building_id != 1099][train.meter == i][['timestamp','meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(title = meter_dict[i], label = 'By Day')\n train[train.building_id != 1099][train.meter == i][['timestamp','meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(title = meter_dict[i], label = 'By Month')\n #plt.title('No 1099: Energy consumption by time by meter type')\n plt.legend()\n plt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So once building 1099 was removed, the steam energy trend follows a more continuous pattern starting high at Jan, reducing gradually to a low between June to Sep and climbing up again back to jan levels by the end of the year. This is also the same trend for hotwater usage - kinda nice given steam and hotwater are close cousins:)"},{"metadata":{},"cell_type":"markdown","source":"## 3.3A Analyze Hourly Mean Energy Consumption by Site Id - Original Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.subplot(1,3,1)\ntrain.groupby('site_id')['meter_reading'].sum().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Total Energy consumption in sites 0 to 15')\nplt.subplot(1,3,2)\ntrain.groupby('site_id')['building_id'].nunique().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'purple', title = 'Count of Buildings in sites 0 to 15')\nplt.subplot(1,3,3)\ntrain.groupby('site_id')['meter_reading'].mean().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Mean Energy consumption in sites 0 to 15')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.3B Analyze Hourly Mean Energy Consumption by Site Id - Outlier Building 1099 removed"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.subplot(1,3,1)\ntrain[train.building_id != 1099].groupby('site_id')['meter_reading'].sum().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Total Energy consumption in sites 0 to 15')\nplt.subplot(1,3,2)\ntrain[train.building_id != 1099].groupby('site_id')['building_id'].nunique().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'purple', title = 'Count of Buildings in sites 0 to 15')\nplt.subplot(1,3,3)\ntrain[train.building_id != 1099].groupby('site_id')['meter_reading'].mean().sort_values(ascending = False).plot(kind = 'bar', figsize = (20,6), color = 'green', title = 'Mean Energy consumption in sites 0 to 15')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.4A Check Time Series Hourly and Daily energy consumption by Site Id - Original Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Plot energy consumption by site\nplt.figure(figsize = (20,20))\nfor i in range(16):\n plt.subplot(8,2,i+1)\n train[train.site_id == i][['timestamp','meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(title = 'Site {}'.format(i), label = 'By Hour')\n train[train.site_id == i][['timestamp','meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(title = 'Site {}'.format(i), label = 'By Day')\n plt.legend()\n plt.tight_layout() # Add tight layout in loop to prevent overlapping text\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.4B Check Time Series Hourly and Daily energy consumption by Site Id - Remove outlier building 1099"},{"metadata":{},"cell_type":"markdown","source":"Check Building 1099 belongs to which site"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata[building_metadata.building_id == 1099].site_id.unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is site 13. Lets check the consumption trend only for Site 13 after removing building 1099."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize = (20,6))\ntrain[(train.building_id != 1099) & (train.site_id == 13)][['timestamp','meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(title = 'Site 13'.format(i), label = 'By Hour')\ntrain[(train.building_id != 1099) & (train.site_id == 13)][['timestamp','meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(label = 'By Day')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Site 13 trend clearly changed meaning that the steam meter did not dominate after removing 1099 and other buildings have only other meter types."},{"metadata":{},"cell_type":"markdown","source":"## 3.5A Energy consumption by Site by meter type - Original Data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"pivot_df = train.groupby(['site_id','meter_name'])['meter_reading'].mean().sort_values(ascending = False).reset_index()\npivot_df.head()\npivot_df = pivot_df.pivot(index='site_id', columns='meter_name', values='meter_reading')\npivot_df['AllMeters'] = pivot_df.sum(axis = 1)\npivot_df.sort_values('AllMeters', ascending = False).drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'Mean Energy consumption by Site Id by meter', stacked = True)\nplt.tight_layout()\n\n\n## Show values in bar plot\n#ax = pivot_df.sort_values('AllMeters', ascending = False).drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'Mean Energy consumption by Site Id by meter', stacked = True)\n#for p in ax.patches:\n # ax.annotate(str(p.get_height()), (p.get_x() * 1.005, p.get_height() * 1.005))\n#plt.tight_layout() \n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.5B Energy consumption by Site by meter type - Outlier Building 1099 removed"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"pivot_df = train[train.building_id != 1099].groupby(['site_id','meter_name'])['meter_reading'].mean().sort_values(ascending = False).reset_index()\npivot_df.head()\npivot_df = pivot_df.pivot(index='site_id', columns='meter_name', values='meter_reading')\npivot_df['AllMeters'] = pivot_df.sum(axis = 1)\npivot_df.sort_values('AllMeters', ascending = False).drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'Mean Energy consumption by Site Id by meter', stacked = True)\nplt.tight_layout()\n\n\n## Show values in bar plot\n#ax = pivot_df.sort_values('AllMeters', ascending = False).drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'Mean Energy consumption by Site Id by meter', stacked = True)\n#for p in ax.patches:\n # ax.annotate(str(p.get_height()), (p.get_x() * 1.005, p.get_height() * 1.005))\n#plt.tight_layout() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Site 7 and Site 13 exchanged places as removed building 1099 belonged to site 13. This gives a much more concrete picture of which meters are used in which sites. After Steam chilled water plays a major role in large energy consumption."},{"metadata":{},"cell_type":"markdown","source":"# 4. Analyze Building data features"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We so far analyzed energy consumption based on buildings, sites, primary use of buildings, type of meter used and across the time scale.\n\nWe have a few more building attributes to analyze\n\n1. **square_feet** - Gross Area occupied by the building, more the area more the energy usage. This includes all of the floors of the building.\n2. **year_built **- year in which building was built. energy is started to get consumed by the building only after the year_built for a building. \n3. **floor_count** - Number of floors in the building. more the number of floors more the energy consumption.\n\n**A note on square_feet and floor_count :** Given that square_feet also includes the area for the total number of floors, total number of floors may perhaps not give additional information to the model. We might choose to ignore the floor count for building a prediction model.If we were to engineer a feature called square_feet / floor, it still cannot predict without also having information on number of floors."},{"metadata":{},"cell_type":"markdown","source":"Before we analyze these 3 features, lets check for missing values for these features."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Define function to report the missing data in a data frame\ndef report_missing_data(df):\n print('Total Number of rows :', len(df))\n for column in df.columns:\n print(column,':', 'Missing rows:', sum(df[column].isnull()), '|', '% Missing: {:.2f}'.format(sum(df[column].isnull())*100/len(df)),'%')\n\n # Report missing data for building data\nreport_missing_data(building_metadata)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have square_feet intact with no missing values while floor count is missing in 75% of buildings which is a huge gap. But as mentioned above, if we skip floor_count feature, we do not need to worry about the missing values as well. Since square feet does not have missing data, we are good with it for modeling.\n\nThe year_built feature has 50% missing values. we may not use this feature directly as it is not clear how it will inform the model. Moreover there is no easy way to impute the year_built values so it is quite hard to be able to use this feature at all for prediction. If at all, we plan to use this feature, it will be based on how many years have elapsed since year_built. We will explore this down below at some point.\n\nAnyway, lets visualize these features to get a quick perspective of these features by itself."},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Check the distribution of square_feet area for each primary_use"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize = (100,100))\ng = sns.FacetGrid(building_metadata, col = 'primary_use',col_wrap = 4)\ng.map(sns.distplot,'square_feet', kde = False, label = 'primary_use')\ng.set_xticklabels(rotation=90) # rotate all x-axis ticks for all facet subplots\nfor ax in g.axes.flat:\n plt.setp(ax.get_xticklabels(), visible=True) # Show x axis ticklabels for each facet sub plot\n plt.setp(ax.get_yticklabels(), visible=True) # Show y axis ticklabels for each facet sub plot\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The square_feet values are all skewed towards right which means more values at the lower ranges of square_feet. We might want to log transform these before we fit linear regression models."},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Check the distribution of floor_count area for each primary_use"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize = (100,100))\ng = sns.FacetGrid(building_metadata, col = 'primary_use',col_wrap = 4)\ng.map(sns.distplot,'floor_count', kde = False, label = 'primary_use')\ng.set_xticklabels(rotation=90) # rotate all x-axis ticks for all facet subplots\nfor ax in g.axes.flat:\n plt.setp(ax.get_xticklabels(), visible=True) # Show x axis ticklabels for each facet sub plot\n plt.setp(ax.get_yticklabels(), visible=True) # Show y axis ticklabels for each facet sub plot\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We do not have floor counts for Services, Food Sales and Service and Religious worship. Also Education, Lodging, office, entertainment, public services have more range of values for floor counts. At this moment, we are not considering this as a valuable feature for prediction."},{"metadata":{},"cell_type":"markdown","source":"## 4.3 Analyze Year in which buildings were built\nWe can visualize trends around year_built in multiple ways to see if we can consider year_built as a valuable feature.\n\n1. Trend of count of buildings by year - Increases as years progress\n2. Trend of energy consumption based on elapsed years - maybe more the elapsed years, more the energy consumed by buildings\n3. Trend of meter usage by years - number of each types of meters as years progress"},{"metadata":{},"cell_type":"markdown","source":"### 4.3.1 Trend of count of buildings by year - Increases as years progress"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize = (25,6))\ntrain.groupby('year_built')['building_id'].nunique().plot(kind = 'bar', color = 'green', title = 'Number of Buildings built by Year', rot = 90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The trend has been more cyclic with too many buildings built in year 1976 as compared to other years. However as we saw considering we have 50% missing values, we do not have sufficient info to conclude anything."},{"metadata":{},"cell_type":"markdown","source":"### 4.3.2 Trend of energy consumption based on elapsed years"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Plot energy consumption by elapsed years\ntrain.groupby('building_id')['elapsed_years','meter_reading'].mean().sort_values('elapsed_years').plot(x = 'elapsed_years',y = 'meter_reading', figsize = (20,6))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No clear trend that elapsed years of buildings are correlated with energy consumption in a meaningful way. We can can ignore this feature for building models."},{"metadata":{},"cell_type":"markdown","source":"### 4.3.3 Trend of meter usage by years - number of each types of meters as years progress"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"pivot_df = train.groupby(['year_built','meter_name'])['meter_reading'].mean().reset_index()\npivot_df.head()\npivot_df = pivot_df.pivot(index='year_built', columns='meter_name', values='meter_reading')\npivot_df.plot(kind = 'bar', figsize = (20,5), colormap = 'Set1', title = 'Mean Energy consumption by Site Id by meter', stacked = True)\n\n#pivot_df['AllMeters'] = pivot_df.sum(axis = 1)\n#pivot_df.drop('AllMeters',axis = 1).plot(kind = 'bar', figsize = (20,10), colormap = 'Set1', title = 'Mean Energy consumption by Site Id by meter', stacked = True)\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Steam and chilled water pretty much dominate in all the years and chilled water proportion has been growing considerably over the last 20 years edging out steam in atleast last 15 years.\n\nIt would be interesting to look at this trend against the square feet area of each building"},{"metadata":{},"cell_type":"markdown","source":"### 4.3.4 Trend of meter usage by building square feet"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train[['year_built', 'square_feet']].groupby('year_built').mean().plot(kind = 'bar',figsize = (20,5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It can be seen that the spikes in mean energy consumption in previous plot is not necessarily due to square_feet area as much as it is due to meter type. We can compare plots 4.3.3 and 4.3.4 for below years:\n\n1. 1922 - Sq.ft is very high but energy consumption is pretty low - due to electricity meter being used almost fully.\n2. 1930 - lower Sq.Ft but higher higher energy consumption as steam being used 40% roughly.\n3. 1981 - higher energy for lower sq.ft due to chilled water usage.\n\nFor other years like 1938, 1952, 1979 etc. sq.ft correlates with energy consumption only because steam or chilled water is used.\n\nThe importance of meter type type is probably more than sq.ft feature."},{"metadata":{},"cell_type":"markdown","source":"In conclusion for year_built trend, key takeaway is that chilled water usage is more in last 20 years. year_built by itself does not seem to be a very critical feature. Square feet is correlated, but is overshadowed by meter type. "},{"metadata":{},"cell_type":"markdown","source":"# 5. Analyze Weather Features for Train set"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('col index for air_temperature is :', train.columns.get_loc('air_temperature'))\nprint('col index for wind_speed is :', train.columns.get_loc('wind_speed'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.iloc[:,9:16].info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have 7 features from air_temperature to wind_speed. All of these are at site level and hence will impact buildings based on which site they are located in. Also all seem to be numeric.\n\nBefore we dive into these features, let us remind ourselves how 1449 buildings are allocated to the 16 sites."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sns.set_style('whitegrid')\nsite_building = building_metadata.groupby('site_id')['building_id'].nunique().sort_values(ascending=False).plot(kind = 'bar', figsize = (10,3), color = 'skyblue')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Site 3 has the highest number of buildings twice that of the second highest site which is 13. \n\n75% of the sites have atleast 50 buildings each."},{"metadata":{},"cell_type":"markdown","source":"### Understand Weather data summary stats"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Understand summary stats\nweather_train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.1 Weather Features - Univariate Plots"},{"metadata":{},"cell_type":"markdown","source":"#### Weather Features - Histograms"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Select only weather data columns from final train set\ncolumns = train.columns[9:16]\nprint(columns)\n\n# Create a function to draw histograms for weather columns\ndef plot_columns(df,columns,plot_type = 'hist'):\n plt.figure(figsize=(20,6))\n for i,column in enumerate(columns):\n plt.subplot(3,3,i+1)\n df[column].plot(kind = plot_type, label = column)\n plt.legend(frameon = False)\n plt.tight_layout() \n plt.show() \n\nplot_columns(train,columns) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Weather Features - Box Plots"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Select only weather data columns from final train set\ncolumns = train.columns[9:16]\nprint(columns)\n\n# Create a function to draw histograms for weather columns\ndef plot_columns(df,columns,plot_type = 'hist'):\n plt.figure(figsize=(20,6))\n for i,column in enumerate(columns):\n plt.subplot(3,3,i+1)\n sns.boxplot(x = df[column])\n plt.legend(frameon = False)\n plt.tight_layout() \n plt.show() \n\nplot_columns(train,columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Cloud_Coverage"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.sort(train.iloc[:,9:16].cloud_coverage.unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that cloud coverage feature seems to be categorical and not float as shown in data. https://en.wikipedia.org/wiki/Okta - This link shows that cloud coverage is measured from 0 to 9 levels.So we need to convert cloud coverage to categorical. Also we can see that 75% of the values are between 0 and 4."},{"metadata":{},"cell_type":"markdown","source":"#### Precip_depth_1_hr\nNow let us look at the feature : **precip_depth_1_hr** - From the .describe() stats, its clear that about 75% of the values are 0 and hence the single bar in the histogram. Also the min value is -1. Lets do a log transform to see the actual shape of the histogram"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.log(train.precip_depth_1_hr+1.001).plot(kind = 'hist') # usually we add 1 to number to be log transformed to ensure input to log is positive. here since w have -1 as min value, we need to add more than 1.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inspite of log transform we see a very high skew especially with a value of 0. With this kind of data, this feature may not inform the model much.We might ignore this feature while building models."},{"metadata":{},"cell_type":"markdown","source":"#### Wind Speed\nWe can see that wind direction is having a uniform distribution. I guess we can just use this feature as is. "},{"metadata":{},"cell_type":"markdown","source":"#### Other weather features:\nAs regards other weather features - **Air Temp, Dew Temp,Sea level pressure and wind speed**, they have minor skews and we can use the **log transform** of these features for building models."},{"metadata":{},"cell_type":"markdown","source":"## 5.2 Correlation of weather features with mean energy consumption\nWe can look for strength of relationship between the weather features and mean energy consumption by using a heat map. We can check correlation at different aggregations of data as this is time series data."},{"metadata":{},"cell_type":"markdown","source":"We cannot determine correlation across the entire data set in a time series data as there will be upward and downward movements in different time periods. It makes sense to break the overall time scale into 3-4 buckets where there is a consitent trend and check correlations only within that bucket so determine if weather features have strong impact on energy consumption.\n\nLet us review the earlier energy consumption plotted against time by different intervals, hourly, daily and monthly - **with outlier building 1099 removed to show the true trend.**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15, 6), dpi=100)\ntrain[train.building_id != 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('No 1099: Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Considering the **energy consumption pattern** is having a distinct pattern in different time periods of the year, We can split the time series data into 4 chunks - \n\n1. **Jan to April - Downward trend**\n2. **May to Aug - Upward trend**\n3. **Sep to Oct - Downward trend**\n4. **Nov to Dec - slight upward trend**\n\nWe will also visualize the weather parameters in these time periods so that we can connect them to energy consumption trend across the year."},{"metadata":{},"cell_type":"markdown","source":"### Weather parameters over time in the same period"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize=(15, 6), dpi=100)\ntrain[train.building_id != 1099][['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes[0,0], label='By hour', alpha=0.8)#.set_ylabel('Air Temperature', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes[0,0], label='By day', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'air_temperature']].set_index('timestamp').resample('M').mean()['air_temperature'].plot(ax=axes[0,0], label='By Month', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\naxes[0,0].set_title('Air Temperature by hour, day and month', fontsize=16);\naxes[0,0].legend();\n\ntrain[train.building_id != 1099][['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes[0,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes[0,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'dew_temperature']].set_index('timestamp').resample('M').mean()['dew_temperature'].plot(ax=axes[0,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[0,1].set_title('Dew Temperature by hour, day and month', fontsize=16);\naxes[0,1].legend();\n\ntrain[train.building_id != 1099][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('M').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,0].set_title('Sea level pressure by hour, day and month', fontsize=16);\naxes[1,0].legend();\n\ntrain[train.building_id != 1099][['timestamp', 'wind_speed']].set_index('timestamp').resample('H').mean()['wind_speed'].plot(ax=axes[1,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'wind_speed']].set_index('timestamp').resample('D').mean()['wind_speed'].plot(ax=axes[1,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain[train.building_id != 1099][['timestamp', 'wind_speed']].set_index('timestamp').resample('M').mean()['wind_speed'].plot(ax=axes[1,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,1].set_title('Wind Speed by hour, day and month', fontsize=16);\naxes[1,1].legend();\n\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that Air temperature and Dew temperature are highly positively correlated. Sea level pressure and wind speed are in the opposite direction of Air temperature and dew temperature and hence negatively correlated though the variation is in a small range around the mean unlike air temperature and dew temperature. \n\nSo we can consider one of air temperature or dew temperature to be more important features than sea level pressure and wind speed."},{"metadata":{},"cell_type":"markdown","source":"### Break Time Series into 4 buckets "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split whole time series data into 4 different periods of the year with a consistent energy consumption trend\ntrain_jan_to_apr = train[train.building_id != 1099][train[train.building_id != 1099].month.isin([1,2,3,4])]\ntrain_may_to_aug = train[train.building_id != 1099][train[train.building_id != 1099].month.isin([5,6,7,8])]\ntrain_sep_to_oct = train[train.building_id != 1099][train[train.building_id != 1099].month.isin([9,10])]\ntrain_nov_to_dec = train[train.building_id != 1099][train[train.building_id != 1099].month.isin([11,12])]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can draw the heatmap correlation plot within each of these periods to see if the weather in the period is correlated with the energy trend in that period."},{"metadata":{},"cell_type":"markdown","source":"### 5.2.1A Jan to Apr - Energy Consumption Trend - Decreasing"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15,3), dpi=100)\ntrain_jan_to_apr[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('No 1099: Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Energy consumption Correlation with Hourly Data: This captures hourly variations in weather which is probably realistic. But there is too much noise in hourly energy consumption that we do not see any sustained correlation. \n\nEnergy consumption Correlation with Daily Data: Daily frequency smoothens the noise in measurements of weather as well as meter readings. \n\nWe will review the weather pattern and energy consumption based on **daily averaged values**."},{"metadata":{},"cell_type":"markdown","source":"### 5.2.1B Jan to Apr - Correlation Heat Map for Energy usage and weather features"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Plot a heatmap between mean daily consumption and the weather features for the period Jan to Apr\n\n# Create Correlation matrix data frame to be plotted as heatmap - subsetting data for Jan to Apr\ntrain_jan_to_apr_corr_day = train_jan_to_apr.groupby('day')['meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ].mean().reset_index().corr()\ntrain_jan_to_apr_corr_hour = train_jan_to_apr[['hour','meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ]].corr()\ntrain_jan_to_apr_corr_month = train_jan_to_apr.groupby('month')['meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ].mean().reset_index().corr()\n\n# Create heatmap\nplt.figure(figsize=(20,6))\n#mask = np.zeros_like(train_corr)\n#mask[np.tril_indices_from(mask)] = True\nplt.subplot(1,3,1, title = 'Jan to Apr Hourly energy consumption vs Weather') # Add titles for individual subplots\nsns.heatmap(train_jan_to_apr_corr_hour,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\nplt.subplot(1,3,2, title = 'Jan to Apr Daily energy consumption vs Weather')\nsns.heatmap(train_jan_to_apr_corr_day,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\n#plt.subplot(1,3,3, title = 'Jan to Apr Monthly energy consumption vs Weather')\n#sns.heatmap(train_jan_to_apr_corr_month,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Energy consumption has reasonable negative correlation with Air temperature, Dew temperature and positive correlation with sea level pressure. Also Air/Dew Temperature are positively correlated with each other and negatively correlated with sea level pressure. "},{"metadata":{},"cell_type":"markdown","source":"### 5.2.1C Jan to Apr - Weather Pattern"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize=(15, 6), dpi=100)\ntrain_jan_to_apr[['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes[0,0], label='By hour', alpha=0.8)#.set_ylabel('Air Temperature', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes[0,0], label='By day', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'air_temperature']].set_index('timestamp').resample('M').mean()['air_temperature'].plot(ax=axes[0,0], label='By Month', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\naxes[0,0].set_title('Air Temperature by hour, day and month', fontsize=16);\naxes[0,0].legend();\n\ntrain_jan_to_apr[['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes[0,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes[0,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'dew_temperature']].set_index('timestamp').resample('M').mean()['dew_temperature'].plot(ax=axes[0,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[0,1].set_title('Dew Temperature by hour, day and month', fontsize=16);\naxes[0,1].legend();\n\ntrain_jan_to_apr[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('M').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,0].set_title('Sea level pressure by hour, day and month', fontsize=16);\naxes[1,0].legend();\n\ntrain_jan_to_apr[['timestamp', 'wind_speed']].set_index('timestamp').resample('H').mean()['wind_speed'].plot(ax=axes[1,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'wind_speed']].set_index('timestamp').resample('D').mean()['wind_speed'].plot(ax=axes[1,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_jan_to_apr[['timestamp', 'wind_speed']].set_index('timestamp').resample('M').mean()['wind_speed'].plot(ax=axes[1,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,1].set_title('Wind Speed by hour, day and month', fontsize=16);\naxes[1,1].legend();\n\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the Air Temp and Dew Temp are rising continuously from Jan to Apr. Sea level pressure dips a little bit. Wind speed remained almost steady with some variation around the mean.\n\nThis is consistent with the observation that the energy usage is negatively correlated with air/dew temp and positively correlated with sea level pressure."},{"metadata":{},"cell_type":"markdown","source":"### 5.2.2A May to Aug - Energy Consumption Trend - Increasing"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15,3), dpi=100)\ntrain_may_to_aug[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain_may_to_aug[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain_may_to_aug[['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('No 1099: Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2.2B May to Aug - Correlation Heat Map for Enery usage and weather features"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Create Correlation matrix data frame to be plotted as heatmap - subsetting data for May to Aug\ntrain_may_to_aug_corr_day = train_may_to_aug.groupby('day')['meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ].mean().reset_index().corr()\ntrain_may_to_aug_corr_hour = train_may_to_aug[['hour','meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ]].corr()\ntrain_may_to_aug_corr_month = train_may_to_aug.groupby('month')['meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ].mean().reset_index().corr()\n\n## Create heatmap - May to Aug\nplt.figure(figsize=(20,6))\n#mask = np.zeros_like(train_corr)\n#mask[np.tril_indices_from(mask)] = True\nplt.subplot(1,3,1, title = 'May to Aug Hourly energy consumption vs Weather') # Add titles for individual subplots\nsns.heatmap(train_may_to_aug_corr_hour,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\nplt.subplot(1,3,2, title = 'May to Aug Daily energy consumption vs Weather')\nsns.heatmap(train_may_to_aug_corr_day,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\n#plt.subplot(1,3,3, title = 'May to Aug Monthly energy consumption vs Weather')\n#sns.heatmap(train_may_to_aug_corr_month,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Energy consumption has a positive correlation with air/dew temperature and sea level pressure. Air/Dew Temperature are positively correlated with each other. **wind speed** is **negatively correlated** with **air/dew temperature and sea level pressure**."},{"metadata":{},"cell_type":"markdown","source":"### 5.2.2C May to Aug - Weather Pattern"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize=(15, 6), dpi=100)\ntrain_may_to_aug[['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes[0,0], label='By hour', alpha=0.8)#.set_ylabel('Air Temperature', fontsize=14);\ntrain_may_to_aug[['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes[0,0], label='By day', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\ntrain_may_to_aug[['timestamp', 'air_temperature']].set_index('timestamp').resample('M').mean()['air_temperature'].plot(ax=axes[0,0], label='By Month', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\naxes[0,0].set_title('Air Temperature by hour, day and month', fontsize=16);\naxes[0,0].legend();\n\ntrain_may_to_aug[['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes[0,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_may_to_aug[['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes[0,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_may_to_aug[['timestamp', 'dew_temperature']].set_index('timestamp').resample('M').mean()['dew_temperature'].plot(ax=axes[0,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[0,1].set_title('Dew Temperature by hour, day and month', fontsize=16);\naxes[0,1].legend();\n\ntrain_may_to_aug[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_may_to_aug[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_may_to_aug[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('M').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,0].set_title('Sea level pressure by hour, day and month', fontsize=16);\naxes[1,0].legend();\n\ntrain_may_to_aug[['timestamp', 'wind_speed']].set_index('timestamp').resample('H').mean()['wind_speed'].plot(ax=axes[1,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_may_to_aug[['timestamp', 'wind_speed']].set_index('timestamp').resample('D').mean()['wind_speed'].plot(ax=axes[1,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_may_to_aug[['timestamp', 'wind_speed']].set_index('timestamp').resample('M').mean()['wind_speed'].plot(ax=axes[1,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,1].set_title('Wind Speed by hour, day and month', fontsize=16);\naxes[1,1].legend();\n\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that Air/Dew Temp and sea level pressure are increasing throughout while wind speed is decreasing slightly."},{"metadata":{},"cell_type":"markdown","source":"### 5.2.3A Sep to Oct - Energy consumption trend - Decreasing"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15,3), dpi=100)\ntrain_sep_to_oct[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('No 1099: Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2.3B Sep to Oct - Correlation Heat Map for Enery usage and weather features"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Create Correlation matrix data frame to be plotted as heatmap - subsetting data for Sep to Oct\ntrain_sep_to_oct_corr_day = train_sep_to_oct.groupby('day')['meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ].mean().reset_index().corr()\ntrain_sep_to_oct_corr_hour = train_sep_to_oct[['hour','meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ]].corr()\ntrain_sep_to_oct_corr_month = train_sep_to_oct.groupby('month')['meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ].mean().reset_index().corr()\n\n## Create heatmap - Sep to Oct\nplt.figure(figsize=(20,6))\n#mask = np.zeros_like(train_corr)\n#mask[np.tril_indices_from(mask)] = True\nplt.subplot(1,3,1, title = 'Sep to Oct Hourly energy consumption vs Weather') # Add titles for individual subplots\nsns.heatmap(train_sep_to_oct_corr_hour,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\nplt.subplot(1,3,2, title = 'Sep to Oct Daily energy consumption vs Weather')\nsns.heatmap(train_sep_to_oct_corr_day,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\n#plt.subplot(1,3,3, title = 'Sep to Oct Monthly energy consumption vs Weather')\n#sns.heatmap(train_sep_to_oct_corr_month,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Energy consumption is **negatively correlated** with weather features. Air / Dew Temp are negatively correlated with sea level pressure and wind speed. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize=(15, 6), dpi=100)\ntrain_sep_to_oct[['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes[0,0], label='By hour', alpha=0.8)#.set_ylabel('Air Temperature', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes[0,0], label='By day', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'air_temperature']].set_index('timestamp').resample('M').mean()['air_temperature'].plot(ax=axes[0,0], label='By Month', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\naxes[0,0].set_title('Air Temperature by hour, day and month', fontsize=16);\naxes[0,0].legend();\n\ntrain_sep_to_oct[['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes[0,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes[0,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'dew_temperature']].set_index('timestamp').resample('M').mean()['dew_temperature'].plot(ax=axes[0,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[0,1].set_title('Dew Temperature by hour, day and month', fontsize=16);\naxes[0,1].legend();\n\ntrain_sep_to_oct[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('M').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,0].set_title('Sea level pressure by hour, day and month', fontsize=16);\naxes[1,0].legend();\n\ntrain_sep_to_oct[['timestamp', 'wind_speed']].set_index('timestamp').resample('H').mean()['wind_speed'].plot(ax=axes[1,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'wind_speed']].set_index('timestamp').resample('D').mean()['wind_speed'].plot(ax=axes[1,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_sep_to_oct[['timestamp', 'wind_speed']].set_index('timestamp').resample('M').mean()['wind_speed'].plot(ax=axes[1,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,1].set_title('Wind Speed by hour, day and month', fontsize=16);\naxes[1,1].legend();\n\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Air / Dew Temp are falling while sea level pressure is slightly increasing. Wind speed remains more or less constant."},{"metadata":{},"cell_type":"markdown","source":"### 5.2.4A Nov to Dec - Energy consumption trend - Increasing"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(15,3), dpi=100)\ntrain_nov_to_dec[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes, label='By Month', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('No 1099: Mean Meter reading by hour, day and month', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2.4B Nov to Dec - Correlation Heat Map for Enery usage and weather features"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Create Correlation matrix data frame to be plotted as heatmap - subsetting data for Nov to Dec\ntrain_nov_to_dec_corr_day = train_nov_to_dec.groupby('day')['meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ].mean().reset_index().corr()\ntrain_nov_to_dec_corr_hour = train_nov_to_dec[['hour','meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ]].corr()\ntrain_nov_to_dec_corr_month = train_nov_to_dec.groupby('month')['meter_reading', 'air_temperature', 'dew_temperature', 'sea_level_pressure', 'wind_speed' ].mean().reset_index().corr()\n\n## Create heatmap - Nov to Dec\nplt.figure(figsize=(20,6))\n#mask = np.zeros_like(train_corr)\n#mask[np.tril_indices_from(mask)] = True\nplt.subplot(1,3,1, title = 'Nov to Dec Hourly energy consumption vs Weather') # Add titles for individual subplots\nsns.heatmap(train_nov_to_dec_corr_hour,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\nplt.subplot(1,3,2, title = 'Nov to Dec Daily energy consumption vs Weather')\nsns.heatmap(train_nov_to_dec_corr_day,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\n#plt.subplot(1,3,3, title = 'Nov to Dec Monthly energy consumption vs Weather')\n#sns.heatmap(train_nov_to_dec_corr_month,annot=True, cmap=sns.color_palette(\"BrBG\", 7), center = 0) # did not use masking\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Energy consumption is negatively correlated with Air / Dew Temp which are also negatively correlated with wind speed. "},{"metadata":{},"cell_type":"markdown","source":"### 5.2.4C Nov to Dec - Weather Pattern"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize=(15, 6), dpi=100)\ntrain_nov_to_dec[['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes[0,0], label='By hour', alpha=0.8)#.set_ylabel('Air Temperature', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes[0,0], label='By day', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'air_temperature']].set_index('timestamp').resample('M').mean()['air_temperature'].plot(ax=axes[0,0], label='By Month', alpha=1)#.set_ylabel('Air Temperature', fontsize=14);\naxes[0,0].set_title('Air Temperature by hour, day and month', fontsize=16);\naxes[0,0].legend();\n\ntrain_nov_to_dec[['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes[0,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes[0,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'dew_temperature']].set_index('timestamp').resample('M').mean()['dew_temperature'].plot(ax=axes[0,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[0,1].set_title('Dew Temperature by hour, day and month', fontsize=16);\naxes[0,1].legend();\n\ntrain_nov_to_dec[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('M').mean()['sea_level_pressure'].plot(ax=axes[1,0], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,0].set_title('Sea level pressure by hour, day and month', fontsize=16);\naxes[1,0].legend();\n\ntrain_nov_to_dec[['timestamp', 'wind_speed']].set_index('timestamp').resample('H').mean()['wind_speed'].plot(ax=axes[1,1], label='By hour', alpha=0.8)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'wind_speed']].set_index('timestamp').resample('D').mean()['wind_speed'].plot(ax=axes[1,1], label='By day', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\ntrain_nov_to_dec[['timestamp', 'wind_speed']].set_index('timestamp').resample('M').mean()['wind_speed'].plot(ax=axes[1,1], label='By Month', alpha=1)#.set_ylabel('dew_temperature', fontsize=14);\naxes[1,1].set_title('Wind Speed by hour, day and month', fontsize=16);\naxes[1,1].legend();\n\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n**In Summary: **\n\n1. **Energy usage is in some periods positively correlated and in some periods negatively correlated** with **air / dew temperature**\n2. **Sea level pressure and Wind speed **have mostly **negative correlation** with **air / dew temperature**\n3. **Air and Dew Temperature** have significant **positive correlation with each other** and we may even use only one of the two can be used for building a model. \n"},{"metadata":{},"cell_type":"markdown","source":"## Weather Pattern across sites"},{"metadata":{},"cell_type":"markdown","source":"### Air Temperature by Site across the year"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_air_temp_sites(df):\n fig, axes = plt.subplots(8,2, figsize=(15, 15), dpi=100)\n for i in range(16):\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes[(i%8),(i//8)], label='By hour', alpha=0.8).set_ylabel('Temp in Deg C ', fontsize=10);\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes[(i%8),(i//8)], label='By day', alpha=1).set_ylabel('Temp in Deg C', fontsize=10);\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'air_temperature']].set_index('timestamp').resample('M').mean()['air_temperature'].plot(ax=axes[(i%8),(i//8)], label='By Month', alpha=1).set_ylabel('Temp in Deg C', fontsize=10);\n axes[(i%8),(i//8)].set_title('site {}'.format(i), fontsize=10);\n axes[(i%8),(i//8)].legend(frameon = False, ncol = 3);\n fig.suptitle('Air Temperature by Site across Time', y = 1) # y positions the super title\n plt.tight_layout() \nplot_air_temp_sites(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dew Temperature by Site across the year"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_dew_temp_sites(df):\n fig, axes = plt.subplots(8,2, figsize=(15, 15), dpi=100)\n for i in range(16):\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes[(i%8),(i//8)], label='By hour', alpha=0.8).set_ylabel('Temp in Deg C ', fontsize=10);\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes[(i%8),(i//8)], label='By day', alpha=1).set_ylabel('Temp in Deg C', fontsize=10);\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'dew_temperature']].set_index('timestamp').resample('M').mean()['dew_temperature'].plot(ax=axes[(i%8),(i//8)], label='By Month', alpha=1).set_ylabel('Temp in Deg C', fontsize=10);\n axes[(i%8),(i//8)].set_title('site {}'.format(i), fontsize=10);\n axes[(i%8),(i//8)].legend(frameon = False, ncol = 3);\n fig.suptitle('Dew Temperature by Site across Time', y = 1) # y positions the super title\n plt.tight_layout() \nplot_dew_temp_sites(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Sea Level Pressure pattern across time by Site"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_sea_pressure_sites(df):\n fig, axes = plt.subplots(8,2, figsize=(15, 15), dpi=100)\n for i in range(16):\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes[(i%8),(i//8)], label='By hour', alpha=0.8).set_ylabel('Pressure', fontsize=10);\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes[(i%8),(i//8)], label='By day', alpha=1).set_ylabel('Pressure', fontsize=10);\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('M').mean()['sea_level_pressure'].plot(ax=axes[(i%8),(i//8)], label='By Month', alpha=1).set_ylabel('Pressure', fontsize=10);\n axes[(i%8),(i//8)].set_title('site {}'.format(i), fontsize=10);\n axes[(i%8),(i//8)].legend(frameon = False, ncol = 3);\n fig.suptitle('Sea Level Pressure by Site across Time', y = 1) # y positions the super title\n plt.tight_layout() \nplot_sea_pressure_sites(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Wind speed pattern across time by Site"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_wind_speed_sites(df):\n fig, axes = plt.subplots(8,2, figsize=(15, 15), dpi=100)\n for i in range(16):\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'wind_speed']].set_index('timestamp').resample('H').mean()['wind_speed'].plot(ax=axes[(i%8),(i//8)], label='By hour', alpha=0.8).set_ylabel('Meters per sec', fontsize=10);\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'wind_speed']].set_index('timestamp').resample('D').mean()['wind_speed'].plot(ax=axes[(i%8),(i//8)], label='By day', alpha=1).set_ylabel('Meters per sec', fontsize=10);\n df[(df.building_id != 1099) & (df.site_id == i)][['timestamp', 'wind_speed']].set_index('timestamp').resample('M').mean()['wind_speed'].plot(ax=axes[(i%8),(i//8)], label='By Month', alpha=1).set_ylabel('Meters per sec', fontsize=10);\n axes[(i%8),(i//8)].set_title('site {}'.format(i), fontsize=10);\n axes[(i%8),(i//8)].legend(frameon = False, ncol = 3);\n fig.suptitle('Wind Speed by Site across Time', y = 1) # y positions the super title\n plt.tight_layout() \nplot_wind_speed_sites(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove outlier building 1099 from train to use in all data analyses\ntrain_no_outlier = train[train.building_id != 1099]\ntrain_no_outlier.set_index('timestamp', inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Energy Consumption by Meter with Daily Weather trend across the year "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Create a dataframe with meter reading, meter type, time stamp and weather parameters.\ntrain_meter_time = train_no_outlier[['meter_reading','air_temperature','dew_temperature', 'sea_level_pressure','wind_speed', 'meter_name']].groupby('meter_name')['meter_reading','air_temperature','dew_temperature', 'sea_level_pressure','wind_speed'].resample('D').mean()\ntrain_meter_time.reset_index(inplace = True)\ntrain_meter_time.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sns.pairplot(train_meter_time, hue = 'meter_name', markers = ['o','s','D','^'], diag_kws=dict(shade=False))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at how energy consumption by meter type varied with weather parameters - we can see that \n\n1. As Air/Dew Temperature increased, steam and hotwater consumption came down while chilled water consumption increased. Electricity did not show much of a trend.\n2. Sea Level Pressure and Wind speed did not have a clear correlation for any of the meters."},{"metadata":{},"cell_type":"markdown","source":"### Daily Energy consumption and weather features"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_no_outlier[['cloud_coverage', 'meter_reading','air_temperature', 'dew_temperature', 'sea_level_pressure','wind_speed']].resample('D').mean().plot(figsize = (20,15), subplots = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No significant correlation between energy usage and cloud coverage"},{"metadata":{},"cell_type":"markdown","source":"### Energy consumption and Weather variation by time of the day and month of the year"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_no_outlier[['meter_reading','hour','month', 'air_temperature','dew_temperature', 'sea_level_pressure','wind_speed', 'cloud_coverage']].groupby('hour')['meter_reading','air_temperature','dew_temperature', 'sea_level_pressure','wind_speed'].mean().plot(figsize = (10,10), subplots = True, title = 'By Hour')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Energy consumption increases during the day peaking at noon and finally coming down by night time. This is as expected. \n\nAir temp seems to go down during the day time and increase during the night time which is not making sense. Dew Temp though is behaving as expected - increasing from start to peak at noon and reducing to a low by end of day.\n\nSea level pressure and Dew temperature seem to be following a similar pattern within a day \n\nwind speed and Air temperature seem to be following a similar pattern within a day "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_no_outlier[['meter_reading','hour','month', 'air_temperature','dew_temperature', 'sea_level_pressure','wind_speed']].groupby('month')['meter_reading','air_temperature','dew_temperature', 'sea_level_pressure','wind_speed'].mean().plot(figsize = (10,10), subplots = True, title = 'By Month')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":""},{"metadata":{},"cell_type":"markdown","source":"### Analyzing pair plot of energy usage and other weather features by cloud coverage"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Create a dataframe with meter reading, meter type, time stamp and weather parameters.\ntrain_meter_cloud = train_no_outlier[['meter_reading','air_temperature','dew_temperature', 'sea_level_pressure','wind_speed', 'cloud_coverage']].groupby('cloud_coverage')['meter_reading','air_temperature','dew_temperature', 'sea_level_pressure','wind_speed'].resample('D').mean()\ntrain_meter_cloud.reset_index(inplace = True)\ntrain_meter_cloud.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sns.pairplot(train_meter_cloud, hue = 'cloud_coverage')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"No clear pattern between meter reading and cloud coverage"},{"metadata":{},"cell_type":"markdown","source":"### Coming up\n\nI will start setting up some baseline prediction models and accuracy to get on the leaderboard and start climbing from there."},{"metadata":{},"cell_type":"markdown","source":"Do not hestitate to Upvote if you like :) . Leave a comment for improvement, i will really appreciate and incorporate!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}