{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **The Great Energy Predicator - EDA**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The American Society of Heating, Refrigerating and Air-Conditioning Engineers was established more than 125 ago. It is one of the leading forces in advanced heating, ventilation, air conditioning and refrigeration (HVAC&R) systems and it is operating in 132 countries worldwide. ASHRAE funds multiple research projects to improve energy efficency, technical standards, sustainable development etc. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# **Table of content**\n\n* [Introduction](#1)\n* [Initial data preperation](#2)\n* [Missing data check](#3)\n* [Initial outlier detection](#4)\n* [EDA](#5)\n* [Square feet and EE](#6)\n* [Linear correlations](#7)\n* [Categorical differences in energy usage](#8)\n* [Building data analysis](#9)\n* [Temperature data analysis](#10)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"> </a>\n# **Introduction**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The purpose of this exploratory data analysis is to provide some interesting insights and analyse the effect different variables have on energy consumption.  My personal goal is to use this notebook to further improve my data visualization skills and experiment with some additional plotting options.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a>\n# **Initial data preperation**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport missingno as msno\nimport datetime as dt\nimport gc\n\nplt.style.use('fivethirtyeight')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below is a very useful memory reduction script with a link to the original notebook. It reduces the memory of the CSV files by altering the data type for each column. I highly suggest you keep it in mind when running larger datasets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Memory reduction script\n#https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\ndef reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.        \n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n    \n    for col in df.columns:\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype('category')\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df\n\n\ndef import_data(file):\n    \"\"\"create a dataframe and optimize its memory usage\"\"\"\n    df = pd.read_csv(file, parse_dates=True, keep_date_col=True)\n    df = reduce_mem_usage(df)\n    return df\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Importing of files and the reduction of memory usage\nprint('-' * 80)\nprint('train')\ntrain = import_data(\"../input/ashrae-energy-prediction/train.csv\")\n\nprint('-' * 80)\nprint(\"building_metadata\")\nbuilding_metadata = import_data(\"../input/ashrae-energy-prediction/building_metadata.csv\")\n\nprint('-' * 80)\nprint(\"weather_train\")\nweather_train = import_data(\"../input/ashrae-energy-prediction/weather_train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = train.merge(building_metadata, on=\"building_id\", how=\"left\")\ndf_train = df_train.merge(weather_train, on=[\"site_id\",\"timestamp\"], how=\"left\")\n\ndf_train = df_train.sort_values(by=[\"timestamp\"])\ndf_train[\"timestamp\"] = pd.to_datetime(df_train[\"timestamp\"].astype(\"object\"))\n\ndf_train = df_train.set_index(df_train[\"timestamp\"])\ndf_train = df_train.drop([\"timestamp\"],axis=1)\n\n\n#Rename for easier ploting\nmetermap = {0: \"Electricity\", 1: \"Chilledwater\", 2: \"Steam\", 3: \"Hotwater\"}\ndf_train[\"meter\"].replace(metermap,inplace=True)\n\n\ngc.collect()\n\ndel building_metadata,weather_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a>\n\n# **Missing data check**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Missingno is one of the best plots to visualize missing data. It offers a wide variety of plots and options. For example, the nullity_filter offers the ability to only present the columns that have a certain percentage of missing data. This is very useful for us since the combined dataset has a lot of columns and displaying them all would result in","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax = plt.subplots(figsize=(12,5))\n\n#Plot\nfiltered_data = msno.nullity_filter(df_train, filter='bottom', n=15, p=0.999)\nmsno.bar(filtered_data,fontsize=20)\n\nplt.xticks(fontsize=14, weight=\"bold\", alpha=0.75)\n\n#Title\nax.text(x=-0.5,y=1.2,s=\"Visualization of missing data on the training dataset.\",fontsize=30,weight=\"bold\",alpha=0.75)\nax.text(x=-0.5,y=1.15,s=\"The feature with the most missing data is floor_count where 82.6% of the data is missing.\",fontsize=22,alpha=0.75)\n#plt.savefig(\"missing_values_plot.png\")\n\ndel filtered_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"><a/>\n# **Initial outlier detection**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this segment, I will do a descriptive statistical analysis of the data and visual outlier detection. For the visual outlier detection, I have to give a shoutout to the Kaggle user [Roman](https://www.kaggle.com/nroman). You have inspired me to do my own take on this.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.options.display.float_format = '{:20,.2f}'.format\ndf_train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Resample daily\ndaily_consumption = df_train[\"meter_reading\"].resample(\"D\").mean()\n#Resample monthly\n#monthly_consumption = df_train[\"meter_reading\"].resample(\"M\").mean()\n#Resample hourly\nhourly_consumption = df_train[\"meter_reading\"].resample(\"H\").mean()\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (20,7))\n\n#PLot\nplt.plot(daily_consumption,linestyle=\"solid\",linewidth=3, alpha=0.7, label=\"Daily Consumption Mean\")\nplt.plot(hourly_consumption,linestyle=\"solid\",color=\"gray\",linewidth=3 ,alpha=0.75,label=\"Hourly Consumption Mean\")\nplt.xticks(fontsize=14,alpha=0.75)\nplt.yticks(fontsize=14,alpha=0.75)\n\nplt.legend(loc=\"upper right\")\n\n#Title\nax.text(x=dt.date(2015, 12, 1),y=12200,s=\"Daily and hourly mean consumption for the training dataset.\", weight=\"bold\", fontsize=\"22\",alpha=0.75)\nax.text(x=dt.date(2015, 12, 1),y=11500,s=\"The consumption was resampled using the timestamp feature to generate a clear timeline for the training dataset. \", fontsize=\"18\",alpha=0.75)\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Something is obviously not alright with the daily and monthly consumption data. The numbers are too high for a mean consumption and this suggests that there is an outlier that is heavily affecting the graph. I will do a step by step visual analysis to narrow down the problematic meter.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_site = df_train[\"site_id\"].unique()\n\nfig, ax = plt.subplots(figsize=(20,30),sharex=True)\n\n#Plot\nfor number, site in enumerate(unique_site, start=1):\n    \n    site_d = df_train[df_train[\"site_id\"] == site][\"meter_reading\"].resample(\"D\").mean()\n    site_h = df_train[df_train[\"site_id\"] == site][\"meter_reading\"].resample(\"H\").mean()\n    \n    plt.subplot(8,2,number)\n    site_d.plot(linestyle=\"solid\", color=\"blue\", alpha=0.8, label=\"Daily Mean\")\n    plt.tick_params(labelbottom=False)\n    site_h.plot(linestyle=\"solid\", color=\"grey\", alpha=0.8, label=\"Hourly Mean\")\n    plt.tick_params(labelbottom=False)\n    plt.title('Site_id {}'.format(site), loc=\"left\", fontsize=16, alpha=0.75, weight=\"bold\")\n    plt.xlabel(\"\")\n#Title\nplt.text(x=dt.date(2014, 10, 1),y=1400,s=\"Site-level daily and hourly mean consumption for the training dataset.\", weight=\"bold\", fontsize=\"22\",alpha=0.75)   \n#plt.savefig(\"consumption_per_site_id.png\")\n    \ngc.collect()\ndel unique_site","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am going to take four out of the above-presented sites for a deeper look. Two of which (site 0 and site 10) have a fairly normal-looking consumption. The other two sites (site 9 and site 13) have suspicious-looking consumption when compared to the other sites.\n\nLooking at a normal consumption distribution will give me a good initial baseline about what can or can't be expected from consumption data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"site = 0\nunique_meter = df_train[df_train[\"site_id\"] == site][\"meter\"].unique()\n\nfig, ax = plt.subplots(figsize=(20,30),sharey=True)\n\nfor number, meter in enumerate(unique_meter,start=1):\n    \n    meter_d = df_train[(df_train[\"meter\"] == meter) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"D\").mean()\n    meter_h = df_train[(df_train[\"meter\"] == meter) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"H\").mean()\n    \n    plt.subplot(4,2,number)\n    meter_d.plot(linestyle=\"solid\", color=\"blue\", alpha=0.75, label=\"Daily Mean\")\n    meter_h.plot(linestyle=\"solid\", color=\"grey\", alpha=0.75, label=\"Hourly Mean\")\n    \n    plt.title(\"Meter_type {}\".format(meter), loc=\"left\", fontsize=16, alpha=0.75, weight=\"bold\")\n\n#Title\n\nplt.text(x=dt.date(2015, 2, 1),y=15100,s=\"Site 0 - daily and hourly mean consumption for the training dataset.\", weight=\"bold\", fontsize=\"22\",alpha=0.75)\n    \ngc.collect()\ndel unique_meter","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"site = 9\nunique_meter = df_train[df_train[\"site_id\"] == site][\"meter\"].unique()\n\nfig, ax = plt.subplots(figsize=(20,30),sharey=True)\n\nfor number, meter in enumerate(unique_meter,start=1):\n    \n    meter_d = df_train[(df_train[\"meter\"] == meter) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"D\").mean()\n    meter_h = df_train[(df_train[\"meter\"] == meter) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"H\").mean()\n    \n    plt.subplot(4,2,number)\n    meter_d.plot(linestyle=\"solid\", color=\"blue\", alpha=0.75, label=\"Daily Mean\")\n    meter_h.plot(linestyle=\"solid\", color=\"grey\", alpha=0.75, label=\"Hourly Mean\")\n    \n    plt.title(\"Meter_type {}\".format(meter), loc=\"left\", fontsize=16, alpha=0.75,weight=\"bold\")\n#Title\nplt.text(x=dt.date(2015, 12, 10),y=13000,s=\"Site 9 - daily and hourly mean consumption for the training dataset.\", weight=\"bold\", fontsize=\"22\",alpha=0.75)    \n\ngc.collect()\ndel unique_meter","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"site = 10\nunique_meter = df_train[df_train[\"site_id\"] == site][\"meter\"].unique()\n\nfig, ax = plt.subplots(figsize=(20,30),sharey=True)\n\nfor number,meter in enumerate(unique_meter,start=1):\n    \n    meter_d = df_train[(df_train[\"meter\"] == meter) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"D\").mean()\n    meter_h = df_train[(df_train[\"meter\"] == meter) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"H\").mean()\n    \n    plt.subplot(4,2,number)\n    meter_d.plot(linestyle=\"solid\", color=\"blue\", alpha=0.75, label=\"Daily Mean\")\n    meter_h.plot(linestyle=\"solid\", color=\"grey\", alpha=0.75, label=\"Hourly Mean\")\n    \n    plt.title(\"Meter_type {}\".format(meter), loc=\"left\", fontsize=16, alpha=0.75, weight=\"bold\")\n\n#Title\nplt.text(x=dt.date(2015, 12, 10),y=4300,s=\"Site 10 - daily and hourly mean consumption for the training dataset.\", weight=\"bold\", fontsize=\"22\",alpha=0.75)\n\ngc.collect()\ndel unique_meter","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"site = 13\nunique_meter = df_train[df_train[\"site_id\"] == site][\"meter\"].unique()\n\nfig, ax = plt.subplots(figsize=(20,30),sharey=True)\n\nfor number, meter in enumerate(unique_meter,start=1):\n    \n    meter_d = df_train[(df_train[\"meter\"] == meter) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"D\").mean()\n    meter_h = df_train[(df_train[\"meter\"] == meter) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"H\").mean()\n    \n    plt.subplot(4,2,number)\n    meter_d.plot(linestyle=\"solid\", color=\"blue\", alpha=0.8, label=\"Daily Mean\")\n    meter_h.plot(linestyle=\"solid\", color=\"grey\", alpha=0.8, label=\"Hourly Mean\")\n    plt.xlabel(\"\")\n    \n    plt.title(\"Meter_type {}\".format(meter), loc=\"left\", fontsize=16,weight=\"bold\",alpha=0.75)\n\n#Title\nplt.text(x=dt.date(2015, 12, 10),y=625000,s=\"Site 13 - daily and hourly mean consumption for the training dataset.\", weight=\"bold\", fontsize=\"22\",alpha=0.75)\n#plt.savefig(\"consumption_per_site13_meter.png\")\n\ngc.collect()\ndel unique_meter","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The major outlier that is effecting the whole dataset is most definitely on the site 13 Steam meter.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"site = 13\nunique_use = df_train[df_train[\"site_id\"] == site][\"primary_use\"].unique()\n\nfig, ax = plt.subplots(figsize=(20,30),sharey=True)\n\nfor number,use in enumerate(unique_use,start=1):\n     \n    use_d = df_train[(df_train[\"primary_use\"] == use) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"D\").mean()\n    use_h = df_train[(df_train[\"primary_use\"] == use) & (df_train[\"site_id\"] == site)][\"meter_reading\"].resample(\"H\").mean()\n    \n    plt.subplot(7,2,number)\n    plt.subplots_adjust(bottom=0.15, top=1.5)\n    use_d.plot(linestyle=\"solid\", color=\"blue\", alpha=0.75, label=\"Daily Mean\")\n    use_h.plot(linestyle=\"solid\", color=\"grey\", alpha=0.75, label=\"Hourly Mean\")\n    plt.xlabel(\"\")\n    \n    plt.title(\"Used for: {}\".format(use), loc=\"left\", fontsize=16, weight=\"bold\", alpha=0.75)\n\n#Title\nplt.text(x=dt.date(2015, 12, 10),y=3000,s=\"Site 13 - daily and hourly mean consumption per primary use\", weight=\"bold\", fontsize=\"22\",alpha=0.75)\nplt.text(x=dt.date(2015, 12, 10),y=2975,s=\"Primary use indicates that there are multiple uses for the building, but the primary one represents the industry of the largest building tenant.\", fontsize=\"17\",alpha=0.75)\n    \nplt.savefig(\"consumption_per_site_meter_use.png\")\n        \ngc.collect()\ndel unique_use","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above plot narrows down further by eliminating every building that isnt primarily used for education.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"site = 13\nbuildings = df_train[(df_train[\"site_id\"] == 13) & (df_train[\"primary_use\"] == \"Education\") \n         & (df_train[\"meter\"] == \"Steam\")][\"building_id\"].unique()\n\n\nfig, ax = plt.subplots(figsize=(20,30),sharey=True)\n\nfor number,build in enumerate(buildings,start=1):\n    \n    \n    build_d = df_train[(df_train[\"primary_use\"] == \"Education\") & (df_train[\"site_id\"] == 13) & (df_train[\"meter\"] == \"Steam\") &\n                      (df_train[\"building_id\"] == build) & (df_train[\"building_id\"]==build)][\"meter_reading\"].resample(\"D\").mean()\n    \n    build_h = df_train[(df_train[\"primary_use\"] == \"Education\") & (df_train[\"site_id\"] == 13) &\n                      (df_train[\"meter\"] == \"Steam\") & (df_train[\"building_id\"]==build)][\"meter_reading\"].resample(\"H\").mean()\n    \n    plt.subplot(9,2,number)\n    plt.subplots_adjust(bottom=0.20, top=1.5)\n    build_d.plot(linestyle=\"solid\", color=\"blue\", alpha=0.75, label=\"Daily Mean\")\n    build_h.plot(linestyle=\"solid\", color=\"grey\", alpha=0.75, label=\"Hourly Mean\")\n    plt.xlabel(\"\")\n    \n    plt.title(\"Building ID: {}\".format(build), loc=\"left\", fontsize=16, weight=\"bold\", alpha=0.75)\n    \n#Title   \nplt.text(x=dt.date(2015, 12, 10),y=59000,s=\"Site 13 - Daily and hourly mean consumption per building ID.\", weight=\"bold\", fontsize=\"22\",alpha=0.75)\nplt.text(x=dt.date(2015, 12, 10),y=58500,s=\"The consumption includes all Steam meters that are primarily used for Education.\", fontsize=\"17\",alpha=0.75)\n#plt.savefig(\"consumption_per_site_id_meter_use_building.png\")\ngc.collect()\ndel buildings","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_clean = df_train[~((df_train[\"building_id\"] == 1099) & (df_train[\"meter\"]==\"Steam\"))]\n\ngc.collect()\ndel df_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_daily_mean = df_train_clean[\"meter_reading\"].resample(\"D\").mean()\nnew_hourly_mean = df_train_clean[\"meter_reading\"].resample(\"H\").mean()\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,7))\n\nplt.plot(new_daily_mean, linestyle=\"solid\", color=\"blue\", alpha=0.75, label=\"Daily Mean\")\nplt.plot(new_hourly_mean, linestyle=\"solid\", color=\"grey\", alpha=0.75, label=\"Hourly Mean\")\n\n\nplt.text(x=dt.date(2015, 12, 10),y=1350,s=\"New daily and hourly consumption on the training dataset\", weight=\"bold\", fontsize=\"22\",alpha=0.75)\nplt.text(x=dt.date(2015, 12, 10),y=1290,s=\"Acquired by removing the buidling_id 1099 Steam meter from the dataset.\", fontsize=\"16\",alpha=0.75)\nplt.legend(fontsize=\"x-large\")\nplt.savefig(\"new_consumption.png\")\n\ngc.collect()\n\ndel new_daily_mean, new_hourly_mean","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The new plot looks a lot better than its previous counterpart. However the data overall still cannot be considered clean. The issue I have found that prevents one-method-for-all approaches is the heterogenic nature of the dataset. Each of the buildings is an entity on its own with a unique consumption profile that prevents general solutions. This is especially true of electricity meters because they are not affected by temperature in the same way the HVAC meters are.\n\nWhile temperature has a role in the profiling of electricity meters its minor compared to internal factors that dictate its consumption. The approach I will try while producing a model is to plot the electricity consumption for each building and mark the outliers manually. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# **<a id=\"5\"> EDA </a>**\n\nNow that the initial cleaning of the data is finished, I am going to take a look at some interesting features of the dataset and their relationship with the target variable. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**<a id=\"6\"> Square feet and EE </a>**\n\nSquare feet is interesting to look at. The data is heavily left-skewed since there tend to be more small/medium buildings then skyscrapers. Because of this, we need to do a logarithmic transformation of the data that will even out the distribution.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10,5))\n\nplt.hist(np.log1p(df_train_clean[\"square_feet\"]))\n\nax.text(x=4.5,y=6300000,s=\"Distribution of square feet on the training dataset.\", weight=\"bold\", fontsize=\"18\",alpha=0.75)\nax.text(x=4.5,y=6000000,s=\"Log transformation was used to address the initial right-skewed data distribution.\", fontsize=\"14\",alpha=0.75)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now you will probably ask: OG, why is the square feet column interesting to look at? It's just a distribution that needs a logarithmic transformation. \n\nWell, it's interesting because consumption per square feet is a go-to metric in evaluating the energy efficiency of buildings. The method is slightly flawed but very useful in a lot of situations.It is usually applied after the building undergoes renovations to improve isolation. That being said, a lot of times all consumption is taken into account when calculating kWh per sq ft. However, I find that its better used when separating meter types.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ee = df_train_clean.groupby([\"site_id\",\"building_id\",\"meter\"]).agg({\"meter_reading\":\"sum\",\"square_feet\":\"max\"})\nee[\"consumption_per_squaref\"] = ee[\"meter_reading\"]/ ee[\"square_feet\"]\nee.sort_values(\"consumption_per_squaref\",ascending=False, inplace=True)\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,5))\n#Data\nee_steam = ee[ee.index.get_level_values(\"meter\") == \"Steam\"]\n\n#Plot\nplt.barh(ee_steam.index.get_level_values(\"building_id\")[:20].astype(\"str\"),ee_steam[\"consumption_per_squaref\"][:20])\n\nplt.xlabel(\"Energy consumption per sq ft\", fontsize=14, alpha=0.75)\nplt.ylabel(\"Building ID number\",fontsize=14, alpha=0.75)\n\nax.annotate(\"Consumption outlier\",xy=(6000,0.5),xytext=(6000,5),arrowprops=dict(facecolor='black', shrink=0.05),weight=\"bold\",alpha=0.75)\n\n#Title\n\nax.text(x=0,y=22,s=\"Energy efficiency - Steam meter consumption per square feet.\", weight=\"bold\", fontsize=\"18\",alpha=0.75)\nax.text(x=0,y=20.75,s=\"Top 20 most energy inefficient buildings.\", fontsize=\"14\",alpha=0.75)\n\ndel ee_steam\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,5))\n#Data\nee_steam = ee[ee.index.get_level_values(\"meter\") == \"Chilledwater\"]\n\n#Plot\nplt.barh(ee_steam.index.get_level_values(\"building_id\")[:20].astype(\"str\"),ee_steam[\"consumption_per_squaref\"][:20])\n\nplt.xlabel(\"Energy consumption per sq ft\", fontsize=14, alpha=0.75)\nplt.ylabel(\"Building ID number\",fontsize=14, alpha=0.75)\n\nax.annotate(\"Consumption outlier\",xy=(3500,0.5),xytext=(3500,5),arrowprops=dict(facecolor='black', shrink=0.05),weight=\"bold\",alpha=0.75)\n\n#Title\n\nax.text(x=0,y=22,s=\"Energy efficiency - Chilledwater meter consumption per square feet.\", weight=\"bold\", fontsize=\"18\",alpha=0.75)\nax.text(x=0,y=20.75,s=\"Top 20 most energy inefficient buildings.\", fontsize=\"14\",alpha=0.75)\n\ndel ee_steam\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,5))\n#Data\nee_steam = ee[ee.index.get_level_values(\"meter\") == \"Hotwater\"]\n\n#Plot\nplt.barh(ee_steam.index.get_level_values(\"building_id\")[:20].astype(\"str\"),ee_steam[\"consumption_per_squaref\"][:20])\n\nplt.xlabel(\"Energy consumption per sq ft\", fontsize=14, alpha=0.75)\nplt.ylabel(\"Building ID number\",fontsize=14, alpha=0.75)\n\n#Title\n\nax.text(x=0,y=22,s=\"Energy efficiency - Hotwater meter consumption per square feet.\", weight=\"bold\", fontsize=\"18\",alpha=0.75)\nax.text(x=0,y=20.75,s=\"Top 20 most energy inefficient buildings.\", fontsize=\"14\",alpha=0.75)\n\ndel ee_steam, ee\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have marked some consumption outliers above that are targets for future data cleaning.\n\nOne problem with this comparison is that not all building are exposed to the same temperature. That is especially important on HVAC meters that are extremely sensitive to it.\n\nAnother interesting metric for the evaluation of energy efficiency is kWh per degree. This is rarely used as requires additional data to be gathered and it can be very inaccurate. Furthermore, it's a bit pointless when comparing buildings in the same area or country. However, this might be an interesting take as we have no idea where these buildings are.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_df = df_train_clean.copy()\n\n\ntemp_df[\"air_temperature\"] = np.abs(temp_df[\"air_temperature\"].astype(\"float32\"))\ntemp_df.dropna(subset=[\"air_temperature\"],inplace=True)\ntemp_df[\"consu_per_temp\"] = temp_df[\"meter_reading\"] / (temp_df[\"air_temperature\"] + 1)\n\nee = temp_df.groupby([\"site_id\",\"building_id\",\"meter\"])[\"consu_per_temp\"].agg([\"mean\"])\n\nee.sort_values(by=\"mean\", inplace=True, ascending=False)\n\n\ngc.collect()\n\ndel temp_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(14,7))\n\nee_steam = ee[ee.index.get_level_values(\"meter\")  == \"Steam\"]\n\nplt.barh(ee_steam.index.get_level_values(\"building_id\")[1:21].astype(\"str\"),ee_steam[\"mean\"][1:21])\n\nplt.xlabel(\"Energy consumption per temperature degree\", fontsize=14, alpha=0.75)\nplt.ylabel(\"Building ID number\",fontsize=14, alpha=0.75)\n\n#Title\n\nax.text(x=0,y=21.5,s=\"Energy efficiency - Steam meter consumption per air temperature degree.\", weight=\"bold\", fontsize=\"18\",alpha=0.75)\nax.text(x=0,y=20.5,s=\"Top 20 most energy inefficient buildings(Outliers removed).\", fontsize=\"14\",alpha=0.75)\n\ngc.collect()\n\ndel ee_steam","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(14,7))\n\nee_chill = ee[ee.index.get_level_values(\"meter\")  == \"Chilledwater\"]\n\nplt.barh(ee_chill.index.get_level_values(\"building_id\")[1:21].astype(\"str\"),ee_chill[\"mean\"][1:21])\n\nplt.xlabel(\"Energy consumption per temperature degree\", fontsize=14, alpha=0.75)\nplt.ylabel(\"Building ID number\",fontsize=14, alpha=0.75)\n\n#Title\n\nax.text(x=0,y=21.5,s=\"Energy efficiency - Chilledwater meter consumption per air temperature degree.\", weight=\"bold\", fontsize=\"18\",alpha=0.75)\nax.text(x=0,y=20.5,s=\"Top 20 most energy inefficient buildings(Outliers removed).\", fontsize=\"14\",alpha=0.75)\n\ngc.collect()\n\ndel ee_chill","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(14,7))\n\nee_hot = ee[ee.index.get_level_values(\"meter\")  == \"Hotwater\"]\n\nplt.barh(ee_hot.index.get_level_values(\"building_id\")[2:22].astype(\"str\"),ee_hot[\"mean\"][2:22])\n\nplt.xlabel(\"Energy consumption per temperature degree\", fontsize=14, alpha=0.75)\nplt.ylabel(\"Building ID number\",fontsize=14, alpha=0.75)\n\n#Title\n\nax.text(x=0,y=21.5,s=\"Energy efficiency - Hotwater meter consumption per air temperature degree.\", weight=\"bold\", fontsize=\"18\",alpha=0.75)\nax.text(x=0,y=20.5,s=\"Top 20 most energy inefficient buildings(Outliers removed).\", fontsize=\"14\",alpha=0.75)\n\ndel ee_hot\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**<a id=\"7\"> Linear correlations </a>**\n\nChecking for linear correlation between the feature is a good practice. However, the heterogenic nature of the dataset I mentioned previously is most likely going to cause issues.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_heatmap =  df_train_clean.select_dtypes(\"number\")\ndf_heatmap.drop([\"site_id\", \"building_id\"], axis=1,inplace=True)\n\ntest = df_heatmap.corr()\n\ngc.collect()\ndel df_heatmap","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax, fig = plt.subplots(figsize=(20,10))\n\nax = sns.heatmap(test,annot=True,annot_kws={\"size\": 14},linewidths=.5,center=0,cbar=False)\n\nplt.xticks(rotation=15,weight=\"bold\", fontsize=16, alpha=0.75)\nplt.yticks(weight=\"bold\", fontsize=16, alpha=0.75)\n\nbottom, top = plt.ylim()\nax.set_ylim(top - 0.5 , bottom + 0.5)\n\n#Title\nplt.text(x=-1.4,y=12,s=\"Heatmap of the numeric correlations\", weight=\"bold\",alpha=0.75, fontsize=27)\nplt.text(x=-1.4,y=11.5,s=\"There are almost no linear correlations between the individual columns.\",alpha=0.75, fontsize=22)\n\nplt.savefig(\"corr.png\")\n\ngc.collect()\n\ndel test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is some correlation between the features, but overall the linear correlation heatmap is not saying much. The obvious correlation is that of square feet and numbers of floors (who would have guessed) and air and dew temperature. However, this was more or less expected as the dataset has a few columns of categorical data and its unreasonable to expect each category to behave the same.\n\n The exploration of dependencies within categories needs a different approach that has whiskers - the box and whiskers plot","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**<a id=\"8\" > 🐱 Categorical differences in energy usage </a>**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Box plots are a great non-parametric way of analysing the same feature with different categories in mind. They also offer a wide range of information from the mean to potential outliers in the data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_clean[\"log_meter\"] = np.log1p(df_train_clean[\"meter_reading\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax, fig = plt.subplots(figsize=(14,7))\n\nsns.boxplot(y=\"meter\", x=\"log_meter\",palette=\"GnBu_d\",data=df_train_clean)\n\nplt.xlabel(\"Logaritmic consumption\",weight=\"bold\", alpha=0.75, fontsize=15)\nplt.ylabel(\"Meter type\",weight=\"bold\", alpha=0.75, fontsize=15)\n\n#Ttile\n\nplt.text(x=-2.2,y=-1,s=\"Distribution of energy consumption per meter type\", weight=\"bold\",fontsize=20,alpha=0.75)\nplt.text(x=-2.2,y=-0.8,s=\"There is a lot of variability between and within the individual meter types.\",fontsize=15,alpha=0.75)\n\nplt.savefig(\"box_meter.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Steam is most definitely the largest consumer of electricity. While Chilled water and Electricity seam to have the most potential outliers. The whiskers long range also indicates that there is a lot of variety in consumption for all meters.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ax, fig = plt.subplots(figsize=(14,7))\n\nbins = [1900,1910,1920,1930,1940,1950,1960,1970,1980,1990,2000,2010,2020]\ndf_train_clean[\"binns\"] = pd.cut(df_train_clean[\"year_built\"], bins)\n\nax = sns.boxplot(y=\"binns\", x=\"log_meter\",palette=\"GnBu_d\",data=df_train_clean)\n\nplt.xlabel(\"Logaritmic consumption\",weight=\"bold\", alpha=0.75, fontsize=15)\nplt.ylabel(\"Construction decade\",weight=\"bold\", alpha=0.75, fontsize=15)\n\nax.set_yticklabels([\"1900-1910\",\"1910-1920\",\"1920-1930\",\"1930-1940\",\"1940-1950\",\"1950-1960\",\"1960-1970\",\"1970-1980\",\"1980-1990\",\"1990-2000\",\"2000-2010\",\"2010-2020\"], alpha=0.75)\n\n#Title\nplt.text(x=-1.7,y=-1.5,s=\"Distribution of the energy consumption\", weight=\"bold\",fontsize=20,alpha=0.75)\nplt.text(x=-1.7,y=-1.05,s=\"The categorization is based on the construction decade of the buildings.\",fontsize=15,alpha=0.75)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"My initial hypothesis for this plot was an increased consumption for the early construction dates. The logic behind this assumption is that the modern construction and isolation methods would greatly surpass old standards and in turn result in less consumption. However, this was a one-sided approach on my part. The isolation of the older buildings might be worse, but the main reason behind the data we see above is the size of the buildings. Modern buildings are much larger in comparison to the buildings made over a century ago.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ax, fig = plt.subplots(figsize=(14,7))\n\nax = sns.boxplot(y=\"primary_use\", x=\"log_meter\",palette=\"GnBu_d\",data=df_train_clean)\n\nplt.xlabel(\"Logaritmic consumption\",weight=\"bold\", alpha=0.75, fontsize=15)\nplt.ylabel(\"Primary use of the building\",weight=\"bold\", alpha=0.75, fontsize=15)\n\n#Title\nplt.text(x=-4.2,y=-1.5,s=\"Distribution of energy consumption per primary use\", weight=\"bold\",fontsize=20,alpha=0.75)\nplt.text(x=-4.2,y=-0.8,s=\"Again, there is a lot of variability between and within the individual types.There are also many more outliers than in the previous boxplot.\",fontsize=15,alpha=0.75)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Like I mentioned previously, primary-use is a very broad term. This just means that the main consumer of electricity operates in that industry and it isn't taking into account the other tenants. To add to the confusion, the industry descriptions like \"Service\" are also very broad with a lot of variety between the individual service industry branches.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_clean.reset_index(inplace=True)\ndf_train_clean[\"timestamp\"] = pd.to_datetime(df_train_clean[\"timestamp\"])\ndf_train_clean[\"dayofweek\"] = df_train_clean[\"timestamp\"].dt.day_name()\ndf_train_clean[\"Month\"] = df_train_clean[\"timestamp\"].dt.month\ndf_train_clean[\"Hour\"] = df_train_clean[\"timestamp\"].dt.hour","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax, fig = plt.subplots(figsize=(14,7))\n\n#Data\norder_of_days = [\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\",\"Saturday\",\"Sunday\"]\n\n#Plot\nax = sns.boxplot(y=\"dayofweek\", x=\"log_meter\",palette=\"GnBu_d\",data=df_train_clean,order=order_of_days)\n\nplt.xlabel(\"Logaritmic consumption\",weight=\"bold\", alpha=0.75, fontsize=15)\nplt.ylabel(\"Weekday\",weight=\"bold\", alpha=0.75, fontsize=15)\n\n#Title\nplt.text(x=-2.35,y=-1.1,s=\"Distribution of energy consumption per weekday\", weight=\"bold\",fontsize=20,alpha=0.75)\nplt.text(x=-2.35,y=-0.8,s=\"There is a slight reduction in consumption on the weekend days.\",fontsize=15,alpha=0.75)\n\ngc.collect()\ndel order_of_days","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The last of the famous box and whiskers plots. I initially expected to see a larger difference in the mean consumptions between the workdays and non-workdays. There is a small dip, but its nothing substantial.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**<a id=\"9\">Building data analysis</a>**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This segment will be geared towards looking at the building metadata that was handed to us. I will take a look at the number of sites and the buildings within those sites and the construction date distribution of the dataset. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ax, fig = plt.subplots(figsize=(14,7))\n\n#Data\ncount_building_id = df_train_clean.groupby(\"site_id\")[\"building_id\"].nunique()\ncount_building_id = count_building_id.reset_index()\ncount_building_id.sort_values(by=\"building_id\",ascending=False,inplace=True)\n\n#Plot\nbars = plt.bar(count_building_id[\"site_id\"].astype(\"str\"),count_building_id[\"building_id\"],color=\"blue\",alpha=0.5)\nplt.xticks(alpha=0.75)\nplt.yticks(alpha=0.75)\nplt.xlabel(\"Individual site ID\", alpha=0.75, weight=\"bold\", fontsize=16)\nplt.ylabel(\"Number of buildings\", alpha=0.75, weight=\"bold\", fontsize=16)\n\nfor bar in bars:\n    yval = bar.get_height()\n    plt.text(bar.get_x() + 0.20, yval + .015, yval, color = \"black\")\n    \n#Title\nax.text(x=0.05,y=0.95,s=\"Number of buildings per site\", weight=\"bold\",fontsize=18,alpha=0.75)\nax.text(x=0.05,y=0.92,s=\"The most buildings can be found on site 3, with 274 individual buildings located there.\",fontsize=14,alpha=0.75)\n\n#plt.savefig(\"building_per_site.png\")\n\ndel count_building_id\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Considering the ASHRAE is a primary American company, we can assume that the sites with the most buildings are in the US. The sites that belong in the lower 50% could potentially be from other parts of the world since ASHRAE has a global outreach.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ax, fig = plt.subplots(figsize=(14,7))\n\n#Data\nbuild_date = df_train_clean.groupby(\"building_id\").agg({\"meter_reading\" : \"mean\", \"year_built\" : \"max\"})\n\n#Plot\n\nplt.hist(build_date[\"year_built\"], bins=50,color = \"blue\", alpha=0.5)\n\nplt.xlabel(\"Year of construction\", alpha=0.75, weight=\"bold\")\nplt.ylabel(\"Number of constructions\", alpha=0.75, weight=\"bold\")\n\n#Title\n\nax.text(x=0.08, y=0.92,s=\"The distribution of construction years on the training dataset.\", weight=\"bold\",fontsize=16,alpha=0.75)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#df_train_clean.set_index(df_train_clean[\"timestamp\"],inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**<a id=\"10\" > Temperature data analysis </a>**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Temp dataset\ntemp_df = df_train_clean[[\"timestamp\",\"site_id\",\"air_temperature\"]]\n\n#Droping duplicates\nkey = [\"timestamp\",\"site_id\"]\ntemp_df = temp_df[key + [\"air_temperature\"]].drop_duplicates(subset=key)\n\n#Sorting values and extracting hours\ntemp_df = temp_df.sort_values(by=key)\ntemp_df[\"Hour\"] = temp_df.timestamp.dt.hour\n\n#Groupby mean and unstack\ng_temp = temp_df.groupby([\"site_id\",\"Hour\"])[\"air_temperature\"].agg([\"mean\"]).unstack(level=1)\n\n#Divide the column temperatures by the row max\ng_temp = g_temp / g_temp.max(axis=1).values.reshape((-1,1))  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax, fig = plt.subplots(figsize=(20,10))\n\nax = sns.heatmap(g_temp,linewidths=.5,cmap=\"YlOrRd\")\n\nax.set_xticklabels(np.arange(0,24).astype(\"str\"))\n\nplt.xticks(weight=\"bold\", fontsize=16, alpha=0.75)\nplt.yticks(weight=\"bold\", fontsize=16, alpha=0.75)\n\nplt.xlabel(\"Hour\",weight=\"bold\", fontsize=25,alpha=0.75)\nplt.ylabel(\"Site ID\",weight=\"bold\", fontsize=25,alpha=0.75)\n\nbottom, top = plt.ylim()\nax.set_ylim(top - 0.5 , bottom + 0.5)\n\n#Title\nplt.text(x=-1.4,y=17,s=\"Mean temperature per hour\", weight=\"bold\",alpha=0.75, fontsize=27)\nplt.text(x=-1.4,y=16.3,s=\"The heatmap displays the mean temperature per hour for each site in the training dataset.\",alpha=0.75, fontsize=22)\n\n\ngc.collect()\n\ndel g_temp, temp_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The heatmap above represents the average hourly temperature for each site in the training dataset.  To properly compare the sites I have divided each mean by the maximum amount of the site. This allows for a better visual presentation of the data and overall quality.\n\nThe first thing that is immediately noticeable is that some of the sits don't align well with the rest of the data with their peak temperatures. Peak temperature misalignment can happen due to location differences, problems with timezone settings, etc. This is definitely a topic to research further.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The following plots will contain data on how each individual meter interacts with air temperature.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_clean.set_index(df_train_clean[\"timestamp\"],inplace=True)\n#df_train_clean.drop(\"timstamp\",axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i = 0\nm_types = df_train_clean[\"meter\"].unique()\n\nfig,ax = plt.subplots(figsize=(20,20))\n\nfor meter in m_types:\n    i+=1\n    meter_type = df_train_clean[df_train_clean[\"meter\"] == meter][\"log_meter\"].resample(\"D\").mean()\n    plt.subplot(8,2,i)\n    plt.plot(meter_type,color=\"blue\",alpha=0.5)\n    plt.title(\"Consumption - Meter type: {}\".format(meter), weight=\"bold\", alpha=0.75)\n    \n    i+=1\n    weather_data = df_train_clean[df_train_clean[\"meter\"] == meter][\"air_temperature\"].resample(\"D\").mean()\n    plt.subplot(8,2,i)\n    plt.plot(weather_data, color = \"red\")\n    plt.title(\"Temperature - Meter type: {}\".format(meter), weight=\"bold\", alpha=0.75)\n    plt.tight_layout(pad=1)\n    \nplt.text(x=dt.date(2014, 10, 15),y=192,s=\"Comparison of consumption and temperature.\",weight=\"bold\",fontsize=25,alpha=0.75)\nplt.text(x=dt.date(2014, 10, 15),y=187,s=\"The electricity meters are the only ones not absolutely affected by the outside temperature.\",fontsize=20,alpha=0.75)\n\n#plt.savefig(\"consumption_vs_temperature.png\")\n\ndel meter_type, m_types, weather_data\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax = plt.subplots(figsize=(10,5))\n\nmeter_type = df_train_clean[df_train_clean[\"meter\"] == \"Steam\"][\"log_meter\"].resample(\"D\").mean()\nweather_data = df_train_clean[df_train_clean[\"meter\"] == \"Steam\"][\"air_temperature\"].resample(\"D\").mean()\n\nax.plot(meter_type, color=\"blue\",alpha=0.5)\nax.set_ylabel(\"Consumption\",color=\"blue\",alpha=0.5,fontsize=14)\n#ax.grid(False)\n\nax2=ax.twinx()\nax2.plot(weather_data,color=\"grey\",alpha=0.5)\nax2.set_ylabel(\"Temperature\",color=\"grey\",fontsize=14)\nax2.grid(False)\n#plt.title(\"Steam Meter : Consumption vs Temperature\")\n\n#Title\nax.text(x=dt.date(2015, 12, 1),y=7.2,s=\"Steam meters - consumption vs temperature.\",weight=\"bold\",fontsize=16,alpha=0.75)\nax.text(x=dt.date(2015, 12, 1),y=7.0,s=\"Steam as a heating meter is inversely proportional to temperature.\",fontsize=12,alpha=0.75)\n#plt.savefig(\"steam_vs_temp.png\")\n\ndel meter_type, weather_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax = plt.subplots(figsize=(10,5))\n\n#Data\nmeter_type = df_train_clean[df_train_clean[\"meter\"] == \"Electricity\"][\"log_meter\"].resample(\"D\").mean()\nweather_data = df_train_clean[df_train_clean[\"meter\"] == \"Electricity\"][\"air_temperature\"].resample(\"D\").mean()\n\n#Plot\nax.plot(meter_type, color=\"blue\",alpha=0.5)\nax.set_ylabel(\"Consumption\",color=\"blue\",alpha=0.5,fontsize=14)\n\nax2=ax.twinx()\nax2.plot(weather_data,color=\"grey\",alpha=0.5)\nax2.set_ylabel(\"Temperature\",color=\"grey\",alpha=0.5,fontsize=14)\nax2.grid(False)\n\n#Title\nax.text(x=dt.date(2015, 12, 1),y=4.5,s=\"Electricity meters - consumption vs temperature.\",weight=\"bold\",fontsize=16,alpha=0.75)\nax.text(x=dt.date(2015, 12, 1),y=4.45,s=\"Electricity meters have some correlation with the temperature data, but it needs to be researched further.\",fontsize=12,alpha=0.75)\n#plt.savefig(\"steam_vs_temp.png\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax = plt.subplots(figsize=(10,5))\n\n#Data\nmeter_type = df_train_clean[df_train_clean[\"meter\"] == \"Chilledwater\"][\"log_meter\"].resample(\"D\").mean()\nweather_data = df_train_clean[df_train_clean[\"meter\"] == \"Chilledwater\"][\"air_temperature\"].resample(\"D\").mean()\n#Plot\nax.plot(meter_type, color=\"blue\",alpha=0.5)\nax.set_ylabel(\"Consumption\",color=\"blue\",alpha=0.5,fontsize=14)\n\nax2=ax.twinx()\nax2.plot(weather_data,color=\"grey\",alpha=0.5)\nax2.set_ylabel(\"Temperature\",color=\"grey\",fontsize=14)\nax2.grid(False)\n\n#Title\nax.text(x=dt.date(2015, 12, 1),y=6.1,s=\"Chilledwater meters - consumption vs temperature.\",weight=\"bold\",fontsize=16,alpha=0.75)\nax.text(x=dt.date(2015, 12, 1),y=5.95,s=\"Chilledwater as a cooling meter is direclty proportional to temperature.\",fontsize=12,alpha=0.75)\n\ndel meter_type, weather_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax = plt.subplots(figsize=(10,5))\n\n#Data\nmeter_type = df_train_clean[df_train_clean[\"meter\"] == \"Hotwater\"][\"log_meter\"].resample(\"D\").mean()\nweather_data = df_train_clean[df_train_clean[\"meter\"] == \"Hotwater\"][\"air_temperature\"].resample(\"D\").mean()\n\n#Plot\nax.plot(meter_type, color=\"blue\",alpha=0.5)\nax.set_ylabel(\"Consumption\",color=\"blue\",alpha=0.5,fontsize=14)\n\nax2=ax.twinx()\nax2.plot(weather_data,color=\"grey\",alpha=0.5)\nax2.set_ylabel(\"Temperature\",color=\"grey\",fontsize=14)\nax2.grid(False)\n\n#Title\nax.text(x=dt.date(2015, 12, 1),y=5.4,s=\"Hotwater meters - consumption vs temperature.\",weight=\"bold\",fontsize=16,alpha=0.75)\nax.text(x=dt.date(2015, 12, 1),y=5.25,s=\"Hotwater as a heating meter is inversely proportional to temperature.\",fontsize=12,alpha=0.75)\n\ndel meter_type,weather_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**<a id=11> Final words <a/>**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Generally speaking, this notebook only scratches the surface of a topic as deep as energy usage. There are many more additions that can be made, especially regarding other the relationship between temperature and consumption.\n\nHopefully, I have managed to offer a somewhat new perspective of the dataset and I hope that it was of some use for you. Regarding the future of the notebook, I always tend to come back to my earlier works and apply some new knowledge to them so that updates can be expected.\n\n\nCheers,\n\nO.G.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}