{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThe aim of this notebook is to analyse data that can spoil averages.\n\nTo show the data with less noise, a daily grouping has been performed."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport plotly.express as px\nfrom IPython.display import display\nfrom ashrae_utils import reduce_mem_usage","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepare data"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"data_path = '../input/ashrae-energy-prediction/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## train"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"X_train = pd.read_csv(data_path + 'train.csv', engine='python')\nX_train, na_list = reduce_mem_usage(X_train)\nX_train['timestamp'] = pd.to_datetime(X_train['timestamp'], format='%Y-%m-%d %H:%M:%S')\nX_train['meter'] = pd.Categorical(X_train['meter']).rename_categories({0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'})","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## building metadata"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"building_metadata = pd.read_csv(data_path + 'building_metadata.csv', engine='python')\nbuilding_metadata, na_list = reduce_mem_usage(building_metadata)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Group data in a daily basis"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"daily_train = X_train\ndaily_train['date'] = daily_train['timestamp'].dt.date\ndaily_train = daily_train.groupby(['date', 'building_id', 'meter']).sum()\ndaily_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Aggregate the data for buildings"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"daily_train_agg = daily_train.groupby(['date', 'meter']).agg(['sum', 'mean', 'idxmax', 'max'])\ndaily_train_agg = daily_train_agg.reset_index()\nlevel_0 = daily_train_agg.columns.droplevel(0)\nlevel_1 = daily_train_agg.columns.droplevel(1)\nlevel_0 = ['' if x == '' else '-' + x for x in level_0]\ndaily_train_agg.columns = level_1 + level_0\ndaily_train_agg.rename_axis(None, axis=1)\ndaily_train_agg.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig_total = px.line(daily_train_agg, x='date', y='meter_reading-sum', color='meter', render_mode='svg')\nfig_total.update_layout(title='Total kWh per energy aspect')\nfig_total.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The sum, facetted for each energy aspect, shows some aberrant values."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig_maximum = px.line(daily_train_agg, x='date', y='meter_reading-max', color='meter', render_mode='svg')\nfig_maximum.update_layout(title='Maximum kWh value per energy aspect')\nfig_maximum.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at the max value for each day, and for each energy aspect, shows that only a single building (for day and energy aspect) is causing the aberrant peaks."},{"metadata":{},"cell_type":"markdown","source":"# Identifying outliers"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"daily_train_agg['building_id_max'] = [x[1] for x in daily_train_agg['meter_reading-idxmax']]\ndaily_train_agg.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def show_building(building, energy_aspects=None):\n    fig = px.line(daily_train.loc[(slice(None), building, slice(None)), :].reset_index(),\n                  x='date',\n                  y='meter_reading',\n                  color='meter',\n                  render_mode='svg')\n    if energy_aspects:\n        if 'electricity' not in energy_aspects:\n            fig['data'][0].visible = 'legendonly'\n        if 'chilledwater' not in energy_aspects:\n            fig['data'][1].visible = 'legendonly'\n        if 'steam' not in energy_aspects:\n            fig['data'][2].visible = 'legendonly'\n        if 'hotwater' not in energy_aspects:\n            fig['data'][3].visible = 'legendonly'\n    fig.update_layout(title='Building ID: {}'.format(building))        \n    fig.show()\n    display(building_metadata[building_metadata['building_id']==building])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Electricity"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of days that a building has the maximum electricity consumption of all the buildings:\\n')\nprint(daily_train_agg[daily_train_agg['meter'] == 'electricity']['building_id_max'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The max values of electricity are caused by only 6 buildings. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"daily_train_electricity = daily_train_agg[daily_train_agg['meter']=='electricity'].copy()\ndaily_train_electricity['building_id_max'] = pd.Categorical(daily_train_electricity['building_id_max'])\nfig_daily_electricity = px.scatter(daily_train_electricity,\n                                   x='date',\n                                   y='meter_reading-max',\n                                   color='building_id_max',\n                                   render_mode='svg')\nfig_daily_electricity.update_layout(title='Maximum consumption values for the day and energy aspect')\nfig_daily_electricity.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"show_building(803, ['electricity'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a building typified as an educational center.\nTherefore, there should not be an industrial electricity consumption.\n\nThe electricity consumption has an average of about 120 000 kWh for 180 000 ft² each day.\n\nThis is 0.66 kWh/ft²/day or 243 kWh/ft²/year.\nWhere the typical consumptions is about 20 kWh/ft²/year.\n\nThis building has a 10 fold more electricity consumption than the typical one.\nMaybe the meter or the software that reads it is not configured correctly."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"show_building(801, ['electricity'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a building typified as an educational center.\nTherefore, there should not be an industrial electricity consumption.\n\nThe electricity consumption has an average of about 110 000 kWh for 500 000 ft² each day.\n\nThis is 0.22 kWh/ft²/day or 80 kWh/ft²/year.\nWhere the typical consumptions is about 20 kWh/ft²/year.\n\nThis building has a 4 fold more electricity consumption than the typical one.\nMaybe the meter or the software that reads it is not configured correctly."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"show_building(799, ['electricity'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This building has a very erractic behaviour with\nsome periods of no data,\na period of increasing consumption,\nand two periods of stable consumptions with a weekly pattern.\n\nThe first of the two periods has an average of 75 000 kWh for 500 000 ft² each day.\nThe second has an average of 175 000 kWh.\n\nThis is 0.15-0.35 kWh/ft²/day or 55-130 kWh/ft²/year.\nWhere the typical consumptions is about 20 kWh/ft²/year.\n\nBoth specific consumptions are from 3 to 7 times the typical.\n\nThis center could be new. The consumption in the summer period is due to office and equipments.\nThen in half september, the course begins. Something changed in November.\n\nThe most problable is that the building is not new, you don't build buildings for half a million square feet in one year. Then, meters have been added progressively, and some of them are not well configured."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"show_building(1088, ['electricity'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This building has a steady period in the cold part of the year and a very noisy period in the hot part of the year.\n\nMaybe the total is the sum of more than one meter, and maybe the meter which measures the HVAC is not well configured."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"show_building(993, ['electricity'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This building has a very steady daily consumption with a weekly pattern, and without a seasonal pattern.\n\nIt has some vacational pauses at July and December, and some peaks."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"show_building(794, ['electricity'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This building has a very steady daily consumption with a weekly pattern, and without a seasonal pattern.\n\nIt has an average of 95 000 kWh for 750 000 ft² each day.\nThe second has an average of 175 000 kWh.\n\nThis is 0.13 kWh/ft²/day or 45 kWh/ft²/year.\nWhere the typical consumptions is about 20 kWh/ft²/year.\nThe specific consumption is more than twice the typical.\n\nSome peak has the maximum of all the buildings at a single day."},{"metadata":{},"cell_type":"markdown","source":"## Chilledwater"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of days that a building has the maximum chilledwater consumption of all the buildings:\\n')\nprint(daily_train_agg[daily_train_agg['meter'] == 'chilledwater']['building_id_max'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The max values of electricity are caused by only 10 buildings. "},{"metadata":{"trusted":true},"cell_type":"code","source":"daily_train_chilledwater = daily_train_agg[daily_train_agg['meter']=='chilledwater'].copy()\ndaily_train_chilledwater['building_id_max'] = pd.Categorical(daily_train_chilledwater['building_id_max'])\nfig_daily_chilledwater = px.scatter(daily_train_chilledwater,\n                                    x='date',\n                                    y='meter_reading-max',  \n                                    color='building_id_max', \n                                    render_mode='svg')\nfig_daily_chilledwater.update_layout(title='Maximum consumption values for the day and energy aspect')\nfig_daily_chilledwater.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Only buildings 778 and 1088 have aberrant values."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_building(778, ['chilledwater'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The max consumption of the non-aberrant buildings is about 700 000 kWh.\nThe consumption of this building is 25 times this.\nAnd only for a span of two months.\n\nProbably, the measure is wrong."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_building(1088, ['chilledwater'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This building has a typical hot part of the year chilledwater consumption,\nwith a maximum of 100 000 kWh.\nIn some other short periods has very high peaks of about a hundred times more."},{"metadata":{},"cell_type":"markdown","source":"## Steam"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of days that a building has the maximum steam consumption of all the buildings:\\n')\nprint(daily_train_agg[daily_train_agg['meter'] == 'steam']['building_id_max'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The max values of electricity are caused by only 4 buildings. "},{"metadata":{"trusted":true},"cell_type":"code","source":"daily_train_steam = daily_train_agg[daily_train_agg['meter']=='steam'].copy()\ndaily_train_steam['building_id_max'] = pd.Categorical(daily_train_steam['building_id_max'])\nfig_daily_steam = px.scatter(daily_train_steam,\n                             x='date',\n                             y='meter_reading-max',\n                             color='building_id_max',\n                             render_mode='svg')\nfig_daily_steam.update_layout(title='Maximum consumption values for the day and energy aspect')\nfig_daily_steam.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The building 1099 has a very large consumption and irreal values: 450 000 000 kWh.\n\nThe buildings 1168 and 1197 have large consumption: 3 000 000 kWh.\n\nAnd the building 1148 has values above 1 000 000 kWh."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_building(1099, ['steam'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When the value is not gargantuan, it is below 50 000 kWh.\n\nProbably, as it is something like a three order magnitude difference,\nis a unit misconfiguration, like kWh instead of Wh."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_building(1168, ['steam'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A 1 000 000 kWh daily average is too big up to a 500 000 ft² office."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_building(1197, ['steam'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Too much steam energy for a not so big building."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_building(1148, ['steam'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Big consumption even for a very huge office."},{"metadata":{},"cell_type":"markdown","source":"## Hotwater"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Number of days that a building has the maximum hotwater consumption of all the buildings:\\n')\nprint(daily_train_agg[daily_train_agg['meter'] == 'hotwater']['building_id_max'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The max values of electricity are caused by only 7 buildings.\nPractically, two of them."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"daily_train_hotwater = daily_train_agg[daily_train_agg['meter']=='hotwater'].copy()\ndaily_train_hotwater['building_id_max'] = pd.Categorical(daily_train_hotwater['building_id_max'])\nfig_daily_hotwater = px.scatter(daily_train_hotwater,\n                                x='date',\n                                y='meter_reading-max',\n                                color='building_id_max',\n                                render_mode='svg')\nfig_daily_hotwater.update_layout(title='Maximum consumption values for the day and energy aspect')\nfig_daily_hotwater.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_building(1021, ['hotwater'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Very big value."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_building(1331, ['hotwater'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's a very big consumption for an education building."},{"metadata":{},"cell_type":"markdown","source":"# Conclusion"},{"metadata":{},"cell_type":"markdown","source":"Taking only the buildings that consume more than the others,\ncould be seen that there are a lot of measure scale errors.\n\nThe error could be:\n\n- The meter is not configured correctly. E.g., a bad voltage or current primary to secondary ratio.\n- The software has not the units configured correctly. E.g., MJ/kg for steam.\n- The software has not the decimal digits configured correctly.\n- Using a power variable instead of an energy one.\n\nThe measure could be done with an unique meter, or the sum of several of them.\n\nSome changes over time, values go to zero or the scale is changed,\nindicates that some buildings have more than one meter.\nOne error in one meter and the overall measure is garbage.\n\nThis notebook has only analised the outliers that influence the maximum consumption in a daily basis.\nThis is only the tip of the iceberg.\nA sound analysis should be done to detect and correct these outliers.\n\nA solution to avoid scale errors is to normalize the values from 0 to 1, for each building and for each energy aspect."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}