{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThe aim of this notebook is to analyse the relation of the meter readings with the heating degree day (HDD), which quantify the demand for energy needed to heat a building, and with the cooling degree day (CDD), which quantifies demand for air conditioning.\n\nThe theory is that the heating requirements for a given building at a specific location are considered to be directly proportional to the number of HDD at that location.\n\nThe normal use of this variable to this day is to use a daily aggregation. But the data is in a hourly basis and a smaller aggregation is better to analyse the data. Then, heating degree hour HDH and cooling degree hour CDH is used in this notebook.\n\n![i1](https://upload.wikimedia.org/wikipedia/commons/d/da/United_States_Heating_Degree_Day_map%2C_1961-1990.jpg)\n\n![i2](https://upload.wikimedia.org/wikipedia/commons/6/6c/United_States_Cooling_Degree_Day_map%2C_1961-1990.jpg)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# !pip install nb_black\n# %load_ext nb_black\nimport numpy as np\nimport pandas as pd\nimport colorlover as cl\nimport plotly.express as px\nimport plotly.io as pio\nimport seaborn as sns\nfrom IPython.core.display import display, HTML\nimport pickle\nfrom tqdm.auto import tqdm","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"pio.templates.default = \"plotly_white\"\ncategory_orders = {\"meter\": [\"electricity\", \"chilledwater\", \"steam\", \"hotwater\"]}\ncolor_discrete_map_meter = {\n    \"electricity\": \"#7844c5\",\n    \"chilledwater\": \"#83c9e9\",\n    \"steam\": \"#dfdeeb\",\n    \"hotwater\": \"#f2957a\",\n}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"color_discrete_map_site = {\n    # Europe/London -> Blues\n    \"1 London\": cl.scales[\"3\"][\"seq\"][\"Blues\"][2],  # London, Europe/London\n    \"5 Leicester\": cl.scales[\"3\"][\"seq\"][\"Blues\"][1],  # Leicester, Europe/London\n    # Europe/Dublin -> Greens\n    \"12 Dublin\": cl.scales[\"3\"][\"seq\"][\"Greens\"][2],  # Dublin\n    # Canada/Eastern -> Reds\n    \"7 Montréal\": cl.scales[\"3\"][\"seq\"][\"Reds\"][2],  # Montréal, Canada/Eastern\n    \"11 Montréal\": cl.scales[\"3\"][\"seq\"][\"Reds\"][1],  # Montréal, Canada/Eastern\n    # US/Eastern -> Purples\n    \"0 Jacksonville\": cl.scales[\"6\"][\"seq\"][\"Purples\"][5],  # Jacksonville, US/Eastern\n    \"3 Philadelphia\": cl.scales[\"6\"][\"seq\"][\"Purples\"][4],  # Philadelphia, US/Eastern\n    \"6 Philadelphia\": cl.scales[\"6\"][\"seq\"][\"Purples\"][3],  # Philadelphia, US/Eastern\n    \"8 Jacksonville\": cl.scales[\"6\"][\"seq\"][\"Purples\"][2],  # Jacksonville, US/Eastern\n    \"14 Philadelphia\": cl.scales[\"6\"][\"seq\"][\"Purples\"][1],  # Philadelphia, US/Eastern\n    \"15 Pittsburgh\": cl.scales[\"6\"][\"seq\"][\"Purples\"][0],  # Pittsburgh, US/Eastern\n    # US/Central -> Greys\n    \"9 San Antonio\": cl.scales[\"3\"][\"seq\"][\"Greys\"][2],  # San Antonio, US/Central\n    \"13 Minneapolis\": cl.scales[\"3\"][\"seq\"][\"Greys\"][1],  # Minneapolis, US/Central\n    # US/Arizona -> Yellows\n    \"2 Phoenix\": cl.scales[\"3\"][\"seq\"][\"YlOrBr\"][0],  # Phoenix, US/Arizona\n    # US/Pacific -> Oranges\n    \"4 San Francisco\": cl.scales[\"3\"][\"seq\"][\"Oranges\"][2],  # San Francisco, US/Pacific\n    \"10 Las Vegas\": cl.scales[\"3\"][\"seq\"][\"Oranges\"][1],  # Las Vegas, US/Pacific\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Import data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"with open(\"../input/ashrae-data-wrangling-train/train.pickle\", \"rb\") as f:\n    train = pickle.load(f)\nwith open(\"../input/ashrae-data-wrangling-train/timeseries.pickle\", \"rb\") as f:\n    timeseries = pickle.load(f)\nwith open(\n    \"../input/ashrae-data-wrangling-building-metadata/building_metadata.pickle\", \"rb\"\n) as f:\n    building_metadata = pickle.load(f)\nwith open(\"../input/ashrae-data-wrangling-weather/weather_train.pickle\", \"rb\") as f:\n    weather_train = pickle.load(f)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Degree hours through the year"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"weather_train[\"site\"] = weather_train[\"site_id\"]\nweather_train[\"site\"].cat.rename_categories(\n    {\n        \"0\": \"0 Jacksonville\",\n        \"1\": \"1 London\",\n        \"2\": \"2 Phoenix\",\n        \"3\": \"3 Philadelphia\",\n        \"4\": \"4 San Francisco\",\n        \"5\": \"5 Leicester\",\n        \"6\": \"6 Philadelphia\",\n        \"7\": \"7 Montréal\",\n        \"8\": \"8 Jacksonville\",\n        \"9\": \"9 San Antonio\",\n        \"10\": \"10 Las Vegas\",\n        \"11\": \"11 Montréal\",\n        \"12\": \"12 Dublin\",\n        \"13\": \"13 Minneapolis\",\n        \"14\": \"14 Philadelphia\",\n        \"15\": \"15 Pittsburgh\",\n    },\n    inplace=True,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"weather_train[\"month\"] = weather_train[\"timestamp\"].dt.month\n\nweather_train_month_columns = [\"month\", \"site\",] + [\n    dh for dh in weather_train.columns.to_list() if \"degree_hours\" in dh\n]\n\nweather_train_month = (\n    weather_train[weather_train_month_columns].groupby([\"month\", \"site\"]).agg([np.sum])\n)\nweather_train_month.reset_index(inplace=True)\nweather_train_month = weather_train_month.droplevel(1, axis=\"columns\")\nweather_train_month","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Heating degree hours"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"site_hdh = weather_train_month.groupby(\"site\").agg(\"sum\")\nsite_hdh_list = (\n    site_hdh.sort_values(by=\"heating_degree_hours_15\", ascending=True)\n    .reset_index()[\"site\"]\n    .to_list()\n)\nfig = px.area(\n    weather_train_month,\n    x=\"month\",\n    y=\"heating_degree_hours_15\",\n    category_orders={\"site\": site_hdh_list},\n    color=\"site\",\n    color_discrete_map=color_discrete_map_site,\n)\nfig.update_layout()\nfig.update_xaxes(dtick=1)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As all the sites are in the northern hemisphere, the heating requirement is low in summer and high in winter.\n\nThe sites are ordered by their total HDH through the year. Then, the sites in Florida have a lesser heating requirement that the ones in Canada.\n\nHottest sites doesn't have heating requirements from May to September. Even sites as Canada don't have heating requirement for July and August.\n\nOther sites, like the ones in the British islands, they have some heating requirements in summer too."},{"metadata":{},"cell_type":"markdown","source":"## Cooling degree hours"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"site_cdh = weather_train_month.groupby(\"site\").agg(\"sum\")\nsite_cdh_list = (\n    site_cdh.sort_values(by=\"cooling_degree_hours_25\", ascending=True)\n    .reset_index()[\"site\"]\n    .to_list()\n)\nfig = px.area(\n    weather_train_month,\n    x=\"month\",\n    y=\"cooling_degree_hours_25\",\n    category_orders={\"site\": site_cdh_list},\n    color=\"site\",\n    color_discrete_map=color_discrete_map_site,\n)\nfig.update_layout()\nfig.update_xaxes(dtick=1)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As all the sites are in the northern hemisphere, the cooling requirement is high in summer and low in winter.\n\nThe sites are ordered by their total CDH through the year. Then, the sites in Arizona and Florida have a greater cooling requirement that the ones in the British islands.\n\nHottest sites have a lot of cooling requirements from May to September. The cooling requirements for sites in the British islands are very low.\n\nOnly sites in Florida and Texas has some cooling requirements in December. The cooling requirements in January are practically null."},{"metadata":{},"cell_type":"markdown","source":"# Correlation"},{"metadata":{},"cell_type":"markdown","source":"## Correlation meter_readings / air_temperature vs meter_readings / degree_hours\nDo the meter_readings have a better correlation with HDH or CDH than with air_temperature?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"degree_hours = [\"air_temperature\"] + [\n    dh for dh in weather_train.columns.to_list() if \"degree_hours\" in dh\n]\n\nfor degree_hour in degree_hours:\n    timeseries[degree_hour] = np.NaN","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_corr(index, building_id, meter):\n    ts = train[(train[\"building_id\"] == building_id) & (train[\"meter\"] == meter)][\n        \"meter_reading\"\n    ]\n    site = building_metadata[building_metadata[\"building_id\"] == building_id][\n        \"site_id\"\n    ].values[0]\n    weather_train_site = weather_train[weather_train[\"site_id\"] == site]\n    for degree_hour in degree_hours:\n        dh = weather_train_site[degree_hour]\n        #         print(index, building_id, meter, degree_hour, len(ts), len(dh))\n        timeseries.loc[index, degree_hour] = np.ma.corrcoef(\n            np.ma.masked_invalid(ts), np.ma.masked_invalid(dh)\n        )[0, 1]\n\n\n#         print(np.ma.corrcoef(np.ma.masked_invalid(ts), np.ma.masked_invalid(dh)))\n\n\n# get_corr(0, \"1\", \"electricity\")\n# timeseries","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"for index, row in tqdm(timeseries.iterrows(), total=timeseries.shape[0]):\n    get_corr(index, row[\"building_id\"], row[\"meter\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"timeseries_heating = timeseries[\n    [\"building_id\", \"meter\", 'air_temperature']\n    + [hdh for hdh in weather_train.columns.to_list() if \"heating_degree_hours\" in hdh]\n]\ntimeseries_heating = pd.wide_to_long(\n    timeseries_heating,\n    stubnames=\"heating_degree_hours_\",\n    i=[\"building_id\", \"meter\", 'air_temperature'],\n    j=\"base_temperature\",\n)\ntimeseries_heating.reset_index(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.scatter(\n    timeseries_heating,\n    x=\"air_temperature\",\n    y=\"heating_degree_hours_\",\n    color=\"meter\",\n    animation_frame=\"base_temperature\",    \n    opacity=0.7,\n    marginal_x=\"violin\",\n    marginal_y=\"violin\",\n    hover_name=\"building_id\",\n    category_orders=category_orders,\n    color_discrete_map=color_discrete_map_meter,\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For a given correlation in air_temperature, there is not a global improve in the correlation in HDH.\n\nSome places have a slightly better correlation with HDH than with air_temperature, and some worse.\n\nWhen the base temperature increases, the correlation of HDH is closer to the correlation of air_temperature as they trends to have the same information."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"timeseries_cooling = timeseries[\n    [\"building_id\", \"meter\", 'air_temperature']\n    + [cdh for cdh in weather_train.columns.to_list() if \"cooling_degree_hours\" in cdh]\n]\ntimeseries_cooling = pd.wide_to_long(\n    timeseries_cooling,\n    stubnames=\"cooling_degree_hours_\",\n    i=[\"building_id\", \"meter\", 'air_temperature'],\n    j=\"base_temperature\",\n)\ntimeseries_cooling.reset_index(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.scatter(\n    timeseries_cooling,\n    x=\"air_temperature\",\n    y=\"cooling_degree_hours_\",\n    color=\"meter\",\n    animation_frame=\"base_temperature\",        \n    opacity=0.7,\n    marginal_x=\"violin\",\n    marginal_y=\"violin\",\n    hover_name=\"building_id\",\n    category_orders=category_orders,\n    color_discrete_map=color_discrete_map_meter,\n)\nfig.update_layout(yaxis=dict(range=[-1, 1]))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For a given correlation in air_temperature, there is not a global improve in the correlation in CDH.\n\nSome places have a slightly better correlation with CDH than with air_temperature, and some worse. The dispersion is a bit higher than in HDH."},{"metadata":{},"cell_type":"markdown","source":"## Correlation meter_readings / degree_hours"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"air_temperature_mean = (\n    timeseries[[\"building_id\", \"meter\", \"air_temperature\"]]\n    .groupby(\"meter\")\n    .agg(np.mean)\n)\nair_temperature_mean","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First of all, there is a correlation between energy and air temperature for each energy aspect.\n\nThe steam and hot water correlation is negative because more energy is needed then the temperature decreases."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.histogram(\n    timeseries_heating,\n    x=\"heating_degree_hours_\",\n    color=\"meter\",\n    facet_col=\"meter\",\n    animation_frame=\"base_temperature\",\n    nbins=100,\n    category_orders=category_orders,\n    color_discrete_map=color_discrete_map_meter,\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The greater the base temperature, the higher the correlation. This is because there are more information when the base temperature is high. When the base temperature is low, the heating requirement is null for more time span."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import plotly.graph_objects as go\n\nheating_mean = (\n    timeseries_heating.groupby([\"meter\", \"base_temperature\"]).agg(np.mean).reset_index()\n)\nfig = px.line(\n    heating_mean,\n    x=\"base_temperature\",\n    y=\"heating_degree_hours_\",\n    color=\"meter\",\n    category_orders=category_orders,\n    color_discrete_map=color_discrete_map_meter,\n)\nfig.update_layout(\n    shapes=[\n        go.layout.Shape(\n            type=\"line\",\n            x0=timeseries_heating[\"base_temperature\"].min(),\n            y0=-air_temperature_mean.loc[\"electricity\"][0],\n            x1=timeseries_heating[\"base_temperature\"].max(),\n            y1=-air_temperature_mean.loc[\"electricity\"][0],\n            line=dict(\n                color=color_discrete_map_meter[\"electricity\"], dash=\"dot\", width=1\n            ),\n        ),\n        go.layout.Shape(\n            type=\"line\",\n            x0=timeseries_heating[\"base_temperature\"].min(),\n            y0=-air_temperature_mean.loc[\"chilledwater\"][0],\n            x1=timeseries_heating[\"base_temperature\"].max(),\n            y1=-air_temperature_mean.loc[\"chilledwater\"][0],\n            line=dict(\n                color=color_discrete_map_meter[\"chilledwater\"], dash=\"dot\", width=1\n            ),\n        ),\n        go.layout.Shape(\n            type=\"line\",\n            x0=timeseries_heating[\"base_temperature\"].min(),\n            y0=-air_temperature_mean.loc[\"steam\"][0],\n            x1=timeseries_heating[\"base_temperature\"].max(),\n            y1=-air_temperature_mean.loc[\"steam\"][0],\n            line=dict(color=color_discrete_map_meter[\"steam\"], dash=\"dot\", width=1),\n        ),\n        go.layout.Shape(\n            type=\"line\",\n            x0=timeseries_heating[\"base_temperature\"].min(),\n            y0=-air_temperature_mean.loc[\"hotwater\"][0],\n            x1=timeseries_heating[\"base_temperature\"].max(),\n            y1=-air_temperature_mean.loc[\"hotwater\"][0],\n            line=dict(color=color_discrete_map_meter[\"hotwater\"], dash=\"dot\", width=1),\n        ),\n    ]\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The correlation approaches the correlation with the air temperature as the base temperature increases."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.histogram(\n    timeseries_cooling,\n    x=\"cooling_degree_hours_\",\n    color=\"meter\",\n    facet_col=\"meter\",\n    animation_frame=\"base_temperature\",\n    nbins=100,\n    category_orders=category_orders,\n    color_discrete_map=color_discrete_map_meter,\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The lesser the base temperature, the higher the correlation. This is because there are more information when the base temperature is low. When the base temperature is high, the cooling requirement is null for more time span."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"cooling_mean = (\n    timeseries_cooling.groupby([\"meter\", \"base_temperature\"]).agg(np.mean).reset_index()\n)\nfig = px.line(\n    cooling_mean,\n    x=\"base_temperature\",\n    y=\"cooling_degree_hours_\",\n    color=\"meter\",\n    category_orders=category_orders,\n    color_discrete_map=color_discrete_map_meter,\n)\nfig.update_layout(\n    shapes=[\n        go.layout.Shape(\n            type=\"line\",\n            x0=timeseries_cooling[\"base_temperature\"].min(),\n            y0=air_temperature_mean.loc[\"electricity\"][0],\n            x1=timeseries_cooling[\"base_temperature\"].max(),\n            y1=air_temperature_mean.loc[\"electricity\"][0],\n            line=dict(\n                color=color_discrete_map_meter[\"electricity\"], dash=\"dot\", width=1\n            ),\n        ),\n        go.layout.Shape(\n            type=\"line\",\n            x0=timeseries_cooling[\"base_temperature\"].min(),\n            y0=air_temperature_mean.loc[\"chilledwater\"][0],\n            x1=timeseries_cooling[\"base_temperature\"].max(),\n            y1=air_temperature_mean.loc[\"chilledwater\"][0],\n            line=dict(\n                color=color_discrete_map_meter[\"chilledwater\"], dash=\"dot\", width=1\n            ),\n        ),\n        go.layout.Shape(\n            type=\"line\",\n            x0=timeseries_cooling[\"base_temperature\"].min(),\n            y0=air_temperature_mean.loc[\"steam\"][0],\n            x1=timeseries_cooling[\"base_temperature\"].max(),\n            y1=air_temperature_mean.loc[\"steam\"][0],\n            line=dict(color=color_discrete_map_meter[\"steam\"], dash=\"dot\", width=1),\n        ),\n        go.layout.Shape(\n            type=\"line\",\n            x0=timeseries_cooling[\"base_temperature\"].min(),\n            y0=air_temperature_mean.loc[\"hotwater\"][0],\n            x1=timeseries_cooling[\"base_temperature\"].max(),\n            y1=air_temperature_mean.loc[\"hotwater\"][0],\n            line=dict(color=color_discrete_map_meter[\"hotwater\"], dash=\"dot\", width=1),\n        ),\n    ]\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The correlation approaches the correlation with the air temperature as the base temperature decreases."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}