{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <div style=\"text-align: center\"> ASHRAE - Great Energy Predictor III\n### <div style=\"text-align: center\"> How much does it cost to cool a skyscraper in the summer?\n\n<img src=\"https://percentotech.com/wp-content/uploads/2019/07/Smart-Buildings.jpg\">\n\n   Developing energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall. One issue preventing more aggressive growth of the energy markets are the lack of cost-effective, accurate, and scalable procedures for forecasting energy use.\n\n  \n **I hope this kernel helpful **\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"top\"></a> <br>\n## Notebook  Content\n\n1. [Understand the Competition](#1)\n1. [Import](#2)\n1. [Load Data](#3)\n1. [Data Description](#4)\n1. [Visualization](#5)\n1. [References](#8)\n"},{"metadata":{},"cell_type":"markdown","source":"**<a id=\"1\"></a> <br>**\n## 1- Understand the Competition"},{"metadata":{},"cell_type":"markdown","source":"Assessing the value of energy efficiency improvements can be challenging as there's no way to truly know how much energy a building would have used without the improvements. The best we can do is to build counterfactual models. Once a building is overhauled the new (lower) energy consumption is compared against modeled values for the original building to calculate the savings from the retrofit. More accurate models could support better market incentives and enable lower cost financing.\n\nThis competition challenges you to build these counterfactual models across four energy types based on historic usage rates and observed weather. The dataset includes three years of hourly meter readings from over one thousand buildings at several different sites around the world.\n\nThankfully, significant investments are being made to improve building efficiencies to reduce costs and emissions. So, are the improvements working? That’s where you come in. Current methods of estimation are fragmented and do not scale well. Some assume a specific meter type or don’t work with different building types.\n\nDeveloping energy savings has two key elements: Forecasting future energy usage without improvements, and forecasting energy use after a specific set of improvements have been implemented, like the installation and purchase of investment-grade meters, whose prices continue to fall. One issue preventing more aggressive growth of the energy markets are the lack of cost-effective, accurate, and scalable procedures for forecasting energy use.\n\nIn this competition, you’ll develop accurate predictions of metered building energy usage in the following areas: chilled water, electric, natural gas, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe.\n\nWith better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies."},{"metadata":{},"cell_type":"markdown","source":"**<a id=\"2\"></a> <br>**\n## 2- Import"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\n\nimport numpy as np\nimport os\nimport seaborn as sns\n\nimport matplotlib.pyplot as plt\n\nfrom statsmodels.tsa.stattools import adfuller","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## 3- Load Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/ashrae-energy-prediction/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Total File sizes')\nprint('-'*10)\nfor f in os.listdir('../input/ashrae-energy-prediction'):\n    if 'zip' not in f:\n        print(f.ljust(30) + str(round(os.path.getsize('../input/ashrae-energy-prediction/' + f) / 10000000, 2)) + 'MB')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%time\ntrain = pd.read_csv('/kaggle/input/ashrae-energy-prediction/train.csv',index_col= 'timestamp', parse_dates=True)\ntest = pd.read_csv('/kaggle/input/ashrae-energy-prediction/test.csv',index_col= 'timestamp', parse_dates=True)\nsample_sub = pd.read_csv('/kaggle/input/ashrae-energy-prediction/sample_submission.csv')\nbuilding_metadata = pd.read_csv('/kaggle/input/ashrae-energy-prediction/building_metadata.csv')\nweather_train = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_train.csv',index_col= 'timestamp', parse_dates=True)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"print('Train # rows: ',train.shape[0])\nprint('Train # Columns: ',train.shape[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('building_metadata # rows: ',building_metadata.shape[0])\nprint('building_metadata # Columns: ',building_metadata.shape[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('weather_train # rows: ',weather_train.shape[0])\nprint('weather_train # Columns: ',weather_train.shape[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n## 4- Data Description"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(list(train.columns))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**train.csv**\n- **building_id **- Foreign key for the building metadata.\n- **meter** - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, hotwater: 3}. Not every building has all meter types.\n- **timestamp** - When the measurement was taken\n- **meter_reading** - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(list(test.columns))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**test.csv**\nThe submission files use row numbers for ID codes in order to save space on the file uploads. test.csv has no feature data; it exists so you can get your predictions into the correct order.\n\n- row_id - Row id for your submission file\n- building_id - Building id code\n- meter - The meter id code\n- timestamp - Timestamps for the test data period"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(list(building_metadata.columns))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**building_meta.csv**\n- site_id - Foreign key for the weather files.\n- building_id - Foreign key for training.csv\n- primary_use - Indicator of the primary category of activities for the building based on EnergyStar property type definitions\n- square_feet - Gross floor area of the building\n- year_built - Year building was opened\n- floor_count - Number of floors of the building"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(list(sample_sub.columns))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**sample_submission.csv**\nA valid sample submission.\n\nAll floats in the solution file were truncated to four decimal places; we recommend you do the same to save space on your file upload.\nThere are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nulls = building_metadata.isnull().sum() # Sum of missing values\nnulls = nulls[nulls > 0]  \nnulls.sort_values(inplace=True)\nnulls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nullsWeather = weather_train.isnull().sum() # Sum of missing values\nnullsWeather = nullsWeather[nullsWeather > 0]  \nnullsWeather.sort_values(inplace=True)\nnullsWeather","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n## 5- Visulization of data"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(7,7))\n \nsns.countplot(y= building_metadata.primary_use,palette=\"Set2\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(7,7))\n \nsns.countplot(y= building_metadata.site_id,palette=\"Set2\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nsns.distplot(building_metadata.year_built, bins=25, hist=True,kde=False, rug=False ).set_title(\"Histogram of Year Built\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(building_metadata.square_feet, bins=25, hist=True,kde=False, rug=False ).set_title(\"Histogram of square_feet\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(building_metadata.floor_count, bins=25, hist=True,kde=False, rug=False ).set_title(\"Histogram of floor_count\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.building_id.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(weather_train.air_temperature, bins=25, hist=True,kde=False, rug=False ).set_title(\"Histogram of Air Temperature\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(weather_train.sea_level_pressure, bins=25, hist=True,kde=False, rug=False ).set_title(\"Histogram of Sea Level Pressure\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(weather_train.wind_speed, bins=25, hist=True,kde=False, rug=False ).set_title(\"Histogram of wind Speed\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Weather = weather_train.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Weather.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Example for site 1 with train"},{"metadata":{"trusted":true},"cell_type":"code","source":"trainsite1 = train[train['building_id'] == 0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainsite1.meter_reading.plot(figsize=(16,8))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Example for site 1 with Weather"},{"metadata":{"trusted":true},"cell_type":"code","source":"site1 = Weather[Weather['site_id'] == 0 ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"site1.air_temperature.plot(figsize=(16,8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nsite1.air_temperature.plot(kind = 'kde')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check Stationary\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def test_stationa(data):\n\n    rolmean = data.rolling(window = 10).mean()\n    \n    #plotting rolling statistics\n    original = plt.plot(data, color = 'blue', label = 'Original')\n    mean = plt.plot(rolmean, color = 'red', label = 'Rolling Mean')\n    \n    plt.legend()\n    plt.title('Rolling  Mean')\n    plt.show()\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_stationa(site1.air_temperature)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"site1.dew_temperature.plot(figsize=(16,8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"site1.sea_level_pressure.plot(figsize=(16,8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(11,11))\ncorrelations = Weather.corr()\nmask = np.zeros_like(correlations)\nmask[np.triu_indices_from(mask)] = True \nwith sns.axes_style(\"white\"):\n    ax = sns.heatmap(correlations, mask=mask, vmax=.9, square=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**<a id=\"6\"></a> <br>**\n## 6- References\n[ASHRAE - Great Energy Predictor III](https://www.kaggle.com/c/ashrae-energy-prediction/data)"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}