{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ASHARE- Househould energy prediction\n## The goal of this project is to predict household energy consumption using the provided data files. \n### First, the given data must be overviewed, analyzed and compared, which will be done in this exploratory data analysis notebook.\n\n"},{"metadata":{},"cell_type":"markdown","source":"### Here, packages are imported"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\n#Importing plotting tool\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### It can be seen that the complete data source consists of 6 data files. The test.csv is the data file that will be used to train the machine learning algorithm and it is the first file to be analyzed:"},{"metadata":{},"cell_type":"markdown","source":"# File \"train.csv\"\n#### The \"train.csv\" looks like this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/ashrae-energy-prediction/train.csv\")\ndf_train.head(5)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first column. \"building_id\" contains the building identifications. Each number represents one building and there are There are presumably the same amount of unique building ids as in building_metadata.\n\nThe second column, \"meter\", contains 4 different numbers: 0, 1, 2 and 3. Each number correspond to one source of energy consumption.\n\n* 0 = electricty\n* 1 = chilled water\n* 2 = steam\n* 3 = hot water\n\nFirst, the amount of building_id readings can be analyzed. It is always a good idea to see if there are any NaN values. The following bar chart illustrates the amount of meter readings for each building ID. The building IDs are ordered from the highest amount of meter readings to the lowest amount of meter readings. From this box chart, it can be concluded that almost none of the buildings contain all possible meter readings. This might be caused by the fact that not all meter types are installed in all buildings, which means that not all builinds use the same source of energy.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique buildings: \", len(df_train.drop_duplicates(subset = \"building_id\")))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Amount of rows with NaN values: \", len(df_train))\nprint(\"Amount of rows without NaN values: \", len(df_train.dropna()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Bar chart for the overall amount of meter readings for each building ID\n\ndf_train[\"building_id\"].value_counts().head()\nxtick_values = []\n\ndf_train[\"building_id\"].value_counts().plot.bar(figsize = (15,6), xlabel = \"Building ID\", ylabel = \"Number of meter readings\", fontsize = 10, xticks = [0], title = \"Number of meter readings for each building ID\")\n\n\n#Alternative plt box chart that I cannot get to work, so I just saved it here for now\n#plt.bar(df_train[\"building_id\"].value_counts(), height = 50000)\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Next, the values of the meter readings will be analyzed"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train[\"meter\"].value_counts().plot.bar( xlabel = \"amount of readings\", ylabel = \"meter type\", title = \"Amount of readings for each meter type\")\n#plt.bar((df_train[\"meter\"].value_counts()), height = 10000)\n\ndf_train.drop([\"building_id\", \"timestamp\"], axis = 1).groupby([\"meter\"]).mean().plot.bar(xlabel = \"mean value of the meter readings\", title = \"Mean meater reading for each meter type\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The first bar chart shows how many meter readings that are made by each meter type. It can be seen most of the readings are made by 0 = electricity. The second bar chart illustrates the mean meter reading (i.e. the electricity consumption) for each meter type. For 2 = steam, the results are significantly higher, meaning that steam heating uses the most electricity. \n\n### The following graph shows all meter readings during the year"},{"metadata":{"trusted":true},"cell_type":"code","source":"xtickx = [0]\n\n#for i in range(len(df_train)):\n    #xtickx += [df_train[\"timestamp\"][i]]\ndf_train.filter([\"meter_reading\", \"timestamp\"]).plot(figsize = (20,7), title = \"all meter readings for each building id\" )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### This graph is not very good, since it displays every meter reading for every building. This means that there are multiple data points for each time stamp.  \n\n### In addition to this, it is difficult to distinguish the different meter types, since they have different mean values. Therefore. the different meter types will be illustrated individually."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Creating one dataframe for each meter type\n\ndf_train_0 = df_train.loc[df_train[\"meter\"] == 0]\ndf_train_1 = df_train.loc[df_train[\"meter\"] == 1]\ndf_train_2 = df_train.loc[df_train[\"meter\"] == 2]\ndf_train_3 = df_train.loc[df_train[\"meter\"] == 3]\n\n#Calculating the mean readings of the buildings\n#In this case, it can be seen that the building_id will not make any sense anymore, \n#since it calculates the mean of the IDs. It doesnt matter since the mean meter_reading is of interest\n\ndf_train_0.groupby(by = \"timestamp\").mean().filter([\"timestamp\", \"meter_reading\"]).plot(figsize =(15,7), ylabel = \"mean meter readings\", title = \"mean hourly electricity readings (index 0)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### In this graph, it can be noted that electricity meter readings increase during the summmer, and slightliy decrease during fall, winter and spring. This might be due to increased usage of airconditioning during the summer. Note that there are momentary peaks that differ vastly compared to the mean values. "},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_1.groupby(by = \"timestamp\").mean().filter([\"timestamp\", \"meter_reading\"]).plot(figsize =(15,7), ylabel = \"mean meter readings\", title = \"mean hourly chilled water readings (index 1)\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_2.groupby(by = \"timestamp\").mean().filter([\"timestamp\", \"meter_reading\"]).plot(figsize =(15,7), ylabel = \"mean meter readings\", title = \"mean daily steam readings (index 2)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### It can be noted that there is something strange about the steam meter readings. It looks very similar to the overall meter reading graph presented before. Also notice that the values are very high compared to the other meter readings. Perhaps there is a building/buildings that use a lot of energy for steam? This can explain why the bar chart indicted that the meter readings where so much higher compared to the rest of the meter readings. This must be analyzed further and potentailly processed or removed, it if is an anomaly."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_3.groupby(by = \"timestamp\").mean().filter([\"timestamp\", \"meter_reading\"]).plot(figsize =(15,7), ylabel = \"mean meter readings\", title = \"mean hourly hot water readings (index 3)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The hot water usage decreases duing the summer and increases during the winter. This is expected as the cold weather makes people shower hotter and longer. There might be some buildings that use hot water and radiators as heating."},{"metadata":{},"cell_type":"markdown","source":"# File \"building_metadata\"\nThis file contains information about each building ID. The first column, \"site_id\" contains an index which corresponds to a building site. In total there are 16 different sites that contain the building IDs.\n\nThe first thing that is investigated is the amount of buildings for each site. It can be seen that site 3 has almost double the amount of buildings compared to the second biggest site. \n\nIt is plausibile that the primary use affects the amount of buildings,so this will be investigated next.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#print(\"Amount of sites: \",len(df_building_metadata.drop_duplicates(subset = \"site_id\")))\ndf_building_metadata = pd.read_csv(\"../input/ashrae-energy-prediction/building_metadata.csv\")\ndf_building_metadata.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_building_metadata[\"site_id\"].value_counts(sort = False).plot.bar(figsize = (15, 7), xlabel = \"Site IDs\", ylabel = \"Amount of building IDs\", fontsize = 10, title = \"Number of buildings on each site\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_building_metadata[\"primary_use\"].value_counts().plot.bar(figsize = (14,5), xlabel = \"Primary use\", ylabel = \"Number of buildings\", fontsize = 10, rot = 90, title = \"Number of buildings used for a particual reason\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#print(df_building_metadata.set_index(keys=[\"site_id\", \"building_id\"]))\n#print(df_building_metadata.filter([\"site_id\", \"primary_use\"]))\n\ndf_building_metadata.filter([\"site_id\", \"primary_use\"]).value_counts().sort_values().reset_index().pivot(index = \"site_id\", columns = \"primary_use\", values = 0).plot.bar(stacked = True,figsize = (20,10), xlabel = \"Site ID\", ylabel = \"Number of buildings\", fontsize = 10, rot = 90, title = \"Number of buildings used for a particual reason\" )\n#","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## What can be concluded from this? There are many types uses for each site, which means that energy usage patterns will most likely vary depending on the primary use. However, it can be seen that education is a big part of many sites."},{"metadata":{},"cell_type":"markdown","source":"# File weather.csv\n### This file looks like the following dataframe. It can be seen that for each site ID and timestamp, numerous recordings of different weather data has been done.\n\n### First of all, which recordings done might have the greatest impact on the meter readings? Temperature might affect the electricity usage because of AC usage. Similarly, steam meter readings will probably decrease"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather = pd.read_csv(\"../input/ashrae-energy-prediction/weather_train.csv\")\ndf_weather.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The air temperature is shown in the following graph"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather.filter([\"timestamp\", \"air_temperature\"]).groupby(by = \"timestamp\").mean().plot(figsize = (15,7), xlabel = \"mean air temperature\", title = \"Mean air temperature for all sites\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### It can be seen that the air temperature increases substantially during the summer, and is somewhat low during the winter. This information tells us that the climate is relatively warm, which can help to decide what factors are important. Also, energy usage for different things is different depending on climate. For example, people living in cold areas will probably not even have airconditioning installed, which means that the electricity usage is different compared to people who live in warmer areas. It would be good to know if the buildings have automatic adjustment of the indoor temperature, and in that case what parameters decide it."},{"metadata":{},"cell_type":"markdown","source":"### Correlation between temperature and different meter values:"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nprint(\"Correlation hot water meter and air temperature: \",df_weather[\"air_temperature\"].corr(df_train_3[\"meter_reading\"]))\nprint(\"Correlation steam meter and air temperature: \",df_weather[\"air_temperature\"].corr(df_train_2[\"meter_reading\"]))\nprint(\"Correlation chilled water meter and air temperature: \",df_weather[\"air_temperature\"].corr(df_train_1[\"meter_reading\"]))\nprint(\"Correlation electricity meter and air temperature: \",df_weather[\"air_temperature\"].corr(df_train_0[\"meter_reading\"]))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}