{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Competition Objective:\nEnergy savings is one of the important area of focus our current world. Energy savings has two key elements: \n* Forecasting future energy usage without improvements\n* Forecasting energy use after a specific set of improvements have been implemented\n\nOnce we have implemented a set of improvements, the value of energy efficiency improvements can be challenging as there's no way to truly know how much energy a building would have used without the improvements. The best we can do is to build counterfactual models. his competition challenges you to build these counterfactual models across four energy types (chilled water, electricity, hot water, and steam) based on historic usage rates and observed weather. \n\n![EnergyForecast](https://encrypted-tbn0.gstatic.com/images?q=tbn%3AANd9GcRkkgt96WRGblQj9jG_28YXs6bLzBVmtiRKDAxQUT92ZZ8GH2dG)\nPicture Source: [wur.nl](https://www.wur.nl/en/project/Energy-demand-forecasting-for-urban-systems-1.htm)\n\n### Notebook Objective:\nThe objective of this notebook is to explore the data and make some inferences on the way.\n\n### Dataset:\nThe dataset includes three years of hourly meter readings from over one thousand buildings at several different sites around the world.\n\nWe are given two main files - `train.csv` and `test.csv` just like other competitions. In addition to these files, we also have couple of more files\n* building_metadata.csv - meta data about the buildings\n* weather.csv - weather information\n\nFirst let us load the train file and look at the top few rows to get an idea about the data.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport seaborn as sns\n\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\n\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\npd.set_option('max_columns', 100)\n\ndata_path = \"/kaggle/input/ashrae-energy-prediction/\"\n\ntrain_df = pd.read_csv(data_path + \"train.csv\")\ntrain_df[\"timestamp\"] = pd.to_datetime(train_df[\"timestamp\"], format='%Y-%m-%d %H:%M:%S')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have details about building id, meter type {0: electricity, 1: chilledwater, 2: steam, hotwater: 3}, timestamp and the meter reading. Meter reading is our target variable. \n\nNow let us get some generic information about the dataset."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from IPython.core.display import display, HTML\n\nnrows = train_df.shape[0]\nnbuildings = train_df[\"building_id\"].nunique()\nmax_rows_building = train_df[\"building_id\"].value_counts().values[0]\nmin_rows_building = train_df[\"building_id\"].value_counts().values[-1]\nmin_date = train_df[\"timestamp\"].min()\nmax_date = train_df[\"timestamp\"].max()\ndisplay(HTML(f\"\"\"
Number of rows in the dataset: {nrows:,}
\n
Number of buildings in the dataset: {nbuildings:,}
\n
Maximum of {max_rows_building:,} rows is available for a building
\n
Minimum of {min_rows_building:,} rows is available for a building
\n
Min date value in train data is {min_date}
\n
Max date value in train data is {max_date}
\n \"\"\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Meter Type Count\n\nLet us get the count of rows for each of the 4 meter type."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"cnt_srs = train_df[\"meter\"].value_counts()\ncnt_srs = cnt_srs.sort_index()\ncnt_srs.index = [\"Electricity\", \"ChilledWater\", \"Steam\", \"HotWater\"]\ntrace = go.Bar(\n x=cnt_srs.index,\n y=cnt_srs.values,\n marker=dict(\n color=\"#1E90FF\",\n ),\n)\n\nlayout = go.Layout(\n title=go.layout.Title(\n text=\"Number of rows for each meter type\",\n x=0.5\n ),\n font=dict(size=14),\n width=800,\n height=500,\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"meter\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Electricty meter type has the most number of rows (~12M) \n\n## Meter Reading Distribution\n\nNow let us look at the meter reading distribution for each of the meter types. Since the error metric is Root Mean Squared Logarithmic Error (RMSLE), let us plot the log of the meter reading."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from bokeh.layouts import gridplot\nfrom bokeh.plotting import figure, show, output_file\nfrom bokeh.io import output_notebook\n\noutput_notebook()\ndef make_plot(title, hist, edges, xlabel):\n p = figure(title=title, tools='', background_fill_color=\"#fafafa\")\n p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:],\n fill_color=\"#1E90FF\", line_color=\"white\", alpha=0.5)\n\n p.y_range.start = 0\n p.xaxis.axis_label = f'Log of {xlabel} meter reading'\n p.yaxis.axis_label = 'Probability'\n p.grid.grid_line_color=\"white\"\n return p\n\ntemp_df = train_df[train_df[\"meter\"]==0]\nhist, edges = np.histogram(np.log1p(temp_df[\"meter_reading\"].values), density=True, bins=50)\np1 = make_plot(\"Meter Reading Distribution for Electricity meter\", hist, edges, \"electricity\")\n\ntemp_df = train_df[train_df[\"meter\"]==1]\nhist, edges = np.histogram(np.log1p(temp_df[\"meter_reading\"].values), density=True, bins=50)\np2 = make_plot(\"Meter Reading Distribution for Chilled Water meter\", hist, edges, 'chill water')\n\ntemp_df = train_df[train_df[\"meter\"]==2]\nhist, edges = np.histogram(np.log1p(temp_df[\"meter_reading\"].values), density=True, bins=50)\np3 = make_plot(\"Meter Reading Distribution for Steam meter\", hist, edges, 'steam')\n\ntemp_df = train_df[train_df[\"meter\"]==3]\nhist, edges = np.histogram(np.log1p(temp_df[\"meter_reading\"].values), density=True, bins=50)\np4 = make_plot(\"Meter Reading Distribution for Hot Water meter\", hist, edges, 'hot water')\n\nshow(gridplot([p1,p2,p3,p4], ncols=2, plot_width=400, plot_height=400, toolbar_location=None))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Distribution of Meter Readings over time\n\nNow let us take some individual buildings and see how the meter readings has changed over time. First let us take the building with most number of readings in the training data `1298`"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp_df = train_df[train_df[\"building_id\"]==1298].reset_index(drop=True)\n\nimport datetime\n\ndef scatter_plot(cnt_srs, color):\n trace = go.Scatter(\n x=cnt_srs.index[::-1],\n y=cnt_srs.values[::-1],\n showlegend=False,\n marker=dict(\n color=color,\n ),\n )\n return trace\n\ntdf = temp_df[temp_df[\"meter\"]==0]\ncnt_srs = tdf[\"meter_reading\"]\ncnt_srs.index = tdf[\"timestamp\"]\ntrace1 = scatter_plot(cnt_srs, 'red')\n\ntdf = temp_df[temp_df[\"meter\"]==1]\ncnt_srs = tdf[\"meter_reading\"]\ncnt_srs.index = tdf[\"timestamp\"]\ntrace2 = scatter_plot(cnt_srs, 'blue')\n\ntdf = temp_df[temp_df[\"meter\"]==2]\ncnt_srs = tdf[\"meter_reading\"]\ncnt_srs.index = tdf[\"timestamp\"]\ntrace3 = scatter_plot(cnt_srs, 'green')\n\ntdf = temp_df[temp_df[\"meter\"]==3]\ncnt_srs = tdf[\"meter_reading\"]\ncnt_srs.index = tdf[\"timestamp\"]\ntrace4 = scatter_plot(cnt_srs, 'purple')\n\nsubtitles = [\"Meter reading over time for electricity meter for building 1298\",\n \"Meter reading over time for chill water meter for building 1298\",\n \"Meter reading over time for steam meter for building 1298\",\n \"Meter reading over time for hot water meter for building 1298\"\n ]\nfig = subplots.make_subplots(rows=4, cols=1, vertical_spacing=0.08,\n subplot_titles=subtitles)\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 2, 1)\nfig.append_trace(trace3, 3, 1)\nfig.append_trace(trace4, 4, 1)\nfig['layout'].update(height=1200, width=1000, paper_bgcolor='rgb(233,233,233)')\npy.iplot(fig, filename='meter-plots')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The training time frame is from January to December 2016 for this building\n* The electricity meter readings are generally in the range of 400 to 600 but becomes 0 at times in between which . \n* We can see an increase in the chill water meter from May to October of 2016 for this building and probably this might be summer time\n* Alternatively 2e can see a dip in the steam meter from May till October\n* we can see a spike in the hot water meter from November till February\n* 28th September to 20th October values are 0 for both electricity and steam meters. \n\nNow let us take one more building id `184` and see the plots"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"temp_df = train_df[train_df[\"building_id\"]==184].reset_index(drop=True)\n\ntdf = temp_df[temp_df[\"meter\"]==0]\ncnt_srs = tdf[\"meter_reading\"]\ncnt_srs.index = tdf[\"timestamp\"]\ntrace1 = scatter_plot(cnt_srs, 'red')\n\ntdf = temp_df[temp_df[\"meter\"]==1]\ncnt_srs = tdf[\"meter_reading\"]\ncnt_srs.index = tdf[\"timestamp\"]\ntrace2 = scatter_plot(cnt_srs, 'blue')\n\ntdf = temp_df[temp_df[\"meter\"]==2]\ncnt_srs = tdf[\"meter_reading\"]\ncnt_srs.index = tdf[\"timestamp\"]\ntrace3 = scatter_plot(cnt_srs, 'green')\n\ntdf = temp_df[temp_df[\"meter\"]==3]\ncnt_srs = tdf[\"meter_reading\"]\ncnt_srs.index = tdf[\"timestamp\"]\ntrace4 = scatter_plot(cnt_srs, 'purple')\n\nsubtitles = [\"Meter reading over time for electricity meter for building 184\",\n \"Meter reading over time for chill water meter for building 184\",\n \"Meter reading over time for steam meter for building 184\",\n \"Meter reading over time for hot water meter for building 184\"\n ]\nfig = subplots.make_subplots(rows=4, cols=1, vertical_spacing=0.08,\n subplot_titles=subtitles)\nfig.append_trace(trace1, 1, 1)\nfig.append_trace(trace2, 2, 1)\nfig.append_trace(trace3, 3, 1)\nfig.append_trace(trace4, 4, 1)\nfig['layout'].update(height=1200, width=1000, paper_bgcolor='rgb(233,233,233)')\npy.iplot(fig, filename='meter-plots')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Steam meter is not available for this building\n* Electricity meter is between 20 to 70 most of the times with a slight dip in the last 15 days of september\n\nPlease feel free to fork the notebook and check the distribution for other buildings as well.\n\n## Building Metadata\n\nNow let us explore the building metadata to get some idea. This data has 1449 buildings which is the same number as that of train data. Let us look at the top few rows."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"building_df = pd.read_csv(data_path + \"building_metadata.csv\")\nbuilding_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"cnt_srs = building_df[\"primary_use\"].value_counts()\n#cnt_srs = cnt_srs.sort_index()\ntrace = go.Bar(\n x=cnt_srs.index,\n y=cnt_srs.values,\n marker=dict(\n color=\"#1E90FF\",\n ),\n)\n\nlayout = go.Layout(\n title=go.layout.Title(\n text=\"Primary use of Buildings - Count\",\n x=0.5\n ),\n font=dict(size=14),\n width=1000,\n height=500,\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"meter\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Education is the one with most number of primary usage followed by office adn entertainment\n\n### Building floor count"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"cnt_srs = building_df[\"floor_count\"].value_counts()\n#cnt_srs = cnt_srs.sort_index()\ntrace = go.Bar(\n x=cnt_srs.index,\n y=cnt_srs.values,\n marker=dict(\n color=\"#1E90FF\",\n ),\n)\n\nlayout = go.Layout(\n title=go.layout.Title(\n text=\"Floor count in the buildings - Count\",\n x=0.5\n ),\n font=dict(size=14),\n width=1000,\n height=500,\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"meter\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Looks like as the floor count increases, the number of buildings count decreases.\n\n### Year Built of the buildings"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"cnt_srs = building_df[\"year_built\"].value_counts()\n#cnt_srs = cnt_srs.sort_index()\ntrace = go.Bar(\n x=cnt_srs.index,\n y=cnt_srs.values,\n marker=dict(\n color=\"#1E90FF\",\n ),\n)\n\nlayout = go.Layout(\n title=go.layout.Title(\n text=\"Year built of the buildings - Count\",\n x=0.5\n ),\n font=dict(size=14),\n width=1000,\n height=500,\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"meter\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We have buildings built from 1900 all the way upto 2017\n* Number of buildings built in 1976 seem to be of higher representation\n\n### Square feet of the buildings"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.scatter(building_df, y=\"square_feet\", x=\"year_built\", color=\"primary_use\", size=\"square_feet\")\nfig.update_layout(showlegend=True)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Weather Data\n\nNow let us explore the weather data "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"weather_df = pd.read_csv(data_path + \"weather_train.csv\", parse_dates=[\"timestamp\"])\nweather_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Air temperature distribution in train"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.line(weather_df, x='timestamp', y='air_temperature', color='site_id')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow looks very colorful :D\n\nOverall it seems that the temperature increases in all the sites towards the middle of the year and decreases at the end of year. \n\n### Site level air temperature distribution\n\nNow let us look at the plot individually at site level. Each represent the change in temperature over time in each of the site ids."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from bokeh.models import Panel, Tabs\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\n\noutput_notebook()\n\ndef get_plots(df):\n p = figure(plot_width=1000, plot_height=350, x_axis_type=\"datetime\", title=\"Air temperature distribution over time\")\n p.line(df['timestamp'], df['air_temperature'], color='navy', alpha=0.5)\n return p\n\ntab_list = []\nfor site in range(16):\n temp_df = weather_df[weather_df[\"site_id\"]==site]\n p = get_plots(temp_df)\n tab = Panel(child=p, title=f\"Site:{site}\")\n tab_list.append(tab)\n\ntabs = Tabs(tabs=tab_list)\nshow(tabs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Please change the tabs to see the temperature changes in each site. \n\n### Site level weather variables distribution\n\nNow let us plot the other weather variables as well in tab format. Now each main tab represents the variable and the internal tabs represent the site."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from bokeh.models import Panel, Tabs\nfrom bokeh.io import output_notebook, show\nfrom bokeh.plotting import figure\n\noutput_notebook()\n\ndef get_plots(df, col, color):\n p = figure(plot_width=1000, plot_height=350, x_axis_type=\"datetime\", title=f\"{col} distribution over time\")\n p.line(df['timestamp'], df[col], color=color, alpha=0.5)\n return p\n\ncol_map = {\n \"dew_temperature\": \"Dew Temperature\",\n \"sea_level_pressure\": \"Sea Level Pressure\",\n \"wind_speed\": \"Wind Speed\",\n \"cloud_coverage\": \"Cloud Coverage\",\n}\n\ncolor_map = {\n \"dew_temperature\": \"brown\",\n \"sea_level_pressure\": \"green\",\n \"wind_speed\": \"red\",\n \"cloud_coverage\": \"blue\",\n}\n\nmain_tabs_list = []\ncols = [\"dew_temperature\", \"sea_level_pressure\", \"wind_speed\", \"cloud_coverage\"]\nfor col in cols:\n tab_list = []\n for site in range(16):\n temp_df = weather_df[weather_df[\"site_id\"]==site]\n p = get_plots(temp_df, col, color_map[col])\n tab = Panel(child=p, title=f\"Site:{site}\")\n tab_list.append(tab)\n tabs = Tabs(tabs=tab_list)\n panel = Panel(child=tabs, title=col_map[col])\n main_tabs_list.append(panel)\n\ntabs = Tabs(tabs=main_tabs_list)\nshow(tabs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**More to come. Stay tuned!**"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}