{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ASHRAE - Great Energy Predictor III"},{"metadata":{},"cell_type":"markdown","source":"In this competition our task is to estimate the energy consumption of buildings over time. Significant investments are being made to improve building efficiencies to reduce costs and emissions but current methods of estimation are fragmented and do not scale well. In this competition, we have to develop accurate predictions of metered building energy usage in the following areas: chilled water, electric, natural gas, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!ls -lh ../input/ashrae-energy-prediction/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm import tqdm_notebook\n\nimport altair as alt\nfrom altair.vega import v5\nfrom IPython.display import HTML\n\n%matplotlib inline\nplt.rc('figure', figsize=(15.0, 8.0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nroot = '../input/ashrae-energy-prediction/'\ntrain_df = pd.read_csv(root + 'train.csv')\nweather_train_df = pd.read_csv(root + 'weather_train.csv')\ntest_df = pd.read_csv(root + 'test.csv')\nweather_test_df = pd.read_csv(root + 'weather_test.csv')\nbuilding_meta_df = pd.read_csv(root + 'building_metadata.csv')\nsample_submission = pd.read_csv(root + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's take a brief overview of the train and test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* `building_id` - Foreign key for the building metadata.\n* `meter` - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, hotwater: 3}. Not every building has all meter types.\n* `timestamp` - When the measurement was taken\n* `meter_reading` - **The target variable.** Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'There are {train_df.shape[0]} rows in train data.')\nprint(f\"There are {train_df['meter'].nunique()} distinct meters in train data.\")\nprint(f\"There are {train_df['building_id'].nunique()} distinct buildings in train data, same as the number of rows in building_meta data\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_meta_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`building_meta.csv` contains meta features for each building present in the dataset, the attributes definitions are as follows:\n* `site_id` - Foreign key for the weather files.\n* `building_id` - Foreign key for training.csv\n* `primary_use` - Indicator of the primary category of activities for the building based on EnergyStar property type definitions\n* `square_feet` - Gross floor area of the building\n* `year_built` - Year building was opened\n* `floor_count` - Number of floors of the building\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'There are {building_meta_df.shape[0]} rows in building meta data.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`weather_[train/test].csv` contains weather data from a meteorological station as close as possible to the site.\n\n* `site_id`\n* `air_temperature` - Degrees Celsius\n* `cloud_coverage` - Portion of the sky covered in clouds, in oktas\n* `dew_temperature` - Degrees Celsius\n* `precip_depth_1_hr` - Millimeters\n* `sea_level_pressure` - Millibar/hectopascals\n* `wind_direction` - Compass direction (0-360)\n* `wind_speed` - Meters per second"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'There are {weather_train_df.shape} rows in weather train data.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Let's take a look at the test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`test.csv`\n\nThe submission files use row numbers for ID codes in order to save space on the file uploads. test.csv has no feature data; it exists so you can get your predictions into the correct order.\n\n* `row_id` - Row id for your submission file\n* `building_id` - Building id code\n* `meter` - The meter id code\n* `timestamp` - Timestamps for the test data period"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'There are {test_df.shape[0]} rows in test data.')\nprint(f\"There are {test_df['meter'].nunique()} distinct meters in test data.\")\nprint(f\"There are {test_df['building_id'].nunique()} distinct buildings in test data.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sorted(train_df['building_id'].unique()) == sorted(test_df['building_id'].unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, the we have 1449 `building_id`s in train and test data and both share the same set of ids and meta features as present in `building_meta.csv`"},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`weather_test_df` has same attribute definitions as that of `weather_train_df`"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'There are {weather_test_df.shape} rows in weather test data.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`row_id` is a foreign_key to `test.csv`, we have to predict `meter_reading` for corresponding row in the test.csv"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'There are {sample_submission.shape} rows in sample submission file.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Let's do some deeper analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train_df.merge(building_meta_df, on='building_id', how='left')\ntrain_df = train_df.merge(weather_train_df, on=['site_id', 'timestamp'], how='left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see `meter` counts"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 10))\nplot = sns.countplot(y=\"meter\", data=train_df, palette=['navy', 'darkblue', 'blue', 'dodgerblue']).set_title('Meter count', fontsize=16)\nplt.yticks(fontsize=14)\nplt.xlabel(\"Count\", fontsize=15)\nplt.ylabel(\"Meter number\", fontsize=15)\nplt.show(plot)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that meter frequency is in order of 0 (electricity) > 1 (chilledwater) > 2 (steam) > 3 (hotwater)"},{"metadata":{},"cell_type":"markdown","source":"Let's analyse meter readings with histograms and violin plots:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (18, 6))\nplt.subplot(1, 2, 1);\nplt.hist(train_df['meter_reading']);\nplt.title('Basic meter_reading histogram');\nplt.subplot(1, 2, 2);\nsns.violinplot(x='meter', y='meter_reading', data=train_df);\nplt.title('Violinplot of meter_reading by meter');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Few interesting inferences:\n\n* Almost all of meter_reading values are close to zero\n* meter_readings from meter number 2 is exceptionally higher compared to other meters"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.query('meter_reading==0').shape, train_df.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, about 10% of meter_readings are zero, let's plot `meter_readings` by `meter` distribution plot one by one."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_dist(meter, clip_from):\n    '''Plots distribution of non zero train data vs meter number''' \n    df = train_df.query(f'meter=={meter} and meter_reading!=0')\n    fig, ax = plt.subplots(figsize = (18, 6))\n    plt.subplot(1, 3, 1);\n    plt.hist(df['meter_reading']);\n    plt.title('Basic meter_reading histogram');\n    plt.subplot(1, 3, 2);\n    sns.violinplot(x='meter', y='meter_reading', data=df);\n    plt.title('Violinplot of meter_reading by meter');\n    plt.subplot(1, 3, 3);\n    sns.violinplot(x='meter', y='meter_reading', data=df.query(f'meter_reading < {clip_from}'));\n    plt.title('*Clipped* Violinplot of meter_reading by meter');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist(0, 2000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist(1, 10000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist(2, 10000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist(3, 2000)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at the correlation plot"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Compute the correlation matrix\ncorr = train_df.corr()\n\n# Generate a mask for the upper triangle\nmask = np.zeros_like(corr, dtype=np.bool)\nmask[np.triu_indices_from(mask)] = True\n\n# Set up the matplotlib figure\nf, ax = plt.subplots(figsize=(11, 9))\n\n# Generate a custom diverging colormap\ncmap = sns.diverging_palette(220, 10, as_cmap=True)\n\n# Draw the heatmap with the mask and correct aspect ratio\nsns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,\n            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see meter_reading has low correlation with other attributes, hmm .."},{"metadata":{},"cell_type":"markdown","source":"### Let's try to find out the trend and seasonality in meter_readings"},{"metadata":{"trusted":true},"cell_type":"code","source":"from statsmodels.tsa.seasonal import seasonal_decompose","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_id = 400\nbuilding = train_df.query(f'building_id == {building_id} and meter==0')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"decomposition = seasonal_decompose(building['meter_reading'].values, freq=24)\ndecomposition.plot();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can try out different building ids to see that the seasonality is not that significant in many cases."},{"metadata":{},"cell_type":"markdown","source":"Work in progress.."},{"metadata":{},"cell_type":"markdown","source":"### Let's analyse the train/test weather data"},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train_df.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_kde(column):\n    plot = sns.jointplot(x=train_df[column][:10000], y=train_df['meter_reading'][:10000], kind='kde', color='blueviolet')\n    plot.set_axis_labels('meter', 'meter_reading', fontsize=16)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_dist_col(column):\n    '''plot dist curves for train and test weather data for the given column name'''\n    fig, ax = plt.subplots(figsize=(10, 10))\n    sns.distplot(weather_train_df[column].dropna(), color='darkorange', ax=ax).set_title(column, fontsize=16)\n    sns.distplot(weather_test_df[column].dropna(), color='purple', ax=ax).set_title(column, fontsize=16)\n    plt.xlabel(column, fontsize=15)\n    plt.legend(['train', 'test'])\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist_col('air_temperature')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist_col('cloud_coverage')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist_col('dew_temperature')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist_col('precip_depth_1_hr')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist_col('sea_level_pressure')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist_col('wind_direction')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_dist_col('wind_speed')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Conclusion: the train and test weather distributions are quite similar"},{"metadata":{},"cell_type":"markdown","source":"### Let's analyse the building meta data"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_meta_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 10))\nplot = sns.countplot(y=\"primary_use\", data=building_meta_df, \n                     palette='YlGn').set_title('Primary category of activities for the building', fontsize=16)\nplt.yticks(fontsize=14)\nplt.xlabel(\"Count\", fontsize=15)\nplt.ylabel(\"Building use type\", fontsize=15)\nplt.show(plot)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 10))\nsns.distplot(building_meta_df['square_feet'], color='indigo', \n             ax=ax).set_title('Gross floor area of the building', fontsize=16)\nplt.xlabel('square_feet', fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 10))\nsns.distplot(building_meta_df['year_built'].dropna(), color='crimson', \n             ax=ax).set_title('Year building was opened', fontsize=16)\nplt.xlabel('year_built', fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All buildings were built between 1990 and 2017"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 10))\nsns.distplot(building_meta_df['floor_count'].dropna(), color='crimson', \n             ax=ax).set_title('Number of floors of the building', fontsize=16)\nplt.xlabel('floor_count', fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Work in Progress .."},{"metadata":{},"cell_type":"markdown","source":"<font size=4 color='red'>Do upvote if you liked this kernel :)</font>"},{"metadata":{},"cell_type":"markdown","source":"## References"},{"metadata":{},"cell_type":"markdown","source":"* https://www.kaggle.com/tarunpaparaju/lyft-competition-understanding-the-data\n* https://www.kaggle.com/artgor/molecular-properties-eda-and-models"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}