{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\n<img src=\"https://upload.wikimedia.org/wikipedia/en/thumb/f/fa/American_Society_of_Heating%2C_Refrigerating_and_Air-Conditioning_Engineers_Logo.svg/1200px-American_Society_of_Heating%2C_Refrigerating_and_Air-Conditioning_Engineers_Logo.svg.png\" width=200 height=200 />\n\nIn this competition we are to create a model to predict an energy usage per building **and** per meter. \n\nWe are given a dataset of around 1450 buildings containing such information as building id, which is a unique identifier of a building across both train and test dataset. Meter - is the type of meter that measures the data - energy consumption in kWh. Different buildings might have different meters installed. Some of them have only one type of meter, some have all four. \n\nSo we need to make a predictions for each meter of each building.\n\n\n# Content\n[Loading data](#1)\n\n[Amount of data and NaNs](#data_amount)\n\n[Mean meter reading by day](#2)\n\n[Number of observations by day](#3)\n\n### Features:\n* [Meter](#meter)\n* [Site_id](#site_id)\n* [primary_use](#primary_use)\n* [square_feet](#square_feet)\n* [year_built](#year_built)\n* [floor_count](#floor_count)\n* [building_id](#building_id)\n* [air_temperature](#air_temperature)\n* [cloud_coverage](#cloud_coverage)\n* [dew_temperature](#dew_temperature)\n* [precip_depth_1_hr](#precip_depth_1_hr)\n* [wind_direction & wind_speed](#wind_direction)\n\n### Engineered features:\n* [Observation hour](#Observation_hour)\n* [Observation day of month](#observ_day)\n* [Observation day of week](#observ_weekday)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nimport gc\nwarnings.simplefilter('ignore')\nmatplotlib.rcParams['figure.dpi'] = 100\nsns.set()\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a>\n# Loading data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"building = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv')\nweather_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv')\nweather_test = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv')\ntrain = pd.read_csv('../input/ashrae-energy-prediction/train.csv')\ntest = pd.read_csv('../input/ashrae-energy-prediction/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Merging everything into two datasets: train and test"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.merge(building, on='building_id', how='left')\ntest = test.merge(building, on='building_id', how='left')\n\ntrain = train.merge(weather_train, on=['site_id', 'timestamp'], how='left')\ntest = test.merge(weather_test, on=['site_id', 'timestamp'], how='left')\ndel weather_train, weather_test,building\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Saving some memory\nd_types = {'building_id': np.int16,\n          'meter': np.int8,\n          'site_id': np.int8,\n          'primary_use': 'category',\n          'square_feet': np.int32,\n          'year_built': np.float16,\n          'floor_count': np.float16,\n          'air_temperature': np.float32,\n          'cloud_coverage': np.float16,\n          'dew_temperature': np.float32,\n          'precip_depth_1_hr': np.float16,\n          'sea_level_pressure': np.float32,\n          'wind_direction': np.float16,\n          'wind_speed': np.float32}\n\nfor feature in d_types:\n    train[feature] = train[feature].astype(d_types[feature])\n    test[feature] = test[feature].astype(d_types[feature])\n    \ntrain[\"timestamp\"] = pd.to_datetime(train[\"timestamp\"])\ntest[\"timestamp\"] = pd.to_datetime(test[\"timestamp\"])\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"data_amount\"></a>\n# Amount of data and NaNs"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_data = (train.count() / len(train)).drop('meter_reading').sort_values().values\nind = np.arange(len(train_data))\nwidth = 0.35\n\nfig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntr = axes.bar(ind, train_data, width, color='royalblue')\n\ntest_data = (test.count() / len(test)).drop('row_id').sort_values().values\ntt = axes.bar(ind+width, test_data, width, color='seagreen')\n\naxes.set_ylabel('Amount of data available');\naxes.set_xticks(ind + width / 2)\naxes.set_xticklabels((train.count() / len(train)).drop('meter_reading').sort_values().index, rotation=40)\naxes.legend([tr, tt], ['Train', 'Test']);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a>\n# Mean meter reading by day"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(14, 6), dpi=100)\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes, label='By hour', alpha=0.8).set_ylabel('Meter reading', fontsize=14);\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes, label='By day', alpha=1).set_ylabel('Meter reading', fontsize=14);\naxes.set_title('Mean Meter reading by hour and day', fontsize=16);\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Doesn't this look weird? First reading are at the avarege level from January to March, then they explode from March to June and then drop down to almost zero with some peak at November.\n\nLets take a closer look."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(8,2,figsize=(14, 30), dpi=100)\nfor i in range(train['site_id'].nunique()):\n    train[train['site_id'] == i][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes[i%8][i//8], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean meter reading', fontsize=13);\n    train[train['site_id'] == i][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes[i%8][i//8], alpha=1, label='By day', color='tab:orange').set_xlabel('');\n    axes[i%8][i//8].legend();\n    axes[i%8][i//8].set_title('site_id {}'.format(i), fontsize=13);\n    plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So what do we see here?\n\n* First of all, as it was already discussed, site_id 0 starts from March. Until that time all measurements are 0's. Maybe buildings in that area were under construction? As a conclusion - we don't need al the data for this particular site_id before that date.\n\n* site_id 13 looks a lot like the overall mean meter_reading, isn't it? Lets take even a closer look."},{"metadata":{},"cell_type":"markdown","source":"### Mean meter reading by primary_use for site_id==13"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(8,2,figsize=(14, 30), dpi=100)\nfor i, use in enumerate(train['primary_use'].value_counts().index.to_list()):\n    try:\n        train[(train['site_id'] == 13) & (train['primary_use'] == use)][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes[i%8][i//8], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean meter reading', fontsize=13);\n        train[(train['site_id'] == 13) & (train['primary_use'] == use)][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes[i%8][i//8], alpha=1, label='By day', color='tab:orange').set_xlabel('');\n        axes[i%8][i//8].legend();\n    except TypeError:\n        pass\n    axes[i%8][i//8].set_title(use, fontsize=13);\n    plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One more level of conclusions:\n* site_id == 13 and primary_use == Education looks a lot like a general mean for meter reading. So it is really invest a lost into the whole data\n* site_id == 13 and primary_use == Technology/Science also have 0's in meter readings in January. Just like site_id 0, that we found earlier.\n\nLets keep digging and see what meter type is responsible for such weird look of the meter reading for site_id 13 and primary_use Education."},{"metadata":{},"cell_type":"markdown","source":"### Mean meter reading by meter type for primary_use==Education and site_id==13"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(3,1,figsize=(14, 18), dpi=100)\nfor i in train[(train['site_id'] == 13) & (train['primary_use'] == 'Education')]['meter'].value_counts(dropna=False).index.to_list():\n    train[(train['site_id'] == 13) & (train['primary_use'] == 'Education') & (train['meter'] == i)][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes[i], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean meter reading', fontsize=13);\n    train[(train['site_id'] == 13) & (train['primary_use'] == 'Education') & (train['meter'] == i)][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes[i], alpha=1, label='By day', color='tab:orange').set_xlabel('');\n    axes[i].legend();\n    axes[i].set_title('Meter: ' + str(i), fontsize=13);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Fine. What we have right now is that site_id 13 with primary_use Education and meter type 2 is responsible for this mess. \n\nBut we need to go deeper. There are 17 buildings that fall under these criteria. No problem. Another plot."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(9,2,figsize=(14, 36), dpi=100)\nfor i, building in enumerate(train[(train['site_id'] == 13) & (train['primary_use'] == 'Education') & (train['meter'] == 2)]['building_id'].value_counts(dropna=False).index.to_list()):\n    train[(train['site_id'] == 13) & (train['primary_use'] == 'Education') & (train['meter'] == 2) & (train['building_id'] == building)][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes[i%9][i//9], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean meter reading', fontsize=13);\n    train[(train['site_id'] == 13) & (train['primary_use'] == 'Education') & (train['meter'] == 2) & (train['building_id'] == building)][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes[i%9][i//9], alpha=1, label='By day', color='tab:orange').set_xlabel('');\n    axes[i%9][i//9].legend();\n    axes[i%9][i//9].set_title('building_id: ' + str(building), fontsize=13);\n    plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And we have a winner. It is a building_id == 1099 (we are going to see that guy a little later again). Now lets plot it side by side with a mean meter reading for all dataset. And also see what this plot would look like if we exclude building_id 1099 from the data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(3,1,figsize=(14, 20), dpi=100)\n\ntrain[(train['meter'] == 2) & (train['building_id'] == 1099)][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes[0], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean meter reading', fontsize=13);\ntrain[(train['meter'] == 2) & (train['building_id'] == 1099)][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes[0], alpha=1, label='By day', color='tab:orange').set_xlabel('');\n\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes[1], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean meter reading', fontsize=13);\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes[1], alpha=1, label='By day', color='tab:orange').set_xlabel('');\n\ntrain[~((train['meter'] == 2) & (train['building_id'] == 1099))][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes[2], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean meter reading', fontsize=13);\ntrain[~((train['meter'] == 2) & (train['building_id'] == 1099))][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes[2], alpha=1, label='By day', color='tab:orange').set_xlabel('');\n\naxes[0].set_title('building_id==1099 and meter==2', fontsize=13);\naxes[1].set_title('Full dataset', fontsize=13);\naxes[2].set_title('building_id 1099 excluded', fontsize=13);\nplt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a>\n# Number of observations by day\n\nThis is something interesting. We can see that number of observations by day in testing dataset is always the same. But not in the training set."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(14, 6))\ntrain['timestamp'].dt.floor('d').value_counts().sort_index().plot(ax=axes).set_xlabel('Date', fontsize=14);\ntest['timestamp'].dt.floor('d').value_counts().sort_index().plot(ax=axes).set_ylabel('Number of observations', fontsize=14);\naxes.set_title('Number of observations by day', fontsize=16);\naxes.legend(['Train', 'Test']);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"meter\"></a>\n# Meter\n\nmeter - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, hotwater: 3}. Not every building has all meter type\n\nLooks like steam is the most unefficient way of heating things up."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_data = train['meter'].value_counts(dropna=False, normalize=True).sort_index().values\nind = np.arange(len(train_data))\nwidth = 0.35\n\nfig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntr = axes.bar(ind, train_data, width, color='royalblue')\n\ntest_data = test['meter'].value_counts(dropna=False, normalize=True).sort_index().values\ntt = axes.bar(ind+width, test_data, width, color='seagreen')\n\naxes.set_ylabel('Normalized number of observations');\naxes.set_xlabel('meter type');\naxes.set_xticks(ind + width / 2)\naxes.set_xticklabels(train['meter'].value_counts().sort_index().index, rotation=0)\naxes2 = axes.twinx()\nmr = axes2.plot(ind, train[['meter', 'meter_reading']].groupby('meter')['meter_reading'].mean().sort_index().values, 'D-', color='tab:orange', label='Mean meter reading');\naxes2.grid(False);\naxes2.tick_params(axis='y', labelcolor='tab:orange');\naxes2.set_ylabel('Mean meter reading by meter type', color='tab:orange');\naxes.legend([tr, tt], ['Train', 'Test'], facecolor='white');\naxes2.legend(loc=5, facecolor='white');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Box plot of meter reading by meter type with outliers removed"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(14, 6))\nsns.boxplot(x='meter', y='meter_reading', data=train, showfliers=False);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Box plot of air temperature by meter type"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"tmp_df = pd.concat([train[['meter', 'air_temperature']], test[['meter', 'air_temperature']]], ignore_index=True)\ntmp_df['dataset'] = 'Train'\ntmp_df.loc[tmp_df.index > len(train), 'dataset'] = 'Test'\n\nfig, axes = plt.subplots(1, 1, figsize=(14, 6))\nsns.boxplot(x='meter', y='air_temperature', data=tmp_df, hue='dataset', palette=\"Set3\");\n\ndel tmp_df\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"site_id\"></a>\n# site_id\n\nsite_id - Foreign key for the weather files.\n\nIn my understanding this is a location."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_data = train['site_id'].value_counts(dropna=False, normalize=True).sort_index().values\nind = np.arange(len(train_data))\nwidth = 0.35\n\nfig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntr = axes.bar(ind, train_data, width, color='royalblue')\n\ntest_data = test['site_id'].value_counts(dropna=False, normalize=True).sort_index().values\ntt = axes.bar(ind+width, test_data, width, color='seagreen')\n\naxes.set_ylabel('Normalized number of observations');\naxes.set_xlabel('site_id');\naxes.set_xticks(ind + width / 2)\naxes.set_xticklabels(train['site_id'].value_counts().sort_index().index, rotation=0)\naxes2 = axes.twinx()\nmr = axes2.plot(ind, train[['site_id', 'meter_reading']].groupby('site_id')['meter_reading'].mean().sort_index().values, 'D-', color='tab:orange', label='Mean meter reading');\naxes2.grid(False);\naxes2.tick_params(axis='y', labelcolor='tab:orange');\naxes2.set_ylabel('Mean meter reading by site_id', color='tab:orange');\naxes.legend([tr, tt], ['Train', 'Test'], facecolor='white');\naxes2.legend(loc=2, facecolor='white');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Box plot of meter reading by site_id with outliers removed"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 1, figsize=(14, 6))\nsns.boxplot(x='site_id', y='meter_reading', data=train, showfliers=False);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Box plot of air temperature by site_id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"tmp_df = pd.concat([train[['site_id', 'air_temperature']], test[['site_id', 'air_temperature']]], ignore_index=True)\ntmp_df['dataset'] = 'Train'\ntmp_df.loc[tmp_df.index > len(train), 'dataset'] = 'Test'\n\nfig, axes = plt.subplots(1, 1, figsize=(14, 6))\nsns.boxplot(x='site_id', y='air_temperature', data=tmp_df, hue='dataset', palette=\"Set3\");\n\ndel tmp_df\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"primary_use\"></a>\n# primary_use\n\nprimary_use - Indicator of the primary category of activities for the building based on [EnergyStar property type definitions](https://www.energystar.gov/buildings/facility-owners-and-managers/existing-buildings/use-portfolio-manager/identify-your-property-type)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_data = train['primary_use'].value_counts(dropna=False, normalize=True).sort_index().values\nind = np.arange(len(train_data))\nwidth = 0.35\n\nfig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntr = axes.bar(ind, train_data, width, color='royalblue')\n\ntest_data = test['primary_use'].value_counts(dropna=False, normalize=True).sort_index().values\ntt = axes.bar(ind+width, test_data, width, color='seagreen')\n\naxes.set_ylabel('Normalized number of observations');\naxes.set_xlabel('primary_use');\naxes.set_xticks(ind + width / 2)\naxes.set_xticklabels(train['primary_use'].value_counts().sort_index().index, rotation=90)\naxes2 = axes.twinx()\nmr = axes2.plot(ind, train[['primary_use', 'meter_reading']].groupby('primary_use')['meter_reading'].mean().sort_index().values, 'D-', color='tab:orange', label='Mean meter reading');\naxes2.grid(False);\naxes2.tick_params(axis='y', labelcolor='tab:orange');\naxes2.set_ylabel('Mean meter reading by primary_use', color='tab:orange');\naxes.legend([tr, tt], ['Train', 'Test'], facecolor='white');\naxes2.legend(loc=5, facecolor='white');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"square_feet\"></a>\n# square_feet\n\nsquare_feet - Gross floor area of the building"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize=(14, 12))\nsns.kdeplot(train['square_feet'], ax=axes[0][0], label='Train');\nsns.kdeplot(test['square_feet'], ax=axes[0][0], label='Test');\nsns.boxplot(x=train['square_feet'], ax=axes[1][0]);\nsns.boxplot(x=test['square_feet'], ax=axes[1][1]);\npd.DataFrame({'train': [train['square_feet'].isnull().sum()], 'test': [test['square_feet'].isnull().sum()]}).plot(kind='bar', rot=0, ax=axes[0][1]);\naxes[0][0].legend();\naxes[0][0].set_title('Train/Test KDE distribution');\naxes[0][1].set_title('Number of NaNs');\naxes[1][0].set_title('Boxplot for train');\naxes[1][1].set_title('Boxplot for test');\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"square_feet is 100% the same as building_id. I am going to show that. Every building_id has only one unique value.\n\nWhich makes sence - building_id is a unique building identifier and each building can have only one possible value of square_feet. Also year_built is going to be the same per unique building_id. Just keep that in mind."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train[train['building_id'] == 0]['square_feet'].value_counts())\nprint(train[train['building_id'] == 1]['square_feet'].value_counts())\nprint(train[train['building_id'] == 100]['square_feet'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train[['building_id', 'square_feet']].sort_index(by='building_id')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets take a look at the mean meter reading by building_id."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1,figsize=(14, 6))\ntrain.groupby('building_id')['meter_reading'].mean().plot();\naxes.set_title('Mean meter reading by building_id', fontsize=14);\naxes.set_ylabel('Mean meter reading', fontsize=14);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is an obvious outlier in data. Lets see what it is.\n\nWell, it is an educational building with no year_build or floor_count provided, so probably it is an old one.\n\nThey have a steam meter which measures such an outstanding amount of electricity used. \n\nIt seems to me that this is a really really old school with a steam heating. "},{"metadata":{"trusted":true},"cell_type":"code","source":"train[(train['building_id']==1099) & (train['meter_reading'] > 3000000)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also there is one more interesting thing, that I want to check. This building is located at site_id 13. So maybe this is some old area? Or this building is the only one consuming so much in the neighbourhood?\n\nSo I have decided to plot mean energy consumption by site_id WITH and WITHOUT this particular building included. And this is what it looks like.\n\nPay attention - the 'Mean meter reading' scale has changed by a factor of 10. So this is the only building with such an enormoush measurements."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(14, 6))\ntrain['site_id'].value_counts(dropna=False, normalize=True).sort_index().plot(kind='bar', rot=0, ax=axes[0]).set_xlabel('site_id value');\ntrain[train['building_id']!=1099]['site_id'].value_counts(dropna=False, normalize=True).sort_index().plot(kind='bar', rot=0, ax=axes[1]).set_xlabel('site_id value');\nax2 = axes[0].twinx()\nax3 = axes[1].twinx()\ntrain.groupby('site_id')['meter_reading'].mean().sort_index().plot(ax=ax2, style='D-', grid=False, color='tab:orange');\ntrain[train['building_id']!=1099].groupby('site_id')['meter_reading'].mean().sort_index().plot(ax=ax3, style='D-', grid=False, color='tab:orange');\nax2.set_ylabel('Mean meter reading', color='tab:orange', fontsize=14);\nax3.set_ylabel('Mean meter reading', color='tab:orange', fontsize=14);\nax2.tick_params(axis='y', labelcolor='tab:orange');\nax3.tick_params(axis='y', labelcolor='tab:orange');\nplt.subplots_adjust(wspace=0.4)\naxes[0].set_title('WITH building_id 1099');\naxes[1].set_title('WITHOUT building_id 1099');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mean meter reading by building_id WITHOUT building_id 1099"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1,figsize=(14, 6))\ntrain[train['building_id'] != 1099].groupby('building_id')['meter_reading'].mean().plot();\naxes.set_title('Mean meter reading by building_id', fontsize=14);\naxes.set_ylabel('Mean meter reading', fontsize=14);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"year_built\"></a>\n# year_built\n\nyear_built - Year building was opened"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1,figsize=(14, 6))\ntrain['year_built'].value_counts(dropna=False).sort_index().plot(ax=axes).set_xlabel('year_built');\ntest['year_built'].value_counts(dropna=False).sort_index().plot(ax=axes).set_ylabel('Number of examples');\naxes.legend(['Train', 'Test']);\naxes.set_title('Number of examples per year_built', fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1,figsize=(14, 6))\ntrain.groupby('year_built')['meter_reading'].mean().plot().set_ylabel('Mean meter reading');\naxes.set_title('Mean meter reading by year_built of the building', fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize=(14, 12))\nsns.kdeplot(train['year_built'], ax=axes[0][0], label='Train');\nsns.kdeplot(test['year_built'], ax=axes[0][0], label='Test');\nsns.boxplot(x=train['year_built'], ax=axes[1][0]);\nsns.boxplot(x=test['year_built'], ax=axes[1][1]);\npd.DataFrame({'train': [train['year_built'].isnull().sum()], 'test': [test['year_built'].isnull().sum()]}).plot(kind='bar', rot=0, ax=axes[0][1]);\naxes[0][0].legend();\naxes[0][0].set_title('Train/Test KDE distribution');\naxes[0][1].set_title('Number of NaNs');\naxes[1][0].set_title('Boxplot for train');\naxes[1][1].set_title('Boxplot for test');\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"floor_count\"></a>\n# floor_count\n\nfloor_count - Number of floors of the building"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(14, 6))\nsns.kdeplot(train['floor_count'], label='Train', ax=axes[0]);\nsns.kdeplot(test['floor_count'], label='Test', ax=axes[0]);\ntest.index += len(train)\naxes[1].plot(train['floor_count'], '.', label='Train');\naxes[1].plot(test['floor_count'], '.', label='Test');\ntest.index -= len(train)\naxes[0].set_title('Train/Test KDE distribution');\naxes[1].set_title('Index versus value: Train/Test distribution');\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1,figsize=(14, 6))\npd.Series(index=train['floor_count'].value_counts().index, \n          data=train.groupby('floor_count')['meter_reading'].transform('mean').value_counts().index).sort_index().plot(kind='bar', rot=0, ax=axes);\naxes.set_xlabel('Floor count');\naxes.set_ylabel('Mean meter reading');\naxes.set_title('Mean meter reading by floor count');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Number of NaNs in floor_count"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"pd.DataFrame({'train': [train['floor_count'].isnull().sum()], 'test': [test['floor_count'].isnull().sum()]}).plot(kind='bar', rot=0);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"building_id\"></a>\n# building_id\n\nbuilding_id - Building id code"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,2,figsize=(14, 6), sharey=True, dpi=100)\naxes[0].plot(train['building_id'].value_counts().sort_index(), '.');\naxes[1].plot(test['building_id'].value_counts().sort_index(), '.');\naxes[0].set_title('Train', fontsize=14);\naxes[1].set_title('Test', fontsize=14);\naxes[0].set_xlabel('building_id');\naxes[1].set_xlabel('building_id');\naxes[0].set_ylabel('Number of examples');\nfig.suptitle('Number of training examples per building_id', fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(14, 6), dpi=100, sharey=True)\ntrain.groupby('building_id')['meter'].nunique().plot(style='.', ax=axes[0]);\ntest.groupby('building_id')['meter'].nunique().plot(style='.', ax=axes[1]);\naxes[0].set_title('Train', fontsize=14);\naxes[1].set_title('Test', fontsize=14);\naxes[0].set_xlabel('building_id');\naxes[1].set_xlabel('building_id');\naxes[0].set_ylabel('Number of meters installed');\nplt.yticks([1, 2, 3, 4]);\nfig.suptitle('Number of meters installed per building_id', fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"air_temperature\"></a>\n# air_temperature\n\nair_temperature - Degrees Celsius"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntrain[['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes, alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean temperature', fontsize=14);\ntest[['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes, alpha=0.8, color='tab:blue', label='');\ntrain[['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes, alpha=1, label='By day', color='tab:orange');\ntest[['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes, alpha=1, color='tab:orange', label='');\naxes.legend();\naxes.text(train['timestamp'].iloc[9000000], -3, 'Train', fontsize=16);\naxes.text(test['timestamp'].iloc[29400000], 30, 'Test', fontsize=16);\naxes.axvspan(test['timestamp'].min(), test['timestamp'].max(), facecolor='green', alpha=0.2);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Air temperature by site_id. Looks like some sites are more to north and some more to south."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(8,2,figsize=(14, 30), dpi=100)\nfor i in range(train['site_id'].nunique()):\n    train[train['site_id'] == i][['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes[i%8][i//8], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean temperature', fontsize=13);\n    test[test['site_id'] == i][['timestamp', 'air_temperature']].set_index('timestamp').resample('H').mean()['air_temperature'].plot(ax=axes[i%8][i//8], alpha=0.8, color='tab:blue', label='').set_xlabel('')\n    train[train['site_id'] == i][['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes[i%8][i//8], alpha=1, label='By day', color='tab:orange')\n    test[test['site_id'] == i][['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes[i%8][i//8], alpha=1, color='tab:orange', label='').set_xlabel('')\n    axes[i%8][i//8].legend();\n    axes[i%8][i//8].set_title('site_id {}'.format(i), fontsize=13);\n    axes[i%8][i//8].axvspan(test['timestamp'].min(), test['timestamp'].max(), facecolor='green', alpha=0.2);\n    plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"cloud_coverage\"></a>\n# cloud_coverage\n\ncloud_coverage - Portion of the sky covered in clouds, in [oktas](https://en.wikipedia.org/wiki/Okta)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1, 2, figsize=(14, 6), dpi=100)\ntrain['cloud_coverage'].value_counts(dropna=False, normalize=True).sort_index().plot(kind='bar', rot=0, ax=axes[0]).set_xlabel('cloud_coverage value');\ntest['cloud_coverage'].value_counts(dropna=False, normalize=True).sort_index().plot(kind='bar', rot=0, ax=axes[1]).set_xlabel('cloud_coverage values');\naxes[0].set_title('Distribution in train', fontsize=14);\naxes[1].set_title('Distribution in test', fontsize=14);\nax2 = axes[0].twinx()\ntrain[['cloud_coverage', 'meter_reading']].replace(np.nan, 'nan').groupby('cloud_coverage')['meter_reading'].mean().plot(ax=ax2, style='D-', grid=False, color='tab:orange');\nax2.tick_params(axis='y', labelcolor='tab:orange');\nax2.set_ylabel('Mean meter reading', color='tab:orange', fontsize=14);\nax2.set_xticklabels(train['cloud_coverage'].value_counts(dropna=False).sort_index().index)\nplt.subplots_adjust(wspace=0.4);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"dew_temperature\"></a>\n# dew_temperature\n\ndew_temperature - Degrees Celsius"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntrain[['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes, alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean dew temperature', fontsize=14);\ntest[['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes, alpha=0.8, color='tab:blue', label='');\ntrain[['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes, alpha=1, label='By day', color='tab:orange');\ntest[['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes, alpha=1, color='tab:orange', label='');\naxes.legend();\naxes.text(train['timestamp'].iloc[9000000], -5, 'Train', fontsize=16);\naxes.text(test['timestamp'].iloc[29400000], 16, 'Test', fontsize=16);\naxes.axvspan(test['timestamp'].min(), test['timestamp'].max(), facecolor='green', alpha=0.2);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mean dew temperature by site_id"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(8,2,figsize=(14, 30), dpi=100)\nfor i in range(train['site_id'].nunique()):\n    train[train['site_id'] == i][['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes[i%8][i//8], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean dew temperature', fontsize=13);\n    test[test['site_id'] == i][['timestamp', 'dew_temperature']].set_index('timestamp').resample('H').mean()['dew_temperature'].plot(ax=axes[i%8][i//8], alpha=0.8, color='tab:blue', label='').set_xlabel('')\n    train[train['site_id'] == i][['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes[i%8][i//8], alpha=1, label='By day', color='tab:orange')\n    test[test['site_id'] == i][['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes[i%8][i//8], alpha=1, color='tab:orange', label='').set_xlabel('')\n    axes[i%8][i//8].legend();\n    axes[i%8][i//8].set_title('site_id {}'.format(i), fontsize=13);\n    axes[i%8][i//8].axvspan(test['timestamp'].min(), test['timestamp'].max(), facecolor='green', alpha=0.2);\n    plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"precip_depth_1_hr\"></a>\n# precip_depth_1_hr\nprecip_depth_1_hr - Millimeters\n\nThere are plenty of NaNs and you can see on the following plot that there is no information for Decemder 2016 - January 2017."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntrain[['timestamp', 'precip_depth_1_hr']].set_index('timestamp').resample('M').mean()['precip_depth_1_hr'].plot(ax=axes, alpha=0.8, label='By month', color='tab:blue').set_ylabel('Mean precip_depth_1_hr', fontsize=14);\ntest[['timestamp', 'precip_depth_1_hr']].set_index('timestamp').resample('M').mean()['precip_depth_1_hr'].plot(ax=axes, alpha=0.8, color='tab:blue', label='');\naxes.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"sea_level_pressure\"></a>\n\n# sea_level_pressure\n\nsea_level_pressure - Millibar/hectopascals"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntrain[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes, alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean sea_level_pressure', fontsize=14);\ntest[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes, alpha=0.8, color='tab:blue', label='');\ntrain[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes, alpha=1, label='By day', color='tab:orange');\ntest[['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes, alpha=1, color='tab:orange', label='');\naxes.legend();\naxes.text(train['timestamp'].iloc[9000000], 1004, 'Train', fontsize=16);\naxes.text(test['timestamp'].iloc[21000000], 1032, 'Test', fontsize=16);\naxes.axvspan(test['timestamp'].min(), test['timestamp'].max(), facecolor='green', alpha=0.2);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And separate by site_id"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(8,2,figsize=(14, 30), dpi=100)\nfor i in range(train['site_id'].nunique()):\n    train[train['site_id'] == i][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes[i%8][i//8], alpha=0.8, label='By hour', color='tab:blue').set_ylabel('Mean sea_level_pressure', fontsize=13);\n    test[test['site_id'] == i][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('H').mean()['sea_level_pressure'].plot(ax=axes[i%8][i//8], alpha=0.8, color='tab:blue', label='').set_xlabel('')\n    train[train['site_id'] == i][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes[i%8][i//8], alpha=1, label='By day', color='tab:orange')\n    test[test['site_id'] == i][['timestamp', 'sea_level_pressure']].set_index('timestamp').resample('D').mean()['sea_level_pressure'].plot(ax=axes[i%8][i//8], alpha=1, color='tab:orange', label='').set_xlabel('')\n    axes[i%8][i//8].legend();\n    axes[i%8][i//8].set_title('site_id {}'.format(i), fontsize=13);\n    axes[i%8][i//8].axvspan(test['timestamp'].min(), test['timestamp'].max(), facecolor='green', alpha=0.2);\n    plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see all values for site_id 5 are NaNs. Maybe there is no a pressure meter installed in that area?"},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train['site_id'] == 5]['sea_level_pressure'].value_counts(dropna=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"wind_direction\"></a>\n# wind_direction & wind_speed\n\nwind_direction - Compass direction (0-360)\n\nwind_speed - Meters per second\n\n\n\nOk, plotting this is tricky, so I am going to do it using [this manual](https://gist.github.com/phobson/41b41bdd157a2bcf6e14)\n\nFirst of all: direction of 0° and 360° is the same thing. But both are presented in the datasets."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Number of observations with 0 degree direction in TRAIN: {} ({:.2f}%)'.format(train[train['wind_direction'] == 0].shape[0], train[train['wind_direction'] == 0].shape[0] / len(train)))\nprint('Number of observations with 0 degree direction in TEST: {} ({:.2f}%)'.format(test[test['wind_direction'] == 0].shape[0], test[test['wind_direction'] == 0].shape[0] / len(test)))\nprint('Number of observations with 360 degree direction in TRAIN: {} ({:.2f}%)'.format(train[train['wind_direction'] == 360].shape[0], train[train['wind_direction'] == 360].shape[0] / len(train)))\nprint('Number of observations with 360 degree direction in TEST: {} ({:.2f}%)'.format(test[test['wind_direction'] == 360].shape[0], test[test['wind_direction'] == 360].shape[0] / len(test)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now lets see how many observations have calm conditions (wind speed is 0 meters per second)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Number of observations that have calm conditions in TRAIN: {} ({:.2f}%)'.format(train[train['wind_speed'] == 0].shape[0], train[train['wind_speed'] == 0].shape[0] / len(train)))\nprint('Number of observations that have calm conditions in TEST: {} ({:.2f}%)'.format(test[test['wind_speed'] == 0].shape[0], test[test['wind_speed'] == 0].shape[0] / len(test)))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def speed_labels(bins:list, units:str) -> list:   \n    labels = list()\n    for left, right in zip(bins[:-1], bins[1:]):\n        if left == bins[0]:\n            labels.append('calm'.format(right))\n        elif np.isinf(right):\n            labels.append('>{} {}'.format(left, units))\n        else:\n            labels.append('{} - {} {}'.format(left, right, units))\n    return labels\n\ndef _convert_dir(directions, N=None):\n    if N is None:\n        N = directions.shape[0]\n    barDir = directions * np.pi/180. - np.pi/N\n    barWidth = 2 * np.pi / N\n    return barDir, barWidth\n\nspd_bins = [-1, 0, 5, 10, 15, 20, 25, 30, np.inf]\nspd_labels = speed_labels(spd_bins, units='m/s')\n\ndir_bins = np.arange(-7.5, 370, 15)\ndir_labels = (dir_bins[:-1] + dir_bins[1:]) / 2","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"calm_count = train[train['wind_speed'] == 0].shape[0]\ntotal_count = len(train)\nrose = (train.assign(WindSpd_bins=lambda df:\n            pd.cut(df['wind_speed'], bins=spd_bins, labels=spd_labels, right=True)).assign(WindDir_bins=lambda df: pd.cut(df['wind_direction'], bins=dir_bins, labels=dir_labels, right=False)).replace({'WindDir_bins': {360: 0}}).groupby(by=['WindSpd_bins', 'WindDir_bins']).size().unstack(level='WindSpd_bins').fillna(0).assign(calm=lambda df: calm_count / df.shape[0]).sort_index(axis=1).applymap(lambda x: x / total_count * 100))\nrose.drop(rose.index[0], inplace=True)\ndirections = np.arange(0, 360, 15)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def wind_rose(rosedata, wind_dirs, palette=None):\n    if palette is None:\n        palette = sns.color_palette('inferno', n_colors=rosedata.shape[1])\n\n    bar_dir, bar_width = _convert_dir(wind_dirs)\n\n    fig, ax = plt.subplots(figsize=(10, 10), subplot_kw=dict(polar=True))\n    ax.set_theta_direction('clockwise')\n    ax.set_theta_zero_location('N')\n\n    for n, (c1, c2) in enumerate(zip(rosedata.columns[:-1], rosedata.columns[1:])):\n        if n == 0:\n            # first column only\n            ax.bar(bar_dir, rosedata[c1].values, \n                   width=bar_width,\n                   color=palette[0],\n                   edgecolor='none',\n                   label=c1,\n                   linewidth=0)\n\n        # all other columns\n        ax.bar(bar_dir, rosedata[c2].values, \n               width=bar_width, \n               bottom=rosedata.cumsum(axis=1)[c1].values,\n               color=palette[n+1],\n               edgecolor='none',\n               label=c2,\n               linewidth=0)\n\n    leg = ax.legend(loc=(0.75, 0.95), ncol=2)\n    xtl = ax.set_xticklabels(['N', 'NE', 'E', 'SE', 'S', 'SW', 'W', 'NW'])\n    \n    return fig","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a wind rose for TRAIN:"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = wind_rose(rose, directions)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"calm_count = test[test['wind_speed'] == 0].shape[0]\ntotal_count = len(test)\nrose = (test.assign(WindSpd_bins=lambda df:\n            pd.cut(df['wind_speed'], bins=spd_bins, labels=spd_labels, right=True)).assign(WindDir_bins=lambda df: pd.cut(df['wind_direction'], bins=dir_bins, labels=dir_labels, right=False)).replace({'WindDir_bins': {360: 0}}).groupby(by=['WindSpd_bins', 'WindDir_bins']).size().unstack(level='WindSpd_bins').fillna(0).assign(calm=lambda df: calm_count / df.shape[0]).sort_index(axis=1).applymap(lambda x: x / total_count * 100))\nrose.drop(rose.index[0], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And wind rose for TEST"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = wind_rose(rose, directions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Engineered features\n\nNow lets add some new features and see if they might be usefull. First I am going to work with timeseries data.\n\n<a id='Observation_hour'></a>\n# Observation hour\n\nAn hour the meter reading was recorded."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['hour'] = train['timestamp'].dt.hour\ntest['hour'] = test['timestamp'].dt.hour","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One more evidence that train and test datasets are very well balanced. Number of observations per hour is almost the same.\n\nAlso we can see that energy consumption raises up during the day and lowers during the night hours."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_data = train['hour'].value_counts(dropna=False, normalize=True).sort_index().values\nind = np.arange(len(train_data))\nwidth = 0.35\n\nfig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntr = axes.bar(ind, train_data, width, color='royalblue')\n\ntest_data = test['hour'].value_counts(dropna=False, normalize=True).sort_index().values\ntt = axes.bar(ind+width, test_data, width, color='seagreen')\n\naxes.set_ylabel('Normalized number of observations');\naxes.set_xlabel('Hour');\naxes.set_xticks(ind + width / 2)\naxes.set_xticklabels(train['hour'].value_counts().sort_index().index, rotation=0)\naxes2 = axes.twinx()\nmr = axes2.plot(ind, train[['hour', 'meter_reading']].groupby('hour')['meter_reading'].mean().sort_index().values, 'D-', color='tab:orange', label='Mean meter reading');\naxes2.grid(False);\naxes2.tick_params(axis='y', labelcolor='tab:orange');\naxes2.set_ylabel('Mean meter reading by hour', color='tab:orange');\naxes.legend([tr, tt], ['Train', 'Test'], facecolor='white');\naxes2.legend(loc=2, facecolor='white');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='observ_day'></a>\n# Observation day of month\n\nA day of month the meter reading was recorded.\n\nNumber of observations for day 30 and 31 are lower than for other days, which makes sence - not every month have 30 or 31 days in it.\n\nAlso pay attention that number of observations for day 29 in training set is higher than in testing set. This is because training set contains observations of the year 2016, which was a leap year."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['day'] = train['timestamp'].dt.day\ntest['day'] = test['timestamp'].dt.day","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_data = train['day'].value_counts(dropna=False, normalize=True).sort_index().values\nind = np.arange(len(train_data))\nwidth = 0.35\n\nfig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntr = axes.bar(ind, train_data, width, color='royalblue')\n\ntest_data = test['day'].value_counts(dropna=False, normalize=True).sort_index().values\ntt = axes.bar(ind+width, test_data, width, color='seagreen')\n\naxes.set_ylabel('Normalized number of observations');\naxes.set_xlabel('day');\naxes.set_xticks(ind + width / 2)\naxes.set_xticklabels(train['day'].value_counts().sort_index().index, rotation=0)\naxes2 = axes.twinx()\nmr = axes2.plot(ind, train[['day', 'meter_reading']].groupby('day')['meter_reading'].mean().sort_index().values, 'D-', color='tab:orange', label='Mean meter reading');\naxes2.grid(False);\naxes2.tick_params(axis='y', labelcolor='tab:orange');\naxes2.set_ylabel('Mean meter reading by day of month', color='tab:orange');\naxes.legend([tr, tt], ['Train', 'Test'], facecolor='white');\naxes2.legend(loc=2, facecolor='white');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"observ_weekday\"></a>\n# Observation day of week"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['weekday'] = train['timestamp'].dt.weekday\ntest['weekday'] = test['timestamp'].dt.weekday","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_data = train['weekday'].value_counts(dropna=False, normalize=True).sort_index().values\nind = np.arange(len(train_data))\nwidth = 0.35\n\nfig, axes = plt.subplots(1,1,figsize=(14, 6), dpi=100)\ntr = axes.bar(ind, train_data, width, color='royalblue')\n\ntest_data = test['weekday'].value_counts(dropna=False, normalize=True).sort_index().values\ntt = axes.bar(ind+width, test_data, width, color='seagreen')\n\naxes.set_ylabel('Normalized number of observations');\naxes.set_xlabel('weekday');\naxes.set_xticks(ind + width / 2)\naxes2 = axes.twinx()\nmr = axes2.plot(ind, train[['weekday', 'meter_reading']].groupby('weekday')['meter_reading'].mean().sort_index().values, 'D-', color='tab:orange', label='Mean meter reading');\naxes2.grid(False);\naxes2.tick_params(axis='y', labelcolor='tab:orange');\naxes2.set_ylabel('Mean meter reading by day of week', color='tab:orange');\naxes.legend([tr, tt], ['Train', 'Test'], facecolor='white');\naxes2.legend(loc=2, facecolor='white');\n\naxes.set_xticklabels(train['weekday'].value_counts().sort_index().index.map({0: 'Monday', 1: 'Tuesday', 2: 'Wednesday', 3: 'Thursday', 4: 'Friday', 5: 'Saturday', 6: 'Sunday'}), rotation=40);","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}