{"cells":[{"metadata":{"_uuid":"b37db323-d009-4aa8-bf50-bc39f44de3b3","_cell_guid":"c823b9e8-55cf-41b2-8533-00cdaf4c0c81","trusted":true},"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d91b16c2-b520-44b6-ad1e-a2a4c8a1575b","_cell_guid":"1c943b08-1a77-4746-b5f2-06f68b0a675d","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\npd.set_option('display.max_columns', None)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using .pkl files from data minification."},{"metadata":{"_uuid":"21c04d84-be56-4153-aa5e-3d85d5920f10","_cell_guid":"0609bd44-aad5-492b-8df1-1753a20834ce","trusted":true},"cell_type":"code","source":"dftrain = pd.read_pickle('/kaggle/input/ashraepkl/train.pkl')\ndftest = pd.read_pickle('/kaggle/input/ashraepkl/test.pkl')\nprint ('Train: %s , Test %s' % (dftrain.shape, dftest.shape) )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is 16 columns with informations like the moment of meter reading, information about the buildings, and the weather. There is also our target variable, meter reading."},{"metadata":{"_uuid":"da1e4244-5695-4a92-81a2-98a62f187ea1","_cell_guid":"87e97832-173e-4804-872a-5719ced09c7b","trusted":true},"cell_type":"code","source":"dftrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the variables are in numeric format."},{"metadata":{"_uuid":"fef00c5e-7079-4b80-ae04-e9bfc153ab87","_cell_guid":"9b068b7f-e2c4-4e32-a782-738a581c6ef8","trusted":true},"cell_type":"code","source":"dftrain.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets see at first how much data is missing in both datasets."},{"metadata":{"_uuid":"1546c269-9e27-4b8a-82fe-841f06e9693c","_cell_guid":"3515dafb-ade3-41bf-a827-0a6f3754082b","trusted":true},"cell_type":"code","source":"ind = np.arange(len(dftrain.columns))\nwidth = 0.35\n\nfig, axes = plt.subplots(figsize=(14, 6))\nplt.title('Missing values by column', fontsize=14)\n\naxes.bar(dftrain.columns, dftrain.isnull().sum(), width, color='orangered')\naxes.bar(ind+width, dftest.isnull().sum(), width, color='lightseagreen')\n\nplt.ylabel('Missing values')\nplt.xticks(rotation=40)\naxes.set_xticks(ind + width / 2)\nplt.legend(title='Data',loc='upper right', labels=['Train', 'Test'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like both train and test datasets have an equivalent quantity of missing values. The floor count column is the one with most missing values followed by year built and cloud coverage"},{"metadata":{"_uuid":"17c4266d-9e8c-411e-840f-de4b29658e33","_cell_guid":"2edd4c19-6c52-4dbb-a688-98ace7801f04","trusted":true},"cell_type":"code","source":"dftrain.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The meter reading have a big quantity of near 0 values. The type of consumed energy have more values on electricity (0)."},{"metadata":{"_uuid":"82a0dce6-daa5-4429-9e8a-dd2f90b06bdf","_cell_guid":"98ece91b-bb5c-4b38-8e84-b8dc45ff4307","trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(12, 6), ncols=2)\nfig.suptitle('Meter reading and meter type distribution', fontsize=14)\ndftrain.meter_reading.plot.hist(color='darkcyan', bins=50, ax=axes[0],log=True)\ndftrain.meter.plot.hist(color='darkcyan', bins=4, ax=axes[1])\naxes[0].set(xlabel='Meter reading')\naxes[1].set(xlabel='Meter type')\naxes[1].set_xticks(np.arange(0, 4, step=1))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6cff9b72-cde6-4cee-8594-3255d8fc9eaa","_cell_guid":"6900fa3b-4f55-4bc3-bace-0d892259d9d6","trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(15, 10), ncols=3, nrows=3)\nfig.suptitle('Features values distribution', fontsize=16)\n\ndftrain.air_temperature.plot.hist(edgecolor='black',color='c', bins=50, ax=axes[0,0])\naxes[0,0].set(xlabel='Air temperature')\n\ndftrain.dew_temperature.plot.hist(edgecolor='black',color='royalblue', bins=50, ax=axes[0,1])\naxes[0,1].set(xlabel='Dew temperature')\n\ndftrain.wind_speed.plot.hist(edgecolor='black',color='c', bins=50, ax=axes[0,2])\naxes[0,2].set(xlabel='Wind speed')\n\ndftrain.sea_level_pressure.plot.hist(edgecolor='black',color='royalblue', bins=50, ax=axes[1,0])\naxes[1,0].set(xlabel='Sea level pressure')\n\ndftrain.wind_direction.plot.hist(edgecolor='black',color='c', bins=50, ax=axes[1,1])\naxes[1,1].set(xlabel='Wind direction')\n\ndftrain.precip_depth_1_hr.plot.hist(edgecolor='black',color='royalblue', bins=50, ax=axes[1,2],log=True)\naxes[1,2].set(xlabel='Precip depth 1hr')\n\ndftrain.cloud_coverage.plot.hist(edgecolor='black',color='c', bins=50, ax=axes[2,0])\naxes[2,0].set(xlabel='Cloud coverage')\n\ndftrain.year_built.plot.hist(edgecolor='black',color='royalblue', bins=50, ax=axes[2,1])\naxes[2,1].set(xlabel='Year built')\n\ndftrain.floor_count.plot.hist(edgecolor='black',color='c', bins=50, ax=axes[2,2])\naxes[2,2].set(xlabel='Floor count')\n\nplt.subplots_adjust(wspace=0.3,hspace=0.3)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Air temperature, dew temperature and sea level pressure are left-tailed distributions. The amount of precipitation depth values is much higher near 0. There are more analyzed buildings that were built in the 60-70s and 2000s. Most buildings have 1 floor."},{"metadata":{"_uuid":"db4b698b-d969-416d-8b76-43a0c863b54b","_cell_guid":"a8627bc0-e4a0-45fc-af76-54dc84c0eeea","trusted":true},"cell_type":"code","source":"temp = dftrain\ncorrmat = round(temp.corr(method='pearson'),2)\nplt.subplots(figsize=(10, 8))\nsns.heatmap(corrmat, vmax=1.0, vmin=-1.0, square=True, annot=True, cmap='RdYlBu')\nplt.title('Correlation', fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at the correlation matrix, there is not a big problem of multicollinearity. Some columns like air temperature and dew temperature have a correlation that seems large. Also the floor count column and area size seem to have a correlation. Let's investigate this further."},{"metadata":{"_uuid":"01f9077e-4dd7-405b-9db1-cfb83bdb5965","_cell_guid":"7c98ec46-3e3a-48e6-a915-4fc90b5f50e8","trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(15, 6), ncols=2, nrows=1)\n\nsns.scatterplot(x='dew_temperature', y='air_temperature', data=dftrain.sample(10000), alpha=0.4, color=\"purple\", ax=axes[0])\naxes[0].set_title('Air temperature by dew temperature', fontsize=14)\nplt.xlabel('dew_temperature', fontsize=12)\nplt.ylabel('air_temperature', fontsize=12)\nsns.scatterplot(x='floor_count', y='square_feet', data=dftrain.sample(10000), alpha=0.4, color=\"purple\", ax=axes[1])\naxes[1].set_title('Floor count by square feet', fontsize=14)\nplt.xlabel('floor_count', fontsize=12)\nplt.ylabel('square_feet', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The correlation between air temperature and dew point seems quite evident, as both features say much the same thing to the model, so it may be interesting to evaluate the exclusion of either later. Since the floor count per area also has a correlation, but not so strong, they can possibly bring different information to the model, so it is better to leave them as features for the model."},{"metadata":{},"cell_type":"markdown","source":"goupying by site Id."},{"metadata":{"_uuid":"3670755c-d630-401d-a535-3d10a29b5ee3","_cell_guid":"5d842bab-cfe7-49f3-853b-291b8bcc9531","trusted":true},"cell_type":"code","source":"site_id_means = dftrain.groupby(by='site_id').mean().reset_index()\nsite_id_means.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"77082c8e-89ce-4926-a7f3-b804acd069f5","_cell_guid":"70947379-1e41-4476-8567-7c573f476afb","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(6,5))\nplt.title('Air temperature mean in each site_id', fontsize=14)\nsns.barplot(x='site_id', y='air_temperature', \n            data = site_id_means,\n            order = site_id_means.sort_values('air_temperature').site_id,\n            palette = 'YlOrRd')\nplt.xlabel('site_id', fontsize=12)\nplt.ylabel('Air temperature', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at the average temperature, we can see that it is quite different for different locations. Location 11 being the coldest and 2 being the hottest. The temperature difference is considerable, indicating that perhaps the buildings may even be in different countries."},{"metadata":{"_uuid":"15131e7a-1227-4c0e-a066-d79513adafd8","_cell_guid":"f7961c1d-a33f-4634-8374-cc5908df30ee","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(6,5))\nplt.title('Meter reading mean in each site_id', fontsize=14)\nsns.barplot(x='site_id', y='meter_reading', \n            data = site_id_means,\n            order = site_id_means.sort_values('meter_reading').site_id,\n            palette = 'RdYlBu')\nplt.xlabel('site_id', fontsize=12)\nplt.ylabel('meter_reading', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems that site 13 consumes a much larger amount of energy than other sites, let's take a closer look at this. I select only the constructions from site 13 and group by construction id, averaging features."},{"metadata":{"_uuid":"f26d0063-f29f-4cb1-9613-47df4ba29a06","_cell_guid":"698c619e-a71e-459d-8f3b-9c3fea6b4116","trusted":true},"cell_type":"code","source":"dftrain[dftrain.site_id==13].groupby('building_id').mean().sort_values(by='meter_reading',ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The energy consumption of a specific id 1099 building is much higher than all the others. This building probably has some data collection error or is an outlier. Further analysis is required later. For now let's remove this building from our dataframe."},{"metadata":{"_uuid":"bfa55707-8cb5-41d7-9a79-b7f8e46d0d73","_cell_guid":"7dcf33a6-f004-4c82-b689-c3af88636129","trusted":true},"cell_type":"code","source":"dftrain = dftrain[dftrain.building_id!=1099]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e1f09a60-4985-491b-9afb-4fee554b041d","_cell_guid":"0b6614d9-87f1-4b2d-86bc-982ac93433f4","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(6,5))\nplt.title('Meter reading mean in each site_id', fontsize=14)\nsns.barplot(x='site_id', y='meter_reading', \n            data = dftrain.groupby(by='site_id').mean().reset_index(),\n            order = site_id_means.sort_values('meter_reading').site_id,\n            palette = 'RdYlBu')\nplt.xlabel('site_id', fontsize=12)\nplt.ylabel('meter_reading', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now it seems that everything is more natural. Let's now check the consumption by type of building use."},{"metadata":{"_uuid":"19af2100-9e9b-4a6d-a830-7702fd63ca53","_cell_guid":"45dc2e05-4ba4-48b6-8a54-6a318f47f450","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\nsns.boxplot(x='primary_use', y='meter_reading', data=dftrain, showfliers=False)\nplt.xticks(rotation='vertical')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The most energy-consuming buildings are health and utilities buildings. The least consuming are religious. For now we finish the exploratory analysis, I will maintain this kernel updated."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}