{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ASHRAE - The first step: EDA with Python\n\nIn this competition we will develop models to predict the energy usage in each building. The dataset contains 1450+ buildings information. Different buildings may have different types of meters.\n\nBefore we start to build a model, we should have some intuition about the data first. So using visualization tools such as Seaborn we can be more familiar with the data we can get.\n\nIf you find this notebook interesting or useful, please do not hesitate to vote for me!\n\n## 1 Start from Here\n\n### 1.1 Loading Data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn import *\nimport gc\nsns.set()\n\nPATH='/kaggle/input/ashrae-energy-prediction/'\ntrain=pd.read_csv(PATH+'train.csv')\ntest=pd.read_csv(PATH+'test.csv')\nweather_train=pd.read_csv(PATH+'weather_train.csv')\nweather_test=pd.read_csv(PATH+'weather_test.csv')\nbuilding=pd.read_csv(PATH+'building_metadata.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# merge the data\ntrain = train.merge(building, on='building_id', how='left')\ntest = test.merge(building, on='building_id', how='left')\ntrain = train.merge(weather_train, on=['site_id', 'timestamp'], how='left')\ntest = test.merge(weather_test, on=['site_id', 'timestamp'], how='left')\ndel weather_train, weather_test,building\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.2 Check some Information about the Data\n\nBefore we start, let's check the missing ratio of each column:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# check whether there are missing values\ndata_na=(train.isnull().sum()/len(train))*100\ndata_na=data_na.drop(data_na[data_na==0].index).sort_values(ascending=False)\nmissing_data=pd.DataFrame({'MissingRatio':data_na})\nprint(missing_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that after merging the dataset some of the columns contain a large percentage of missing values, such as 'floor_count' and 'year_bulit'. We can further check the information of the dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can save the store memory by transforming some of the data types."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Saving the memory space\ndata_types = {'building_id': np.int16,\n          'meter': np.int8,\n          'site_id': np.int8,\n          'square_feet': np.int32,\n          'year_built': np.float16,\n          'floor_count': np.float16,    \n          'cloud_coverage': np.float16,\n          'precip_depth_1_hr': np.float16,\n           'wind_direction': np.float16,     \n          'dew_temperature': np.float32,\n          'air_temperature': np.float32,\n          'sea_level_pressure': np.float32,\n          'wind_speed': np.float32,\n          'primary_use': 'category',}\n\nfor feature in data_types:\n    train[feature] = train[feature].astype(data_types[feature])\n    test[feature] = test[feature].astype(data_types[feature])\n    \ntrain[\"timestamp\"] = pd.to_datetime(train[\"timestamp\"])\ntest[\"timestamp\"] = pd.to_datetime(test[\"timestamp\"])\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's see the data sample."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2 Data Visualization\n\n### 2.1 Boxplot\n\nFirst let's see the relationship between the usage of the building and the age of the building:"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15,10))\nsns.boxplot(x='primary_use', y='year_built', data=train)\nplt.xticks(rotation=90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems that some kinds of buildings were built eariler, such as those used as technology building. What about the relationship between the building area and building types?"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15,10))\nsns.boxplot(x='primary_use', y='square_feet', data=train)\nplt.xticks(rotation=90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above is an intuitive connection, as different types of buildings may install different types of meters, we will the this factor into consideration."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15,10))\nsns.boxplot(x='primary_use', y='square_feet', hue='meter',data=train)\nplt.xticks(rotation=90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The result are clear. Some of the buildings only install two or three types of meters, and the corresponding building area may also have significant differences."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample=pd.DataFrame(train,columns=['site_id','primary_use'])\nsample.drop_duplicates(keep='first')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the buildings with a same 'site_id' may have different primary usage.\n\n### 2.2 View the trends of the time series data\n\nIn this competition the data is shown in time series form, each row has a corresponding Time Label. So it is important for us to visualize the trends of the data.\n\nFirst of all, we will observe the change of data from a monthly perspective. Specifically, the data will be separated by 'site_id' and the meter types."},{"metadata":{"trusted":true},"cell_type":"code","source":"# We can see it by month\nfig, axes = plt.subplots(8,2,figsize=(15, 30))\ncolor_dic={'red':0,'blue':1,'orange':2,'purple':3}\nfor i in range(0,15):    \n    for color,meter in color_dic.items():\n        if(len(train[(train['site_id']==i)&(train['meter']==meter)])!=0):\n            train[(train['site_id']==i)&(train['meter']==meter)][['timestamp', 'meter_reading']].set_index('timestamp').resample('M').mean()['meter_reading'].plot(ax=axes[i%8][i//8], alpha=0.9, label=str(meter), color=color).set_ylabel('Mean meter reading by month', fontsize=13)\n        axes[i%8][i//8].legend();\n        axes[i%8][i//8].set_title('site_id {}'.format(i), fontsize=13);\n        plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the visualization result, we can observe some rules of the data changing. For example, in most buildings the values of meter type 2 and 3 have a significant decrease during the summer, because type 2 provides steam and type 3 provides hot water. But the condition of site_id 13 have a little different.\n\nWe can also observe the data from a weekly perspective:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# We can also see it by week\nfig, axes = plt.subplots(8,2,figsize=(15, 30))\ncolor_dic={'red':0,'blue':1,'orange':2,'purple':3}\nfor i in range(0,15):    \n    for color,meter in color_dic.items():\n        if(len(train[(train['site_id']==i)&(train['meter']==meter)])!=0):\n            train[(train['site_id']==i)&(train['meter']==meter)][['timestamp', 'meter_reading']].set_index('timestamp').resample('W').mean()['meter_reading'].plot(ax=axes[i%8][i//8], alpha=0.9, label=str(meter), color=color).set_ylabel('Mean meter reading by week', fontsize=13)\n        axes[i%8][i//8].legend();\n        axes[i%8][i//8].set_title('site_id {}'.format(i), fontsize=13);\n        plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the weekly perspective,  some data show a cyclical pattern such as the condition of site_id 12.\n\nAnd what about from a daily perspective?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# By day\nfig, axes = plt.subplots(8,2,figsize=(15, 30))\ncolor_dic={'red':0,'blue':1,'orange':2,'purple':3}\nfor i in range(0,15):    \n    for color,meter in color_dic.items():\n        if(len(train[(train['site_id']==i)&(train['meter']==meter)])!=0):\n            train[(train['site_id']==i)&(train['meter']==meter)][['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes[i%8][i//8], alpha=0.9, label=str(meter), color=color).set_ylabel('Mean meter reading by day', fontsize=13)\n        axes[i%8][i//8].legend();\n        axes[i%8][i//8].set_title('site_id {}'.format(i), fontsize=13);\n        plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also further separate the data by hours:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# By hour\nfig, axes = plt.subplots(8,2,figsize=(15, 30))\ncolor_dic={'red':0,'blue':1,'orange':2,'purple':3}\nfor i in range(0,15):    \n    for color,meter in color_dic.items():\n        if(len(train[(train['site_id']==i)&(train['meter']==meter)])!=0):\n            train[(train['site_id']==i)&(train['meter']==meter)][['timestamp', 'meter_reading']].set_index('timestamp').resample('H').mean()['meter_reading'].plot(ax=axes[i%8][i//8], alpha=0.9, label=str(meter), color=color).set_ylabel('Mean meter reading by hour', fontsize=13)\n        axes[i%8][i//8].legend();\n        axes[i%8][i//8].set_title('site_id {}'.format(i), fontsize=13);\n        plt.subplots_adjust(hspace=0.45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.3 The Weather Condition\n\nThe weather condition may also have connections with the energy consumption, we can visualize it, too."},{"metadata":{"trusted":true},"cell_type":"code","source":"# the weather condition(by day)\nfig, axes = plt.subplots(figsize=(20,8))\naxes1=axes.twinx()\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes1,alpha=0.9, label='meter_reading', color='green')\ntrain[['timestamp', 'air_temperature']].set_index('timestamp').resample('D').mean()['air_temperature'].plot(ax=axes,alpha=0.9, label='air_temperature', color='red')\ntrain[['timestamp', 'dew_temperature']].set_index('timestamp').resample('D').mean()['dew_temperature'].plot(ax=axes,alpha=0.9, label='dew_temperature', color='blue')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The green line represents 'meter_reading', we may not find any correlation between the temperature and the meter_reading, and it is strange that the value come across a sudden drop in Jun.\n\nWe can see the cloud condition."},{"metadata":{"trusted":true},"cell_type":"code","source":"# the cloud condition(by day)\nfig, axes = plt.subplots(figsize=(20,8))\naxes1=axes.twinx()\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes1,alpha=0.9, label='meter_reading', color='green')\ntrain[['timestamp', 'cloud_coverage']].set_index('timestamp').resample('D').mean()['cloud_coverage'].plot(ax=axes,alpha=0.9, label='cloud_coverage', color='cyan')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see the wind condition, too."},{"metadata":{"trusted":true},"cell_type":"code","source":"# the wind condition(by day)\nfig, axes = plt.subplots(figsize=(20,8))\naxes1=axes.twinx()\ntrain[['timestamp', 'meter_reading']].set_index('timestamp').resample('D').mean()['meter_reading'].plot(ax=axes1,alpha=0.9, label='meter_reading', color='green')\ntrain[['timestamp', 'wind_speed']].set_index('timestamp').resample('D').mean()['wind_speed'].plot(ax=axes,alpha=0.9, label='wind_speed', color='purple')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the visualization report above, we can see that the weather conditions are almost random, and there seems no correlation between weather conditions and the meter_reading if we just take such a cursory look.\n\n### 2.4 Check the correlation matrix\n\nUsing the heatmap in seaborn can benefit us a lot, we can observe the corrlation of various features in this map."},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(0,4):\n    corr = train[train.meter == i][['timestamp','meter_reading','square_feet','year_built','floor_count',\n             'air_temperature','cloud_coverage','dew_temperature','sea_level_pressure','wind_direction','wind_speed']].corr()\n    f, ax = plt.subplots(figsize=(18, 6))\n    sns.heatmap(corr,annot=True,cmap='RdGy')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we have a basic understanding towards the data and the next step will be feature engineering based on the characteristics of the data and we will try a few types of models."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}