{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport missingno as msno\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.preprocessing import LabelEncoder \nfrom sklearn.cluster import KMeans\nfrom sklearn.preprocessing import StandardScaler,LabelEncoder\n\nimport gc\nimport psutil\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"building_metadata = pd.read_csv(\"../input/ashrae-energy-prediction/building_metadata.csv\")\nsample_submission = pd.read_csv(\"../input/ashrae-energy-prediction/sample_submission.csv\")\ntest = pd.read_csv(\"../input/ashrae-energy-prediction/test.csv\")\ntrain = pd.read_csv(\"../input/ashrae-energy-prediction/train.csv\")\nweather_test = pd.read_csv(\"../input/ashrae-energy-prediction/weather_test.csv\")\nweather_train = pd.read_csv(\"../input/ashrae-energy-prediction/weather_train.csv\")\n\nsns.set()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Basic information\n\n### summary of results:\nthe csv file, **building meta-data** (**count = 1449 **) contains all information regarding each building's structural info like, it's primary use, square feet, year_built and floor count, as well as identifications such as bulding and site ids.\nIt is worth notting that floor count and year built have many missing values and that site id has several categories with many repetitions. \n\nAlso, there is a high amount of variance in the square_feet features which could be a problem. The most populated categories in primary_use are:\n\nEducation                        549\n\nOffice                           279\n\nEntertainment/public assembly    184\n\nPublic services                  156\n\nLodging/residential              147\n\nthe rest have less than 50 ocurrences.\n\nOn the other hand, **training.csv (count=20216100)** has no missing values. The file provides information on the meter_reading and type of meter used to get it as well as the timestamp corresponding to the time of the reading. Meter reading has a standard deviation of the order of 10^5 but 75% of the samples are below a number close to 2* 10^2 which implies a prescence of outliers. The timestamp features shows readings from all of 2016.\n\nFinally **weather_training.csv (count=139773)** , deals with features related to the weather conditions to the sites of the buildings. Precipitation depth and cloud coverage have a significant prescence of missing values. It also includes a timestamp as to when the readings were made in the year 2016.\n\n"},{"metadata":{},"cell_type":"markdown","source":"* ### building metadata"},{"metadata":{"trusted":true},"cell_type":"code","source":"msno.matrix(building_metadata)\nplt.show()\nbuilding_metadata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"lenght: \", len(building_metadata))\n#for numeric features\nbuilding_metadata[['square_feet','floor_count']].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#categorical features\nprint(building_metadata['primary_use'].value_counts())\nprint(building_metadata['site_id'].value_counts())\n\n#dates\nbuilding_metadata.year_built.dropna().sort_values()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training set"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(train))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"msno.matrix(train)\nplt.show()\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#numeric varialbes\ntrain[['meter_reading']].describe()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#categorial\nprint(train['meter'].value_counts())\n#dates\ntrain.timestamp.drop_duplicates().sort_values()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### weather train"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('length: ', len(weather_train))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"msno.matrix(weather_train)\nplt.show()\nweather_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#numeric varialbes\nweather_train[['air_temperature','cloud_coverage','dew_temperature','precip_depth_1_hr',\n                    'sea_level_pressure','wind_direction','wind_speed']].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#dates\nweather_train.timestamp.sort_values().drop_duplicates()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" # Building the dataset\nThe data set is built by merging the building meta data in such a way that the timesatmp meassurements are preserved\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd_data = train.merge(building_metadata, how='left',on='building_id')\npd_data = pd_data.merge(weather_train, how = 'left', on = ['site_id','timestamp'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(train))\nprint(len(building_metadata))\nprint(len(weather_train))\nprint(len(pd_data))\n\npd_data.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_features = ['floor_count','year_built','square_feet','air_temperature','cloud_coverage',\n               'dew_temperature','precip_depth_1_hr','sea_level_pressure','wind_direction'\n               ,'wind_speed']\ncat_features = ['primary_use','site_id','meter']\ntarget = ['meter_reading']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(pd_data[num_features].dtypes)\nfor col in pd_data.columns:\n    if(pd_data[col].dtype == np.float64):\n        pd_data[col] = pd_data[col].astype(np.float32)\n    if(pd_data[col].dtype==np.int64):\n        pd_data[col] = pd_data[col].astype(np.int32)\n        \nprint(\"available RAM:\", psutil.virtual_memory())\n\ngc.collect()\n\nprint(\"available RAM:\", psutil.virtual_memory())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(pd_data[num_features].dtypes)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Basic Statistical description\n\n**1. meter_reading:** There are a wide arrange of values for meter readings. So of the even up to 10^7.\n\n**2. site_id :** The most populated sites are 3 and 13 but otherwise it is pretty evenly distributed inbetween.\n\n**3. meter:** The meter with the most amount of readings is (by a large margin) is electricity (0). Hotwater (3) seems to be rare. \n\n**4. primary_use.** Education is the most common use. Followed by office, entertainment and public services.\n\n**5. year built:**  There are two main spikes: the  70's and early 2000's\n\n**6. meter reagin :** By taking the mean reading for each site and each time unit, one can see that the behaviour of the reading accross the year is different for each site. Some sites present an increase in readings between june and august, while others are mostly uniform during the year. After repeating the same procedure but sepparating each meter type, it is found that chilled water tends to have the highest reading accross the year, but no defined patter is made obvious for all sites.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(pd_data['meter_reading'].describe())\nplt.figure(figsize=(15,7))\n#with respect to meter\nplt.subplot(1,2,1)\nsns.boxplot(y = pd_data['meter_reading'])\nplt.xlabel('meter_reading')\n\nplt.subplot(1,2,2)\nsns.boxplot(y = pd_data['meter_reading'],showfliers=False)\nplt.xlabel('meter_reading')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# site population\ndd = building_metadata.site_id.value_counts().reset_index().rename(columns={'index':'site','site_id':'count'})\nplt.figure(figsize=(10,6))\nsns.barplot(x = 'site',y='count',data = dd)\nplt.title('population for each site')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# meter population\ndd = pd_data.meter.value_counts().reset_index().rename(columns={'index':'meter_type','meter':'count'})\nplt.figure(figsize=(10,6))\nsns.barplot(x = 'meter_type',y='count',data = dd)\nplt.title('population for each type of meter')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#primary use\ndd = building_metadata.primary_use.value_counts().reset_index().rename(columns={'index':'primary_use','primary_use':'count'})\nplt.figure(figsize=(10,6))\nsns.barplot(x = 'primary_use',y='count',data = dd)\nplt.xticks(rotation=90);\nplt.title('population for use')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('year built mode: ', building_metadata['year_built'].dropna().mode())\n# year built\nplt.figure(figsize=(11,7))\nsns.distplot(building_metadata['year_built'].dropna(),kde=True)\nplt.ylabel('occurences')\nplt.xlabel('year built')\nplt.title('distribution of year built')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Meter reading across the year\n\nFirst, we take the mean meter reading for each site at each day and plot the resulting time series. Then, the procedure is repeated but sepparating each type of meter.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#we can look at how the meter reading changes during the year\npd_stamp = pd_data.copy()\npd_stamp['timestamp'] = pd.to_datetime(pd_stamp.timestamp)\npd_stamp['timestamp'] = pd_stamp.timestamp.dt.date\npd_meter = pd_stamp.fillna(0).groupby(['site_id','timestamp'])['meter_reading'].mean().reset_index()\n#pd_weather.isna().sum()\npd_meter = pd_meter.set_index('timestamp')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#meter reading\nplt.figure(figsize=(12,35))\ni=1\nfor s in pd_meter.site_id.unique():\n    plt.subplot(16,1,i)\n    plt.plot(pd_meter[pd_meter['site_id']==s]['meter_reading'],alpha=0.5, color='navy',label=str(s));\n    i+=1\n    plt.legend() \nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd_meter = pd_stamp.fillna(0).groupby(['site_id','meter','timestamp'])['meter_reading'].mean().reset_index()\npd_weather.isna().sum()\npd_meter = pd_meter.set_index('timestamp')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#meter reading\nplt.figure(figsize=(12,35))\ni=1\nfor s in pd_meter.site_id.unique():\n    plt.subplot(16,1,i)\n    for j in range(0,len(pd_meter.meter.unique())):\n        me = pd_meter.meter.unique()[j] \n        plt.plot(pd_meter[(pd_meter['site_id']==s) & (pd_meter['meter']==me)]['meter_reading'],alpha=0.5,label='site_'+str(s)+'_meter_'+str(me));\n    i+=1\n    plt.legend() \nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Outliers\n\nAs will be shown, the distribution for meter reading has heavy tails. Which could be problematic for the algorithm. It is necesary to correctly identify these.\n\nIt is worth notting that almost all of the outliers have missing values for yearl_built and floor_number. In fact, only 10 buildings have a reported year_built. \n\nOn the other hand, sites 9 and 13 have the highest number of outliers. With respect to meter type,steam (type 2) holds the overwelming majority of outlier values, which could be due to the inefficiency of this energy source. Education, also holds the majority of outliers. \n\nGoing more in depth with sites 9 and 13, they hold meters of types 0, 1 and 2 ( electricity, hot water and steam)The weather variables during the year for sites 9 and 13 are not too far from the norm. The only one worth notting is dew temperature whose min and max values tend to be on the higher end of the spectrum.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(pd_data['meter_reading'].describe())\n\nplt.figure(figsize=(11,7))\nsns.distplot(pd_data['meter_reading'],kde=False)\nplt.ylabel('occurences')\nplt.xlabel('meter reading')\nplt.title('all meter readings')\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As can be seen. There are some outliers with more than 300 in meter reading. Otherwise, the distribution seems to be highest around values close to zero."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nplt.figure(figsize=(11,7))\nsns.distplot(pd_data[pd_data['meter_reading']<=(10**3)]['meter_reading'],kde=False)\nplt.ylabel('occurences')\nplt.xlabel('meter reading')\nplt.title('meters below Q3')\nplt.show()\n\nplt.figure(figsize=(11,7))\nsns.distplot(pd_data[(pd_data['meter_reading']>(10**3)) & (pd_data['meter_reading']<=(10**5))]['meter_reading'],kde=False)\nplt.ylabel('occurences')\nplt.xlabel('meter reading')\nplt.title('Above 10^3 and below in 10^5')\nplt.show()\n\nplt.figure(figsize=(11,7))\nsns.distplot(pd_data[pd_data['meter_reading'] >(10**5)]['meter_reading'],kde=False)\nplt.ylabel('occurences')\nplt.xlabel('meter reading')\nplt.title('Above 10^5')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> As can be seen from the graphs above, the tails of the distribution of meter reading are very heavy. We can try to identify those building with more than 10^5 meter reading"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd_outliers = pd_data[pd_data['meter_reading']>(10**4)]\npd_outlier_buildings = pd_outliers.drop_duplicates(subset='building_id',keep='first')\nprint(\"total number of buildings: \",len(pd_data.building_id.unique()) )\nprint(\"number of buildings with more than 10^4 in meter reading: \",len(pd_outlier_buildings) )\nlen(pd_data.building_id.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmsno.matrix(pd_outliers)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# outlier population by site\ndd = pd_outlier_buildings.site_id.value_counts().reset_index().rename(columns={'index':'site','site_id':'count'})\nplt.figure(figsize=(10,6))\nsns.barplot(x = 'site',y='count',data = dd)\nplt.title('outleir population for each site')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can try and take a look at the weather patterns of site 9 and 13 and try to identify a difference between them and the rest"},{"metadata":{"trusted":true},"cell_type":"code","source":"# meter population\ndd = pd_outliers.meter.value_counts().reset_index().rename(columns={'index':'meter_type','meter':'count'})\nplt.figure(figsize=(10,6))\nsns.barplot(x = 'meter_type',y='count',data = dd)\nplt.title('outlier population for each type of meter')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Looking a the meter types in site 9 and 13"},{"metadata":{"trusted":true},"cell_type":"code","source":"b_plot = pd_data[pd_data.site_id.isin([9,13])]['meter'].value_counts().reset_index().rename(columns={'index':'meter_type',\n                                                                                           'meter':'count'})\nplt.figure(figsize=(10,6))\nsns.barplot(x = 'meter_type',y='count',data = b_plot)\nplt.title('meter types in sites 9 and 13')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Looking at the weather in site 9 and 13\n\nThen at meter type and site at the same time\n\nwe can start by taking the mean, max and min of each weather conndition for site and point in time and comparing the time series of each site"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd_stamp = pd_data.copy()\npd_stamp['timestamp'] = pd.to_datetime(pd_stamp.timestamp)\npd_stamp['timestamp'] = pd_stamp.timestamp.dt.date\npd_weather = pd_stamp.fillna(-1).groupby(['site_id','timestamp']).agg({'air_temperature':['mean', 'max','min'], \n                            'cloud_coverage':['mean', 'max','min'],'dew_temperature':['mean', 'max','min'],\n                            'precip_depth_1_hr':['mean', 'max','min'],'sea_level_pressure':['mean', 'max','min'],\n                            'wind_direction':['mean', 'max','min']}).reset_index()\n#pd_weather.isna().sum()\npd_weather = pd_weather.set_index('timestamp')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd_stamp = pd_data.copy()\npd_stamp['timestamp'] = pd.to_datetime(pd_stamp.timestamp)\npd_stamp['timestamp'] = pd_stamp.timestamp.dt.date","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### air temperature"},{"metadata":{"trusted":true},"cell_type":"code","source":"#for air temperature\nplt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['air_temperature']['mean'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['air_temperature']['mean'],alpha=0.2, color='navy');\n\nplt.title('mean air temperature, outlier sites in red')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['air_temperature']['max'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['air_temperature']['max'],alpha=0.2, color='navy');\n\nplt.title('max air temperature, outlier sites in red')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['air_temperature']['min'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['air_temperature']['min'],alpha=0.2, color='navy');\n\nplt.title('min air temperature, outlier sites in red')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### Cloud coverage"},{"metadata":{"trusted":true},"cell_type":"code","source":"#for cloud cover\nplt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['cloud_coverage']['mean'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['cloud_coverage']['mean'],alpha=0.2, color='navy');\n\nplt.title('mean cloud coverage, outlier sites in red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['cloud_coverage']['max'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['cloud_coverage']['max'],alpha=0.2, color='navy');\n\nplt.title('max cloud coverage, outlier sites in red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['cloud_coverage']['min'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['cloud_coverage']['min'],alpha=0.2, color='navy');\n\nplt.title('min coverage, outlier sites in red')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dew Temperature"},{"metadata":{"trusted":true},"cell_type":"code","source":"#fordew\nplt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['dew_temperature']['mean'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['dew_temperature']['mean'],alpha=0.2, color='navy');\n\nplt.title('mean dew_temperature, outlier sites in red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n#for dew\nplt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['dew_temperature']['max'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['dew_temperature']['max'],alpha=0.2, color='navy');\n\nplt.title('max dew_temperature, outlier sites in red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['dew_temperature']['min'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['dew_temperature']['min'],alpha=0.2, color='navy');\n\nplt.title('min dew_temperature, outlier sites in red')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Precipitation depth 1 hours"},{"metadata":{"trusted":true},"cell_type":"code","source":"#precipitation depth\nplt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['precip_depth_1_hr']['mean'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['precip_depth_1_hr']['mean'],alpha=0.2, color='navy');\n\nplt.title('mean precip_depth_1_hr, outlier sites in red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#precipitation depth\nplt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['precip_depth_1_hr']['min'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['precip_depth_1_hr']['min'],alpha=0.2, color='navy');\n\nplt.title('min precip_depth_1_hr, outlier sites in red')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#precipitation depth\nplt.figure(figsize=(15,7))\nfor s in pd_weather.site_id.unique():\n    if(s==13 or s==9):\n        plt.plot(pd_weather[pd_weather.site_id==s]['precip_depth_1_hr']['max'],color='red');\n    else:\n        plt.plot(pd_weather[pd_weather.site_id==s]['precip_depth_1_hr']['max'],alpha=0.2, color='navy');\n\nplt.title('max precip_depth_1_hr, outlier sites in red')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Year built and outliers"},{"metadata":{"trusted":true},"cell_type":"code","source":"#primary use\ndd = pd_outlier_buildings.primary_use.value_counts().reset_index().rename(columns={'index':'primary_use','primary_use':'count'})\nplt.figure(figsize=(10,6))\nsns.barplot(x = 'primary_use',y='count',data = dd)\nplt.xticks(rotation=90);\nplt.title('population for use')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('year built mode: ', pd_outlier_buildings['year_built'].dropna().mode())\nprint('buildings with the data available ', len(pd_outlier_buildings['year_built'].dropna().unique()))\n\n# year built\nplt.figure(figsize=(11,7))\nsns.distplot(pd_outlier_buildings['year_built'].dropna(),kde=False)\nplt.ylabel('occurences')\nplt.xlabel('year built')\nplt.title('distribution of year built')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correlations to Meter\n**correlations numeric**: Square feet has a significant correlation to floor count, wind direction with wind speed\n    A pair plot did not yield any significant correlations. Dew temperature and air temperature have more than .8 correlations.\n\n**categorical correlations** Education has many outliers with high meter values. The distributions are pretty much the same except for health care and utility who tend to have a a higher meter overall.\n\n**correlation to year built** : The highest readings are between 2006 and 2012, old buildings tend to have very small readings \n\n\n**Meter reading and weather variables** : There are complicated correlations between meter reading and weather variables, with meter reading achiving a peak at a certain value of the weather variable and then descending (simmilar to a poisson-like distribution). Precipitation depth maximizes meter reading with values close to zero and sea level  pressure with very high values. On the other hand, cloud coverage has a reading peak at small values.\n\n**Weather and Weather variables :** Dew temperature and air temperature are stronlgy linearly correlated. Wind speed, cloud coverage and precipitation depth have peaks for a certain air temperature. Finally wind direction and air temperature are completely independent.\n\n"},{"metadata":{},"cell_type":"markdown","source":"## numerical variables and meter reading"},{"metadata":{"trusted":true},"cell_type":"code","source":"#correlation matrix of numerical features\ncorrmat = pd_data[target+num_features].corr()\nf, ax = plt.subplots(figsize=(12, 9))\nsns.heatmap(corrmat, vmax=.8, square=True);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Categorical variables and metter reading"},{"metadata":{"trusted":true},"cell_type":"code","source":"# correlation of categorical variables to meter\nplt.figure(figsize=(15,8))\nax = sns.boxplot(x='primary_use', y='meter_reading', data=pd_data);\nax.set_xticklabels(ax.get_xticklabels(),rotation=90)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,8))\nax = sns.boxplot(x='primary_use', y='meter_reading', data=pd_data,showfliers = False);\nax.set_xticklabels(ax.get_xticklabels(),rotation=90)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd_data_in_built = pd_data.copy()\n#pd_data_in_built['year_built'] = pd.to_datetime(pd_data_in_built['year_built']).dt.year\npd_data_in_built = pd_data_in_built.sort_values(by='year_built')\n\npd_data_in_built.dropna(inplace=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(16, 8))\nfig = sns.boxplot(x='year_built', y=\"meter_reading\", data=pd_data_in_built)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(16, 8))\nfig = sns.boxplot(x='year_built', y=\"meter_reading\", data=pd_data_in_built,showfliers=False)\nplt.xticks(rotation=90);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Weather and meter"},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_cols = ['meter_reading','air_temperature','cloud_coverage','dew_temperature','precip_depth_1_hr',\n               'sea_level_pressure','wind_direction','wind_speed']\n\npd_stamp= pd_data.copy()\npd_stamp['timestamp'] = pd.to_datetime(pd_stamp.timestamp)\npd_stamp['timestamp'] = pd_stamp.timestamp.dt.date\npd_w= pd_stamp.fillna(0).groupby(['building_id','timestamp'])[weather_cols].mean().reset_index()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(30,30))\ng = sns.pairplot(pd_w[weather_cols])\nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"available RAM:\", psutil.virtual_memory())\n\ngc.collect()\n\nprint(\"available RAM:\", psutil.virtual_memory())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correlations to year built\n\nGrouped by building\n\n**yeat built to mean meter reading**:Not a  lot to see here\n\n**year built to square feet**: There is no obvious correlation between square feet and year built\n\n**year built to site**: Sites 8 to 14 have all  values missing for year built\n\n**year built to meter type**: Nothing\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd_building = pd_data.groupby('building_id')['meter_reading'].mean().reset_index()\nm_cols = ['building_id','year_built','floor_count','square_feet','meter','primary_use','site_id']\npd_building = pd_building.merge(pd_data[m_cols], on='building_id',how='left').sort_values(by='year_built')\npd_building.drop_duplicates(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#year built to meter reading\nf, ax = plt.subplots(figsize=(16, 8))\nfig = sns.boxplot(x='year_built', y=\"meter_reading\", data=pd_building,showfliers=True)\nplt.xticks(rotation=90);\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#year built to size\nf, ax = plt.subplots(figsize=(16, 8))\nfig = sns.boxplot(x='year_built', y=\"square_feet\", data=pd_building,showfliers=True)\nplt.xticks(rotation=90);\nplt.show()\n\n#year built to size\nf, ax = plt.subplots(figsize=(16, 8))\nfig = sns.boxplot(x='year_built', y=\"square_feet\", data=pd_building,showfliers=False)\nplt.xticks(rotation=90);\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#year built to size\nf, ax = plt.subplots(figsize=(16, 8))\nfig = sns.scatterplot(x='year_built', y=\"square_feet\", data=pd_building,hue='meter')\nplt.xticks(rotation=90);\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#year built to site\nf, ax = plt.subplots(figsize=(16, 8))\nfig = sns.violinplot(x='site_id', y=\"year_built\", data=pd_building)\nplt.xticks(rotation=90);\nplt.show()\n\n#year built to site and meter\nf, ax = plt.subplots(figsize=(16, 8))\nfig = sns.violinplot(x='site_id', y=\"year_built\", data=pd_building, hue='meter')\nplt.xticks(rotation=90);\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nprint(\"available RAM:\", psutil.virtual_memory())\n\ngc.collect()\n\nprint(\"available RAM:\", psutil.virtual_memory())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusions\n    The results from this notebook show that no single variable will be able to predict the meter reading. \n    It is possible that variables made from interactions between other variables will greatly improve\n    the performance of the model,so a great deal of feature engineering might be helpfull. \n    I expect that these interaction will be important with weather related variables (as seen from the pairplots) \n    and time related variables (as seen from the meter reading time series). \n    \n    \n    Another important consideration is that outliers seem to have a great deal of missing values,\n    so it has to be decided if one drops these values or tries to fill the missing numbers with some method."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}