{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Machine Learning Pipeline - Exploratory Data Analysis\n\n\n1. **Exploratory Data Analysis**\n2. Feature Engineering\n3. Feature Selection\n4. Model Trainng\n5. Obtaining Predictions / Scoring","metadata":{}},{"cell_type":"markdown","source":"======================================================================================================================\n## Predicting Building Energy Usage\n\nThe aim of the project is to develop an accurate model of metered building energy usage in such areas as chilled water, electric, hot water, and steam meters based on historic usage rates and observed weather over a three-year timeframe.\n\n### Why is this important? \n\nPredicting building energy usage can help to understand if the improvement of building efficiencies brings positive impact. With better estimates of these energy-saving investments, investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies that will help to reduce future energy costs and emissions.\n\n### What is the objective of the machine learning model?\n\nWe aim to minimise the difference between the real energy consumption and the energy usage estimated by our model. Model performance is estimated with the:\n\n1. Root Mean Squared Logarithmic Error (main)\n\n2. Mean Squared error (mse)\n\n3. Root Squared of the mean squared error (rmse)\n\n4. R-squared (r2)\n\n======================================================================================================================\n","metadata":{}},{"cell_type":"markdown","source":"## Data\n\nData is taken from the Kaggle Competition: https://www.kaggle.com/competitions/ashrae-energy-prediction/.\n\nIt consists of the following files:\n\n#### train.csv\n\n- **building_id** - Foreign key for the building metadata.\n- **meter** - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, 3: hotwater}. Not every building has all meter types.\n- **timestamp** - When the measurement was taken\n- **meter_reading** - The target variable. Energy consumption in kWh (or equivalent).\n\n#### building_meta.csv\n\n- **site_id** - Foreign key for the weather files.\n- **building_id** - Foreign key for train.csv\n- **primary_use** - Indicator of the primary category of activities for the building based on EnergyStar property type definitions\n- **square_feet** - Gross floor area of the building\n- **year_built** - Year building was opened\n- **floor_count** - Number of floors of the building\n\n#### weather_[train/test].csv\n\nWeather data from a meteorological station as close as possible to the site.\n\n- **site_id**\n- **air_temperature** - Degrees Celsius\n- **cloud_coverage** - Portion of the sky covered in clouds, in oktas\n- **dew_temperature** - Degrees Celsius\n- **precip_depth_1_hr** - Millimeters\n- **sea_level_pressure** - Millibar/hectopascals\n- **wind_direction** - Compass direction (0-360)\n- **wind_speed** - Meters per second\n\n#### test.csv \n\n- **row_id** - Row id for the submission file\n- **building_id** - Building id code\n- **meter** - The meter id code\n- **timestamp** - Timestamps for the test data period\n\n======================================================================================================================\n\nTarget variable is **meter_reading** from train.csv.","metadata":{}},{"cell_type":"markdown","source":"## 1. Loading the data","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('data/train.csv')\nbuilding = pd.read_csv('data/building_metadata.csv')\nweather_train = pd.read_csv('data/weather_train.csv')\n\nprint('train:', train.shape)\nprint('building_metadata:', building.shape)\nprint('weather_train:', weather_train.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"building.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"weather_train.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1.1 Merging the data","metadata":{}},{"cell_type":"code","source":"print('Number of unique buildings in the dataset:', len(train['building_id'].unique()))\nprint('Number of unique sites:', len(building['site_id'].unique()))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assert sorted(list(train['building_id'].unique())) == list(building['building_id'].unique())\nassert sorted(list(building['site_id'].unique())) == list(weather_train['site_id'].unique())\nassert sorted(list(train['timestamp'].unique())) == list(weather_train['timestamp'].unique())\n\nprint('All unique primary keys in datasets fit to each other. So there is no problem to merge data sources.')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = train.merge(building, on = 'building_id', how = 'left')\ndata = data.merge(weather_train, on = ['site_id', 'timestamp'], how = 'left')\n\nprint(data.shape)\n\ndata.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1.2 Converting timestamp into year, month, day, time","metadata":{}},{"cell_type":"code","source":"data['timestamp'] = pd.to_datetime(data['timestamp'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assert data['timestamp'].dt.minute.unique() == [0]\nassert data['timestamp'].dt.second.unique() == [0]\nassert data['timestamp'].dt.year.unique() == [2016]\n\nprint('Time records have only hours as unique values. Year is 2016 for all the records. So we can leave month, day and hour as features.')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data['month'] = data['timestamp'].dt.month\ndata['day'] = data['timestamp'].dt.day\ndata['hour'] = data['timestamp'].dt.hour","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(data.shape)\ndata.head(2)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\n# train and test sets\nX, Xtest, y, ytest = train_test_split(data.drop('meter_reading', axis = 1), data['meter_reading'], \\\n                                      test_size=0.15, random_state=0, shuffle = True)\n\n# train and validation sets\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.15, random_state=0, shuffle = False)\n\nX_train['meter_reading'] = y_train\ndata = X_train.copy()\ndata = data.sort_values(by = 'timestamp', ascending = True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Analysis \n\n**We will conduct following analysis**:\n\n1. Target variable\n2. Missing data\n3. Variable types (categorical and numerical)\n4. Categorical Variables\n    - Cardinality\n    - Rare Labels\n    - Distribution\n\n5. Discrete variables\n\n6. Continious variables\n\n    \n    \n### 2.1. Target variable","metadata":{}},{"cell_type":"code","source":"# exploring target distribution\n\ndata['meter_reading'].describe()\n\nprint('{}% of the records in data set have zero value.'.format(round(len(data[data['meter_reading'] == 0]) / len(data['meter_reading']) * 100),2))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"75% of the whole target variable lays below 268. And there are some very high values as 2*10^7.","metadata":{}},{"cell_type":"code","source":"random_id = np.random.randint(0, len(data['building_id'].unique()), 3)\n\nprint('Distribution of energy usage for three random buildings')\n\nfig, axs = plt.subplots(nrows=1, ncols=3, figsize=(18,4))\n\nfor i, ax in enumerate(axs.flatten()):\n    ax.hist(data[data['building_id']== random_id[i]]['meter_reading'])\n    ax.set_title('Building {}'.format(random_id[i]))\n    ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Energy usage vs date for three random buildings')\n\nfig, axs = plt.subplots(nrows=1, ncols=3, figsize=(18,4))\n\nfor i, ax in enumerate(axs.flatten()):\n    ax.plot(data[data['building_id']== random_id[i]]['timestamp'],data[data['building_id']== random_id[i]]['meter_reading'] )\n    ax.set_title('Building {}'.format(random_id[i]))\n    ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.groupby('timestamp')['meter_reading'].median().plot(title = 'Mean value of the energy usage during the year')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Target \"meter_reading\" is time-dependent variable with a big variance. \n\nLet's see how target looks without zero value.","metadata":{}},{"cell_type":"code","source":"data['meter_reading'][data['meter_reading']!=0].hist(bins=100, figsize = (7,5))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.log(data['meter_reading'][data['meter_reading']!=0]).hist(bins=100, figsize = (7,5))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we remove zero values of target variable (it is 9% of the dataset) and then make a log transformation of it we will get normay distributed variable.","metadata":{}},{"cell_type":"code","source":"# mode of each column\n\ndata[data['meter_reading']==0].mode()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.2. Missing Variables","metadata":{}},{"cell_type":"code","source":"vars_with_na = [\n    var for var in data.columns\n    if data[var].isnull().mean() > 0 \n]\n\ndata[vars_with_na].isnull().mean().sort_values(ascending = False)","metadata":{"scrolled":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First four variables contain big amount of missing values.","metadata":{}},{"cell_type":"markdown","source":"### 2.3. Variable Types","metadata":{}},{"cell_type":"code","source":"data.dtypes","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# categorical variable\n\ndata['primary_use'].unique()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have the only categorical variable \"primary_use\".","metadata":{}},{"cell_type":"code","source":"# discrete variables (<= 31 unique values (31 - max number of days in month))\n\ndiscrete_vars = [var for var in data.columns if len(data[var].unique())<32 and var != 'primary_use']\n\nprint('Number of discrete variables: ', len(discrete_vars))\nprint(discrete_vars)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# continious variables\n\ncontinious_vars = [var for var in data.columns if var not in discrete_vars and var not in ['primary_use','timestamp', 'building_id', 'year_built']]\n\nprint('Number of continious variables: ', len(continious_vars))\nprint(continious_vars)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[continious_vars].head(2)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.4. Categorical Variable\n\nWe have the only categorical variable \"primary_use\".","metadata":{}},{"cell_type":"code","source":"# cardinality\nprint('Number of unique labels is {}.'.format(len(data['primary_use'].unique())))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# distribution of labels\n\ntmp = data.groupby('primary_use')['meter_reading'].count().sort_values(ascending=False)\n\ntmp.plot.bar(figsize=(12,5))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# target based on 'primary_use'\n\norder = list(tmp.index)\n\nsns.catplot(x='primary_use', y = 'meter_reading', data=data[['primary_use','meter_reading']] , order = order, kind = 'box', height = 6, aspect = 3)\nplt.tight_layout()\nplt.xticks(rotation=45)\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# remove outliers from target variable\n\nQ1 = data['meter_reading'].quantile(0.25)\nQ3 = data['meter_reading'].quantile(0.75)\nIQR = Q3-Q1\n\ndata_noout = data[~((data['meter_reading'] < (Q1 - 1.5 * IQR)) | (data['meter_reading'] > (Q3 + 1.5 * IQR)))]\n\nprint('IQR = {} \\n\\nDescription = \\n{}'.format(IQR, data['meter_reading'].describe()))\nprint('\\n\\n\\nWe removed as outliers {}% of training set.'.format(round((1-data_noout.shape[0]/data.shape[0]) * 100, 2)))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# target without outliers based on 'primary_use'\n\nsns.catplot(x='primary_use', y = 'meter_reading', data=data_noout[['primary_use','meter_reading']] , order=order, kind = 'box', height = 6, aspect = 3)\nplt.tight_layout()\nplt.xticks(rotation=45)\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n- We have the only categorical variable \"primary_use\" which has a low cardinality (only 16 unique values).\n- There are the most records corresponding to Education and Office buildings. \n- Some Education buildigns use very high amount of energy (all outliers are in this category).\n- If we remove outliers the Healthcare and Manufacturing buildings use more energy than Education in general.","metadata":{}},{"cell_type":"markdown","source":"### 2.5. Discrete Variables","metadata":{}},{"cell_type":"code","source":"# target with and without outliers  based on discrete variables\nfor var in discrete_vars:\n    \n    order = list(data[var].unique()).sort()\n    \n    sns.catplot(x=var, y = 'meter_reading', data=data[[var,'meter_reading']], order = order, kind = 'box', height = 6, aspect = 3)\n    sns.catplot(x=var, y = 'meter_reading', data=data_noout[[var,'meter_reading']], order = order, kind = 'box', height = 6, aspect = 3)\n    plt.show()\n        \n    print('=====================================================================================================================')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for var in discrete_vars:\n\n    data.groupby(var)['meter_reading'].count().plot.bar(figsize=(16,5))\n    plt.show()\n    data_noout.groupby(var)['meter_reading'].count().plot.bar(figsize=(16,5))\n    plt.show()\n    \n    print('=====================================================================================================================')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From investigating how discrete variables have influence on the target we can make following conclusions. \n\n**Without outliers**:\n\n- Hotwater (2) has the highst values of energy usage. \n- Target is highly depended on site_id, location of building.\n- Energy usage lineary increasing with the number of floors (as expected).\n- It doesn't really depend on cloud_couverage.\n- Energy usage is increasing from 6 to 9 month. Probably on cooling the buildings.\n- It doesn't really depend on day of the month.\n- However, during the day hours energy usage is increasing \n\n**Outliers**:\n\n- Hotwater (2) has the highst values of energy usage. \n- As well as buildings which are located in site_id = 13.\n- Buildings with the number of floors = 2 has the most outliers.\n- Outliers have mostly even value for the cloud_coverage variable.\n- Months from 7 to 9 almost don't contain outliers.\n- Number of floors which is higher than 7 use much more energy\n- June - September are the mounth which require the most energy usage\n- It doesn't really depend on day of month\n- But around 10am till 5pm during the day mean energy usage is increasing","metadata":{}},{"cell_type":"markdown","source":"### 2.6. Continious Variables","metadata":{}},{"cell_type":"code","source":"print(continious_vars)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[continious_vars].head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[continious_vars].hist(bins = 100, figsize = (20,15))\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# logarithmic transformation of \"square_feet\" disitribution\nnp.log(data['square_feet']).hist(bins = 100, figsize = (7,5))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# distributioin of target variable without outliers\n\ndata_noout['meter_reading'].hist(bins=100, figsize = (7,5))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Sea_level_pressure, air_temperature, dew_temperature can be described by normal distribution\n- square_feet after logarithmic tranformation also can be described as normal distribution \n","metadata":{}},{"cell_type":"markdown","source":"## 3. Other Data Dependencies\n\nWe can suppose that floor_count and average year_built highly depends on primary_use. As well as weather variables depend on month. \n\nIf we are right we can use this information for impacting missing values in corresponded categories.","metadata":{}},{"cell_type":"code","source":"# floor_count vs primary_use\n\nsns.catplot(x='floor_count', y = 'primary_use', data=data[['primary_use','floor_count']],  kind = 'box', height = 6, aspect = 3)\nplt.tight_layout()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# floor_count vs primary_use\n\nsns.catplot(x='year_built', y = 'primary_use', data=X_train[['primary_use','year_built']],  kind = 'box', height = 6, aspect = 3)\nplt.tight_layout()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From or analysis we can observe logical results:\n\n   - Retail, Religious worships, Entertainment, Healthcare buildings has usually less then 3 floors.\n   - Residences, Education and Office buildings has the highest number of floors, sometime with very high values.\n    \n\n   - Science buildings were build in first half of last centure. Manufactures  - till the 1980.\n   - Parking places mostly in 21 centure.\n   - All storage buuldings were build around 1980 year.\n   - After this Reatil buildings started to be built. ","metadata":{}},{"cell_type":"code","source":"# dependencies of weather variables from month\nweather_vars = ['air_temperature', 'cloud_coverage', 'dew_temperature', 'precip_depth_1_hr',\n                           'sea_level_pressure', 'wind_direction', 'wind_speed']\n\nfor var in weather_vars:\n    sns.catplot(x='month', y = var, data=X_train[['month',var]],  kind = 'box', height = 6, aspect = 3)\n    plt.tight_layout()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Air-temperature, dew-temperature highly depend on month (increasing from 5 to 8).\n- Highest values of \"wind_speed\" as weel have an opposite dependency.\n- Wind directions slighly changes from 7 to 10 months.\n- 'sea_level_pressure' slightly goes downduring 5 till 9 months.\n- We don't see as well some not logical otliers among these variables.","metadata":{}}]}