{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\nIn this competition, we have to develop accurate models of metered building energy usage in the following areas: chilled water, electric, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe. With better estimates of these energy-saving investments, large scale investors and financial institutions will be more inclined to invest in this area to enable progress in building efficiencies.\n\n**About the Host**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1095143%2Ff9ab8963dea5e7c1716f47310daa96ab%2FASHRAE_Logo_25.jpg?generation=1570808142334850&alt=media)\n\nFounded in 1894, ASHRAE serves to advance the arts and sciences of heating, ventilation, air conditioning refrigeration and their allied fields. ASHRAE members represent building system design and industrial process professionals around the world. With over 54,000 members serving in 132 countries, ASHRAE supports research, standards writing, publishing and continuing education - shaping tomorrow’s built environment today."},{"metadata":{},"cell_type":"markdown","source":"# 1. Importing Packages and Collecting Data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"'''Importing Data Manipulattion Moduls'''\nimport numpy as np\nimport pandas as pd\nfrom scipy import stats\nimport os, gc\n\n'''Seaborn and Matplotlib Visualization'''\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_style(\"whitegrid\")                    \n%matplotlib inline\n\n'''plotly Visualization'''\nimport plotly.offline as py\nfrom plotly.offline import iplot, init_notebook_mode\nimport plotly.graph_objs as go\ninit_notebook_mode(connected = True)\n\n'''Display markdown formatted output like bold, italic bold etc.'''\nfrom IPython.display import Markdown\ndef bold(string):\n    display(Markdown(string))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Read the dataset from csv file'''\nbuilding = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv')\nweather_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv')\nweather_test = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv')\ntrain = pd.read_csv('../input/ashrae-energy-prediction/train.csv') \ntest = pd.read_csv('../input/ashrae-energy-prediction/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Variable Description and Identification"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Train and test data at a glance.'''\nbold('**Preview of building data**')\ndisplay(building.head(3))\nbold('**Preview of Weather Train Data:**')\ndisplay(weather_train.head(3))\nbold('**Preview of Weather Test Data:**')\ndisplay(weather_test.head(3))\nbold('**Preview of Train Data:**')\ndisplay(train.head(3))\nbold('**Preview of Test Data:**')\ndisplay(test.head(3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Dimension of train and test data'''\nbold('**Shape of our train and test data**')\nprint('Dimension of building:', building.shape) \nprint('Dimension of Weather train:',weather_train.shape) \nprint('Dimension of Weather test:', weather_test.shape)\nprint('Dimension of train:',train.shape) \nprint('Dimension of test:',test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Merging datasets'''\ntrain = train.merge(building, on = 'building_id', how = 'left')\ntest = test.merge(building, on = 'building_id', how = 'left')\n\ntrain = train.merge(weather_train, on = ['site_id', 'timestamp'], how = 'left')\ntest = test.merge(weather_test, on = ['site_id', 'timestamp'], how = 'left')\n\ndel weather_train, weather_test,building","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Function to reduce the DF size'''\n# source: https://www.kaggle.com/kernels/scriptcontent/3684066/download\n\ndef reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.        \n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n    \n    for col in df.columns:\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype('category')\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = reduce_mem_usage(train)\ntest = reduce_mem_usage(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Variable Desicription\n**Train**\n* *building_id* - Foreign key for the building metadata.\n* *meter* - The meter id code. Read as {0: electricity, 1: chilledwater, 2: steam, 3: hotwater}. Not every building has all meter types.\n* *timestamp* - When the measurement was taken\n* *meter_reading* - The target variable. Energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.\n\n**building_meta**\n* *site_id* - Foreign key for the weather files.\n* *building_id* - Foreign key for training.csv\n* *primary_use* - Indicator of the primary category of activities for the building based on EnergyStar property type definitions\n* *square_feet* - Gross floor area of the building\n* *year_built* - Year building was opened\n* *floor_count* - Number of floors of the building\n\n**weather_[train/test]**\n* Weather data from a meteorological station as close as possible to the site.\n* *air_temperature* - Degrees Celsius\n* *cloud_coverage* - Portion of the sky covered in clouds, in oktas\n* *dew_temperature* - Degrees Celsius\n* *precip_depth_1_hr* - Millimeters\n* *sea_level_pressure* - Millibar/hectopascals\n* *wind_direction* - Compass direction (0-360)\n* *wind_speed* - Meters per second"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Variable Description'''\ndef description(df):\n    summary = pd.DataFrame(df.dtypes,columns=['dtypes'])\n    summary = summary.reset_index()\n    summary['Name'] = summary['index']\n    summary = summary[['Name','dtypes']]\n    summary['Missing'] = df.isnull().sum().values    \n    summary['Uniques'] = df.nunique().values\n    summary['First Value'] = df.iloc[0].values\n    summary['Second Value'] = df.iloc[1].values\n    summary['Third Value'] = df.iloc[2].values\n    return summary","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**Variable Description of  train Data:**')\ndescription(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**Variable Description of  train Data:**')\ndescription(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Well! we have lot of missing value in the both train and test data.**"},{"metadata":{},"cell_type":"markdown","source":"# Feature Engineering - Part I\nThe code block below will be expanded on over time as I come up with some new features.\n\nEngineered features include:\n\n* Month of the year\n* Day of the week of the timestamp\n* Hour of the day"},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"timestamp\"] = pd.to_datetime(train[\"timestamp\"])\ntrain[\"hour\"] = np.uint8(train[\"timestamp\"].dt.hour)\ntrain[\"day\"] = np.uint8(train[\"timestamp\"].dt.day)\ntrain[\"weekday_name\"] = train[\"timestamp\"].dt.weekday_name \ntrain[\"weekday\"] = np.uint8(train[\"timestamp\"].dt.weekday)\ntrain[\"month\"] = np.uint8(train[\"timestamp\"].dt.month)\n\ntest[\"timestamp\"] = pd.to_datetime(test[\"timestamp\"])\ntest[\"hour\"] = np.uint8(test[\"timestamp\"].dt.hour)\ntest[\"day\"] = np.uint8(test[\"timestamp\"].dt.day)\ntest[\"weekday\"] = np.uint8(test[\"timestamp\"].dt.weekday)\ntest[\"month\"] = np.uint8(test[\"timestamp\"].dt.month)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['meter'].replace({0:\"Electricity\",1:\"ChilledWater\",2:\"Steam\",3:\"HotWater\"},inplace=True)\ntest['meter'].replace({0:\"Electricity\",1:\"ChilledWater\",2:\"Steam\",3:\"HotWater\"},inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Exploratory Data Analysis (EDA)"},{"metadata":{},"cell_type":"markdown","source":"## 3.1 Target Variable Analysis:- meter_reading"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Function to distribution plot'''\ndef distplot(variable, color):\n    global ax\n    font_size = 16\n    title_size = 20\n    plt.rcParams['figure.figsize'] = (18, 10)\n    ax = sns.distplot(variable, color = color)\n    plt.xlabel('%s' %variable.name, fontsize = font_size)\n    plt.ylabel('Count ', fontsize = font_size)\n    plt.xticks(fontsize = font_size)\n    plt.yticks(fontsize = font_size)\n    plt.title(' Distribution of '+'%s' %variable.name, fontsize = title_size)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Distribution of the Meter Reading'''\ndistplot(train['meter_reading'], 'teal')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**It's seems that meter reading variable is heavily positive skewed with outliears.\nLet's fixed that.**\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Summary of meter reading'''\ntrain['meter_reading'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Log tranformation of meter_reading'''\ntrain['meter_reading'] = np.log1p(train['meter_reading'])\n\nbold('**Distribution after log tranformation**')\ndistplot(train['meter_reading'], 'teal')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Meter Reading and Meter Type\nThere are four different meter types and are displayed below:\n\n* 0: electricity\n* 1: chilledwater\n* 2: steam\n* 3: hotwater\n\nWe can see that the steam meter type tends to have higher meter readings, while electricity tends to have the lowest energy readings."},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**ELECTRICITY THE MOST FREQUENT METER TYPE MEASURED**')\nplt.rcParams['figure.figsize'] = (18, 10)\nax = sns.countplot(data = train, x ='meter', palette = 'CMRmap', alpha = 0.5)\nax.set_ylabel('Count', fontsize = 20)\nax.set_xlabel('Meter Type', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**THE STEAM METER TYPE IS THE LEAST EFFICIENT, ELECTRICITY THE MOST EFFICIENT**')\nplt.rcParams['figure.figsize'] = (18, 10)\n\ntemp_df = train[train[\"meter\"]==\"Electricity\"]\nax = sns.kdeplot(temp_df['meter_reading'], shade = True, label=\"electricity\")\ntemp_df = train[train[\"meter\"]==\"ChilledWater\"]\nax = sns.kdeplot(temp_df['meter_reading'], shade = True, label=\"chill water\", color = 'm')\ntemp_df = train[train[\"meter\"]==\"Steam\"]\nax = sns.kdeplot(temp_df['meter_reading'], shade = True, label=\"steam\", color = 'lime')\ntemp_df = train[train[\"meter\"]==\"HotWater\"]\nax = sns.kdeplot(temp_df['meter_reading'], shade = True, label=\"hot water\", color = 'k')\nax.set_xlabel('Log(Meter Reading)', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.3 Weekday and Meter Reading"},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**SUNDAYS HAVE THE LOWEST READINGS**')\nplt.rcParams['figure.figsize'] = (18, 10)\nax = sns.boxplot(data = train, x ='weekday_name', y = 'meter_reading', color = 'teal', boxprops=dict(alpha=.3))\nax.set_ylabel('Log(Meter Reading)', fontsize = 20)\nax.set_xlabel('weekdays', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.4 Time of Day and Meter Reading"},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**READINGS HIGHEST DURING THE MIDDLE OF THE DAY**')\nplt.rcParams['figure.figsize'] = (18,10)\ntemp_df = train.groupby('hour').meter_reading.sum()\ntemp_df.plot(linewidth = 5, color = 'teal')\nplt.xlabel('Reading Hour', fontsize = 15)\nplt.ylabel('Meter Reading')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reading are significantly higher during traditional work hours and this is to be expected. Time of day appears like it will be a significant predictor in any subsequent model for this competition."},{"metadata":{},"cell_type":"markdown","source":"# 3.5 Primary Use and Meter Reading"},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**MONTHLY READINGS ARE HIGHEST CHANGES BASED ON BUILDING TYPE**')\ntemp_df = train.groupby(['month', 'primary_use']).meter_reading.sum().reset_index()\nax = sns.FacetGrid(temp_df, col=\"primary_use\", col_wrap=2, height=4, aspect=2,  sharey=False)\nax.map(plt.plot, 'month', 'meter_reading', color=\"teal\", linewidth = 3)\nplt.subplots_adjust(hspace=0.45)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are considerable differences between building types as to when meter readings are highest. Almost all the building peak in the end of the year due to winter season.\n\nThe trend holds for most of the different building types, with a few notable exceptions; Manufacturing dips during that peak period outlined above, while Services, Technology, Utility and Warehouse remained fairly constant over the year."},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**UTILITIES AND HEALTHCARE HAVE THE HIGHEST READINGS**')\nplt.rcParams['figure.figsize'] = (18, 15)\nax = sns.boxplot(data = train, y ='primary_use', x = 'meter_reading', color = 'teal', boxprops=dict(alpha=.3))\nax.set_xlabel('Log(Meter Reading)', fontsize = 20)\nax.set_ylabel('primary_use', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that Utility and Healthcare places tend to have the highest readings, while Religious Worship places the least - they’re no doubt frequented less often than the higher energy users."},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**PLACES OF INDUSTRY HIGHEST READINGS ON WEEKDAYS**')\nax = sns.FacetGrid(train, col=\"primary_use\", col_wrap=4, height=4, aspect=1,  sharex=False)\nax.map(sns.boxplot, 'meter_reading', 'weekday_name', color=\"teal\",   boxprops=dict(alpha=.3))\nplt.subplots_adjust(hspace=0.45)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Healthcare, Education, Manufacturing/ Industrial, Techonology/science, Utilities building has highest reading on weekdays compares to the others.\n\nReligious worship places have higher readings on weekends.\n\n"},{"metadata":{},"cell_type":"markdown","source":"## 3.6 Meter Readings over time"},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**READINGS REALLY PEAKED FROM MAY TO OCTOBER**')\nplt.rcParams['figure.figsize'] = (18,10)\ntemp_df = train.groupby(['timestamp', 'month']).meter_reading.sum().reset_index()\nax = sns.lineplot(data = temp_df, x = 'timestamp', y = 'meter_reading', color = 'teal')\nplt.xlabel('Timestamp', fontsize = 15)\nplt.ylabel('Meter Reading')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interestingly, the meter reading per reading began rising in May and peaked between July and October 2016."},{"metadata":{},"cell_type":"markdown","source":"### Meter Readings over time And Primary Use"},{"metadata":{"trusted":true},"cell_type":"code","source":"bold('**MANUFACTURING REALLY BUCKED THE GENERAL TREND**')\ntemp_df = train.groupby(['timestamp', \"primary_use\"]).meter_reading.sum().reset_index()\nax = sns.FacetGrid(temp_df, col=\"primary_use\", col_wrap=2, height=4, aspect=2,  sharey=False)\nax.map(sns.lineplot,'timestamp',  'meter_reading', color=\"teal\")\nplt.subplots_adjust(hspace=0.45)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The trend holds for most of the different building types, with a few notable exceptions; Manufacturing dips during that peak period outlined above, while Services, Technology, Utility and Warehouse remained fairly constant over the year."},{"metadata":{},"cell_type":"markdown","source":"## 3.7 Correlation between meter_reading And Numeric Variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (18,10)\nsns.heatmap(train.corr(), vmin=-1, vmax=1, center=0,\n            square=True, cmap = sns.diverging_palette(20, 220, n=200))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Square feet and floor count highly positive correlated ie, the bigger the building, the higher the reading and year building build and meter type moderate positive correlated also.\n\nwind_speed, air_temperature and cloud_coverage may be important in any model that gets built."},{"metadata":{},"cell_type":"markdown","source":"## 3.8 Square Feet"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Distribution of the Meter Reading'''\ndistplot(train['square_feet'], 'darkgreen')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Square feet size is positively Skewed."},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Log tranformation of meter_reading'''\ntrain['square_feet'] = np.log1p(train['square_feet'])\ntest['square_feet'] = np.log1p(test['square_feet'])\n\nbold('**Distribution after log tranformation**')\ndistplot(train['square_feet'], 'darkgreen')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.9 Year Built"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (18,10)\ntemp_df = train.groupby('year_built').building_id.sum().reset_index()\nax = sns.lineplot(data = temp_df, x = 'year_built', y = 'building_id', color = 'black', linewidth = 3.5)\nplt.xlabel('Year Built', fontsize = 15)\nplt.ylabel('Building_ID', fontsize = 15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.10 Floor Count"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Distribution of the Meter Reading'''\ndistplot(train['floor_count'].dropna(), 'darkred')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 16,709,167 missing records in the floor_count variable. Of the 1449 unique building IDs, 1094 don’t have a floor count.\n\nOf those that do, we can see that there aren’t too many buildings with more than 10 floors, while the median number of floors is 3."},{"metadata":{},"cell_type":"markdown","source":"## 3.11 Air Temperature"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (18,10)\nsns.kdeplot(train['air_temperature'].dropna(), shade = True, color = 'gold')\nplt.xlabel('Air Temperature', fontsize = 15)\nplt.ylabel('Density', fontsize = 15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plotting the variable shows the variable to be fairly normally distributed, with the majority of recordings being between ~10 and 25 degrees."},{"metadata":{},"cell_type":"markdown","source":"## 3.12 Dew Temperature"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (18,10)\nsns.kdeplot(train['dew_temperature'].dropna(), shade = True, color = 'indigo')\nplt.xlabel('Dew Temperature', fontsize = 15)\nplt.ylabel('Density', fontsize = 15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plotting the variable shows the variable to be fairly normally distributed, with the majority of recordings being between ~13 and 25 degrees."},{"metadata":{},"cell_type":"markdown","source":"## 3.13 Wind Speed"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (18,10)\nsns.kdeplot(train['wind_speed'].dropna(), shade = True, color = 'peru')\nplt.xlabel('Wind Speed', fontsize = 15)\nplt.ylabel('Density', fontsize = 15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plotting the variable shows the variable to be fairly normally distributed, with the majority of recordings being between 1.5 and 3.5 Meters per second"},{"metadata":{},"cell_type":"markdown","source":"## 3.14 Wind direction & Wind speed\nIn heatmap plot wind direction and wind speed are highly correlated. So, it is good to plot together.\n\nwind_direction - Compass direction (0-360)\n\nwind_speed - Meters per second\n\nOk, plotting this is tricky, so I am going to do it using [this manual](https://gist.github.com/phobson/41b41bdd157a2bcf6e14)\n\nCode source: https://www.kaggle.com/nroman/eda-for-ashrae\n\nFirst of all: direction of 0° and 360° is the same thing. But both are presented in the datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"def speed_labels(bins:list, units:str) -> list:   \n    labels = list()\n    for left, right in zip(bins[:-1], bins[1:]):\n        if left == bins[0]:\n            labels.append('calm'.format(right))\n        elif np.isinf(right):\n            labels.append('>{} {}'.format(left, units))\n        else:\n            labels.append('{} - {} {}'.format(left, right, units))\n    return labels\n\ndef _convert_dir(directions, N=None):\n    if N is None:\n        N = directions.shape[0]\n    barDir = directions * np.pi/180. - np.pi/N\n    barWidth = 2 * np.pi / N\n    return barDir, barWidth\n\nspd_bins = [-1, 0, 5, 10, 15, 20, 25, 30, np.inf]\nspd_labels = speed_labels(spd_bins, units='m/s')\n\ndir_bins = np.arange(-7.5, 370, 15)\ndir_labels = (dir_bins[:-1] + dir_bins[1:]) / 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"calm_count = train[train['wind_speed'] == 0].shape[0]\ntotal_count = len(train)\nrose = (train.assign(WindSpd_bins=lambda df:\n            pd.cut(df['wind_speed'], bins=spd_bins, labels=spd_labels, right=True)).assign(WindDir_bins=lambda df: pd.cut(df['wind_direction'], bins=dir_bins, labels=dir_labels, right=False)).replace({'WindDir_bins': {360: 0}}).groupby(by=['WindSpd_bins', 'WindDir_bins']).size().unstack(level='WindSpd_bins').fillna(0).assign(calm=lambda df: calm_count / df.shape[0]).sort_index(axis=1).applymap(lambda x: x / total_count * 100))\nrose.drop(rose.index[0], inplace=True)\ndirections = np.arange(0, 360, 15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def wind_rose(rosedata, wind_dirs, palette=None):\n    if palette is None:\n        palette = sns.color_palette('inferno', n_colors=rosedata.shape[1])\n\n    bar_dir, bar_width = _convert_dir(wind_dirs)\n\n    fig, ax = plt.subplots(figsize=(10, 10), subplot_kw=dict(polar=True))\n    ax.set_theta_direction('clockwise')\n    ax.set_theta_zero_location('N')\n\n    for n, (c1, c2) in enumerate(zip(rosedata.columns[:-1], rosedata.columns[1:])):\n        if n == 0:\n            # first column only\n            ax.bar(bar_dir, rosedata[c1].values, \n                   width=bar_width,\n                   color=palette[0],\n                   edgecolor='none',\n                   label=c1,\n                   linewidth=0)\n            # all other columns\n        ax.bar(bar_dir, rosedata[c2].values, \n               width=bar_width, \n               bottom=rosedata.cumsum(axis=1)[c1].values,\n               color=palette[n+1],\n               edgecolor='none',\n               label=c2,\n               linewidth=0)\n\n    leg = ax.legend(loc=(0.75, 0.95), ncol=2)\n    xtl = ax.set_xticklabels(['N', 'NE', 'E', 'SE', 'S', 'SW', 'W', 'NW'])\n    \n    return fig\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a wind rose for TRAIN:"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = wind_rose(rose, directions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Feature Engineering - Part II"},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Imputing Missing variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Imputing missing value of year build'''\ntrain['year_built'] = np.uint8(train['year_built']-1900, inplace = True)\ntest['year_built'] = np.uint8(test['year_built']-1900, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''The following variables are either discrete numerical or continuous numerical variables.So the will be imputed by median'''\nto_impute_by_median = train.loc[:, ['floor_count','air_temperature', 'cloud_coverage', 'dew_temperature',\n                      'precip_depth_1_hr', 'sea_level_pressure', 'wind_direction','wind_speed']]\nfor i in to_impute_by_median.columns:\n    train[i].fillna(train[i].median(), inplace = True)\n\nto_impute_by_median = test.loc[:, ['floor_count','air_temperature', 'cloud_coverage', 'dew_temperature',\n                      'precip_depth_1_hr', 'sea_level_pressure', 'wind_direction','wind_speed']]\nfor i in to_impute_by_median.columns:\n    test[i].fillna(test[i].median(), inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Encoding Categorical Variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Using sklearn's label encoder method'''\nfrom sklearn.preprocessing import LabelEncoder\n\nle = LabelEncoder()\ntrain['primary_use'] = le.fit_transform(train['primary_use'])\ntest['primary_use'] = le.fit_transform(test['primary_use'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Now extract the nominal variables for one hot encoding of train and test data.'''\none_hot_train = pd.get_dummies(train['meter'])\n\none_hot_test = pd.get_dummies(test['meter'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Droping variable'''\ntrain.drop(columns=['meter', 'timestamp', 'weekday_name'], axis = 1, inplace = True)\ntest.drop(columns=['meter', 'timestamp'], axis = 1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"Let's concate one hot encoded, other variables together.\"\"\"\ntrain_processed = pd.concat([one_hot_train, train], axis = 1)\ntest_processed = pd.concat([one_hot_test, test], axis = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Baseline"},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"Let's look at our final train and test data for modelling.\"\"\"\nbold('**Updated train data for modelling:**')\ndisplay(train_processed.head(3))\nbold('**Updated test data for modelling:**')\ndisplay(test_processed.head(3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Setting train, test and target for model'''\ntarget = train_processed['meter_reading']\ntrain = train_processed.drop(['meter_reading'], axis = 1)\ntest = test_processed.drop(['row_id'], axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"Let's have a final look at our data\"\"\"\nbold('**Data Dimension for Model Building:**')\nprint('Input matrix dimension:', train.shape)\nprint('Output vector dimension:',target.shape)\nprint('Test data dimension:', test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_feat = ['ChilledWater', 'Electricity', 'HotWater', 'Steam',\"site_id\", \"building_id\", \"primary_use\", \"hour\", \"weekday\", \"wind_direction\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modeling simple LGBM"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\nimport lightgbm as lgb\nfrom sklearn.model_selection import KFold, StratifiedKFold\nfrom tqdm import tqdm\n\nparams = {\n            'boosting_type': 'gbdt',\n            'objective': 'regression',\n            'metric': {'rmse'},\n            'subsample': 0.25,\n            'subsample_freq': 1,\n            'learning_rate': 0.3,\n            'num_leaves': 20,\n            'feature_fraction': 0.9,\n            'lambda_l1': 1,  \n            'lambda_l2': 1\n            }\n\nfolds = 4\nseed = 55\nkf = StratifiedKFold(n_splits=folds, shuffle=True, random_state=seed)\n# oof_pred = np.zeros(train.shape[0])  # out of fold predictions\nmodels = []\n\n## stratify data by building_id\nfor train_index, val_index in tqdm(kf.split(train, train['building_id']), total=folds):\n    train_X = train.iloc[train_index]\n    val_X = train.iloc[val_index]\n    train_y = target.iloc[train_index]\n    val_y = target.iloc[val_index]\n    lgb_train = lgb.Dataset(train_X, train_y, categorical_feature=cat_feat)\n    lgb_eval = lgb.Dataset(val_X, val_y, categorical_feature=cat_feat)\n    gbm = lgb.train(params,\n                lgb_train,\n                num_boost_round=500,\n                valid_sets=(lgb_train, lgb_eval),\n                early_stopping_rounds=100,\n                verbose_eval = 100)\n    models.append(gbm)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (18,10)\nlgb.plot_importance(models[0], importance_type='gain')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction and Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"i=0\nresult=[]\nstep_size = 50000\nfor j in tqdm(range(int(np.ceil(test.shape[0]/50000)))):\n    result.append(np.expm1(sum([model.predict(test.iloc[i:i+step_size]) for model in models])/folds))\n    i+=step_size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Submission'''\nresult = np.concatenate(result)\nsubmission = pd.read_csv(\"../input/ashrae-energy-prediction/sample_submission.csv\")\nsubmission[\"meter_reading\"] = result\nsubmission.to_csv(\"submission.csv\", index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <font color='teal'>Give me your feedback and if you find my kernel helpful please UPVOTE will be appreciated.</font> "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}