{"cells":[{"metadata":{},"cell_type":"markdown","source":"**!!! PLEASE UPVOTE THE KERNEL IF YOU LIKE IT !!!**"},{"metadata":{},"cell_type":"markdown","source":"# Introduction:\nThe American Society of Heating, Refrigerating and Air-Conditioning Engineers ([ASHRAE](https://en.wikipedia.org/wiki/ASHRAE)) is an American professional association seeking to advance heating, ventilation, air conditioning and refrigeration systems design and construction. ASHRAE has more than 57,000 members in more than 132 countries worldwide. "},{"metadata":{},"cell_type":"markdown","source":"# Problem Statement : \n\nSignificant investments are being made to improve building efficiencies to reduce costs and emissions. But, are the improvements working? Current methods of estimation are fragmented and do not scale well. Some assume a specific meter type or don’t work with different building types.\n\nIn this competition, you’ll develop accurate predictions of metered building energy usage in the following areas: chilled water, electric, natural gas, hot water, and steam meters. The data comes from over 1,000 buildings over a three-year timeframe."},{"metadata":{},"cell_type":"markdown","source":"# Analysis:\n\nHere, we will start working with the datasets and try to build some understanding about the same first using certain visualizations. Then we will see if we can do some feature engineering and try to build some model.\n\nFirst, we import necessary packages"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we list the input files to see that there are 6 files available."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"print(os.listdir('../input/ashrae-energy-prediction'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training Data"},{"metadata":{},"cell_type":"markdown","source":"# a) train.csv"},{"metadata":{},"cell_type":"markdown","source":"### Step 1) Reading and Interpreting Data"},{"metadata":{},"cell_type":"markdown","source":"Because there are 6 files, reading them individually and then interpreting them one by one will make the code longer. So, I have written a small function to help me for this task."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to read and do the initial data interpretation\ndef read_and_interpret_data(filename):\n    path = \"../input/ashrae-energy-prediction\"\n    df = pd.read_csv('{0}/{1}'.format(path,filename))\n    print(\"~~~~~~Shape of the data~~~~~~ : \",df.shape)\n    print(\"~~~~~~Columns and their datatype~~~~~~ : \")\n    print(df.info())\n    print(\"~~~~~~Quick Look at the data~~~~~~ : \")\n    print(df.head())\n    print(\"~~~~~~Description of the data~~~~~~ : \")\n    print(df.describe())\n    print(\"~~~~~~NAs present in the data~~~~~~ : \")\n    print(df.isna().sum())\n    if 'timestamp' in df.columns: \n        df['timestamp'] = pd.to_datetime(df['timestamp'],format = \"%Y-%m-%d %H:%M:%S\")\n        print(\"~~~~~~Year of the data~~~~~~ :\")\n        print(df.timestamp.dt.year.unique())\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = read_and_interpret_data('train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.groupby('building_id')['meter_reading'].agg(['count','min','max','mean','median','std'])\n# We can see that the values for building number 1099 are exceptionally high. These can be safely considered as outliers and can be dropped.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove outliers\ndf_train = df_train [df_train['building_id'] != 1099 ]\ndf_train = df_train.query('not (building_id <= 104 & meter == 0 & timestamp <= \"2016-05-20\")')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 2) Visualizing Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"start = df_train['timestamp'].min()\nend = df_train['timestamp'].max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(start)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We are interested to see how the target variable varies with time. \n\nFor this, I have implemented a function that lets us look at the data. \n\nThe beauty of this function is that -\n- By default, this function plots the whole data starting from building_id = 0 to `num_buildings` specified\n- If provided the start and end data, it will plot the data between this specified period.\n- If provided with parameter `start_building`, it will plot from building_id = `start_building` onwards"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_for_date_range(num_buildings,start_building = 0,start_date=start, end_date=end):\n    plt.figure(figsize=(18,15), facecolor='white')\n    plot_num = 1\n    for i in range(start_building,start_building+num_buildings):\n        ax = plt.subplot(num_buildings, 1, plot_num)\n        data=df_train[df_train.building_id == i].set_index('timestamp').loc[start_date:end_date]\n        data.plot(y='meter_reading', ax=ax, label=i, legend=False)\n        ax.set_title(f'Building id {i}')\n        plot_num +=1\n    \n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**1) Variation of Target Variable `meter_reading` with `timestamp` - For the whole year 2016**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_for_date_range(num_buildings = 10,start_building = 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above graph, we note the following -\n\nGeneral Trend\n - meter reading is very low from Jan to April in general but for some building it is not true\n - There is a sudden spike in May after which the meter reading goes again to very low.\n - From mid-June onwards, the meter_reading follows a noisy time series"},{"metadata":{},"cell_type":"markdown","source":"** 2) Target variable with time - Specified Range**"},{"metadata":{},"cell_type":"markdown","source":"Although, the above graph lets us visualize how `meter_reading` varies over the year 2016 as a whole, we are unable to visualize the seasonal changes in the graph.\n\nFor this, we can use the same function by passing the start and end dates between which we are interested to see the graph"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_for_date_range(7,0,'2016-07-01', '2016-08-01')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Specific Notes\n - For Buildings 2 and 5, we can clearly see that there is a daily seasonality pattern in the data i.e. the meter readings are very low during early morning and night time whereas it peaks during the daytime. I think this pattern denotes a household or a small company which switches off its energy requirements during these wee hours.\n \n - For Building Id 1, we can see that the meter reading is very much constant throughout the day and it keeps on for most of the days. In between there are sudden up-spikes and downspikes. This kind of pattern suggests this building must be a powerplant or a computer data center which needs continuous supply of energy. The spikes might suggest sudden energy surge or server downtime. \n \n - For other buildings, there is no identifiable pattern."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_for_date_range(7,80,'2016-07-01', '2016-08-01')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 3) Feature Engineering"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['month'] = df_train['timestamp'].dt.month\ndf_train['dayofweek'] = df_train['timestamp'].dt.dayofweek\ndf_train['hourofday'] = df_train['timestamp'].dt.hour","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 4) Reducing memory usage"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to reduce the DF size\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = reduce_mem_usage(df_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# b) building_metadata.csv"},{"metadata":{},"cell_type":"markdown","source":"### Step 1) Reading and Interpreting Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_building_metadata = read_and_interpret_data('building_metadata.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 2) Visualizing Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_hist(df,var_name):\n    plt.figure(figsize=(17,8))\n    plt.hist(df[var_name],bins = 50)\n    plt.title(f\"Histogram - {var_name}\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_hist(df_building_metadata,'year_built')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_hist(df_building_metadata,'floor_count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 3) Removing NAs\n\nWe can see that there are 774 NA values out of 1449 rows in the `year_built` column. (50% NA)\n\nProblem : If we impute all NA values with a single value(either mean, median or mode), this is going to significantly distort the distribution.\n\nSolution : For now, let's remove these two columns altogether to simplify things. Later we will try some other strategies for imputation."},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_building_metadata = df_building_metadata.drop(['year_built','floor_count'],axis = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 3.1) Imputation"},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\ndef fill_building_data(df, col):\n    df_notna = df[df[col].notnull()]\n    df_na = df[~df[col].notnull()]\n    filler_list = df[col].value_counts().index.tolist()[0:5]\n    df_na[col] = random.choices(filler_list,k = len(df_na))\n    print(df_na.head())\n    print(df_na.isna().sum())\n    return pd.concat([df_na,df_notna],axis=0).sort_values(\"building_id\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_building_metadata = fill_building_data(df_building_metadata,'year_built')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_building_metadata = fill_building_data(df_building_metadata,'floor_count')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_building_metadata.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_building_metadata.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_building_metadata['year_built'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_building_metadata['year_built'].fillna(1976, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_building_metadata['floor_count'].fillna(1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 4) Encode Categorical Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nle = LabelEncoder()\ndf_building_metadata[\"primary_use\"] = le.fit_transform(df_building_metadata[\"primary_use\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 4) Reducing memory usage"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_building_metadata = reduce_mem_usage(df_building_metadata)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# c) weather_train.csv"},{"metadata":{},"cell_type":"markdown","source":"### Step 1) Reading and Interpreting Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_train = read_and_interpret_data('weather_train.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 2) - Visualizing Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"for var in ['dew_temperature','air_temperature','wind_speed']:\n    plot_hist(df_weather_train,var)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 3) Handling NAs"},{"metadata":{},"cell_type":"markdown","source":"** a) Removing columns which have lot of NAs **"},{"metadata":{"trusted":true},"cell_type":"code","source":"import datetime\ndef fill_weather_dataset(weather_df):\n    \n    # Find Missing Dates\n    time_format = \"%Y-%m-%d %H:%M:%S\"\n#     start_date = datetime.datetime.strptime(weather_df['timestamp'].min(),time_format)\n#     end_date = datetime.datetime.strptime(weather_df['timestamp'].max(),time_format)\n    start_date = weather_df['timestamp'].min().to_pydatetime()\n    end_date = weather_df['timestamp'].max().to_pydatetime()\n#     total_hours = int(((end_date - start_date).total_seconds() + 3600) / 3600)\n#     hours_list = [(end_date - datetime.timedelta(hours=x)).strftime(time_format) for x in range(total_hours)]\n    total_hours = int(((end_date - start_date).total_seconds() + 3600) / 3600)\n    hours_list = np.array([(end_date - datetime.timedelta(hours=x)).strftime(time_format) for x in range(total_hours)])\n\n    missing_hours = []\n    for site_id in range(16):\n        \n        site_tot_hrs = df_weather_train[df_weather_train['site_id'] == 1]['timestamp']\n        site_hours = np.array([x.strftime(time_format) for x in site_tot_hrs])\n#         site_hours = np.array(weather_df[weather_df['site_id'] == site_id]['timestamp'])\n        new_rows = pd.DataFrame(np.setdiff1d(hours_list,site_hours),columns=['timestamp'])\n        new_rows['site_id'] = site_id\n        weather_df = pd.concat([weather_df,new_rows])\n\n        weather_df = weather_df.reset_index(drop=True)           \n\n    # Add new Features\n    weather_df[\"datetime\"] = pd.to_datetime(weather_df[\"timestamp\"])\n    weather_df[\"day\"] = weather_df[\"datetime\"].dt.day\n    weather_df[\"week\"] = weather_df[\"datetime\"].dt.week\n    weather_df[\"month\"] = weather_df[\"datetime\"].dt.month\n    \n    # Reset Index for Fast Update\n    weather_df = weather_df.set_index(['site_id','day','month'])\n\n    # AIR TEMPERATURE\n    air_temperature_filler = pd.DataFrame(weather_df.groupby(['site_id','day','month'])['air_temperature'].mean(),columns=[\"air_temperature\"])\n    weather_df.update(air_temperature_filler,overwrite=False)\n\n    # CLOUD COVERAGE\n    # Step 1\n    cloud_coverage_filler = weather_df.groupby(['site_id','day','month'])['cloud_coverage'].mean()\n    # Step 2\n    cloud_coverage_filler = pd.DataFrame(cloud_coverage_filler.fillna(method='ffill'),columns=[\"cloud_coverage\"])\n    weather_df.update(cloud_coverage_filler,overwrite=False)\n\n    # DEW TEMPERATURE\n    dew_temperature_filler = pd.DataFrame(weather_df.groupby(['site_id','day','month'])['dew_temperature'].mean(),columns=[\"dew_temperature\"])\n    weather_df.update(dew_temperature_filler,overwrite=False)\n\n    # SEA LEVEL PRESSURE\n    # Step 1\n    sea_level_filler = weather_df.groupby(['site_id','day','month'])['sea_level_pressure'].mean()\n    # Step 2\n    sea_level_filler = pd.DataFrame(sea_level_filler.fillna(method='ffill'),columns=['sea_level_pressure'])\n\n    weather_df.update(sea_level_filler,overwrite=False)\n\n    # WIND DIRECTION\n    wind_direction_filler =  pd.DataFrame(weather_df.groupby(['site_id','day','month'])['wind_direction'].mean(),columns=['wind_direction'])\n    weather_df.update(wind_direction_filler,overwrite=False)\n\n    # WIND SPEED\n    wind_speed_filler =  pd.DataFrame(weather_df.groupby(['site_id','day','month'])['wind_speed'].mean(),columns=['wind_speed'])\n    weather_df.update(wind_speed_filler,overwrite=False)\n\n    # PRECIPITATION DEPTH\n    # Step 1\n    precip_depth_filler = weather_df.groupby(['site_id','day','month'])['precip_depth_1_hr'].mean()\n    # Step 2\n    precip_depth_filler = pd.DataFrame(precip_depth_filler.fillna(method='ffill'),columns=['precip_depth_1_hr'])\n\n    weather_df.update(precip_depth_filler,overwrite=False)\n\n    weather_df = weather_df.reset_index()\n    weather_df = weather_df.drop(['datetime','day','week','month'],axis=1)\n        \n    return weather_df\n\ndef limit_dew_temp(air_temp, dew_temp):\n    if dew_temp > air_temp:\n        return air_temp\n    else:\n        return dew_temp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_train = fill_weather_dataset(df_weather_train)\ndf_weather_train['dew_temperature'] = df_weather_train.apply(lambda x: limit_dew_temp(x.air_temperature, x.dew_temperature), axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# drop_cols = ['cloud_coverage','precip_depth_1_hr','sea_level_pressure','wind_direction']\n# df_weather_train = df_weather_train.drop(drop_cols,axis =1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"** b) Imputing with the median value **"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_train.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualizing distributions after median imputations\nfor var in ['dew_temperature','air_temperature','wind_speed']:\n    plot_hist(df_weather_train,var)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observation - \n\nWe can confirm from the histograms above that the imputation performed did not change the variable distribution very much as there were very less rows having NA values. So we are good."},{"metadata":{},"cell_type":"markdown","source":"### Step 4) Reducing memory usage"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_train = reduce_mem_usage(df_weather_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepare Training Data"},{"metadata":{},"cell_type":"markdown","source":"Here, we notice that `df_train` dataset does not have many features available. But other datasets `df_weather_train` and `df_meta` have features which we can use to build model. So, we prepare training data by joining these 3 datasets. This way, we will have features available in the same place in one dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.merge(df_train,df_building_metadata,on = 'building_id')\n# train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_train['timestamp'] = pd.to_datetime(df_weather_train['timestamp'])\ntrain_df = pd.merge(train_df,df_weather_train,on = ['site_id','timestamp'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['square_feet'] =  np.log1p(train_df['square_feet'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ntarget = np.log1p(train_df[\"meter_reading\"])\nfeatures = train_df.drop('meter_reading', axis = 1)\ndel df_train, df_weather_train, train_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features=features.drop(\"timestamp\",axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"[var for var in dir() if isinstance(eval(var), pd.core.frame.DataFrame)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# LGBM Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"import lightgbm as lgb\nfrom sklearn.model_selection import KFold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical_features = [\"building_id\", \"site_id\", \"meter\", \"primary_use\", \"dayofweek\",\"month\",\"hourofday\"]\nparams = {\n    \"objective\": \"regression\",\n    \"boosting\": \"gbdt\",\n    \"num_leaves\": 1280,\n    \"learning_rate\": 0.05,\n    \"feature_fraction\": 0.85,\n    \"reg_lambda\": 2,\n    \"metric\": \"rmse\",\n}\n\nkf = KFold(n_splits=3)\nmodels = []\nfor train_index,test_index in kf.split(features):\n    train_features = features.loc[train_index]\n    train_target = target.loc[train_index]\n    \n    test_features = features.loc[test_index]\n    test_target = target.loc[test_index]\n    \n    d_training = lgb.Dataset(train_features, label=train_target,categorical_feature=categorical_features, free_raw_data=False)\n    d_test = lgb.Dataset(test_features, label=test_target,categorical_feature=categorical_features, free_raw_data=False)\n    \n    model = lgb.train(params, train_set=d_training, num_boost_round=1000, valid_sets=[d_training,d_test], verbose_eval=25, early_stopping_rounds=50)\n    models.append(model)\n    del train_features, train_target, test_features, test_target, d_training, d_test\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del features, target\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for model in models:\n    lgb.plot_importance(model)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Decision Tree Regression Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Import the regression tree model\n# from sklearn.tree import DecisionTreeRegressor\n# regression_model = DecisionTreeRegressor(criterion=\"mse\",min_samples_leaf=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Fit the model\n# x_train = train_df.drop(['meter_reading','timestamp'],axis = 1)\n# y_train = train_df['meter_reading']\n# regression_model.fit(x_train,y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# x_train.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Predict on Training Data\n# predicted_train = regression_model.predict(x_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Training Error -"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Compute the RMSLE\n# def RMSLE(pred,act): \n#     return np.sqrt(np.sum((np.log(pred+1)-np.log(act+1))**2)/len(act))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training Error\n# RMSLE(predicted_train,y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Checking Actual and Predicted values side by side\n# pd.DataFrame(zip(y_train,predicted_train),columns = ['Actual','Predicted']).iloc[2000000:10000000,].head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Testing Data"},{"metadata":{},"cell_type":"markdown","source":"# a) test.csv"},{"metadata":{},"cell_type":"markdown","source":"### Step 1) Reading and Interpreting Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = read_and_interpret_data('test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 2) Reducing memory usage"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = reduce_mem_usage(df_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 3) Feature Engineering"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test['month'] = df_test['timestamp'].dt.month\ndf_test['dayofweek'] = df_test['timestamp'].dt.dayofweek\ndf_test['hourofday'] = df_test['timestamp'].dt.hour","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_test.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = reduce_mem_usage(df_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# b) weather_test.csv"},{"metadata":{},"cell_type":"markdown","source":"### Step 1) Reading and Interpreting Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_test = read_and_interpret_data('weather_test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 2) - Visualizing Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"for var in ['dew_temperature','air_temperature','wind_speed']:\n    plot_hist(df_weather_test,var)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 3) Handling NAs"},{"metadata":{},"cell_type":"markdown","source":"** a) Removing columns which have lot of NAs **"},{"metadata":{"trusted":true},"cell_type":"code","source":"# drop_cols = ['cloud_coverage','precip_depth_1_hr','sea_level_pressure','wind_direction']\n# df_weather_test = df_weather_test.drop(drop_cols,axis =1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_weather_test.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"** b) Imputing with the median value **"},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_weather_test.fillna(df_weather_test.median(),inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_weather_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_weather_test.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualizing distributions after median imputations\n# for var in ['dew_temperature','air_temperature','wind_speed']:\n#     plot_hist(df_weather_test,var)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observation - \n\nWe can confirm from the histograms above that the imputation performed did not change the variable distribution very much as there were very less rows having NA values. So we are good."},{"metadata":{},"cell_type":"markdown","source":"# Prepare Testing Data"},{"metadata":{},"cell_type":"markdown","source":"Same as before, we join `df_test` , `df_weather_test` and `df_meta` to have features available in the same place in one dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.merge(df_test,df_building_metadata,on = 'building_id')\n\ntest_df = pd.merge(test_df,df_weather_test,on = ['site_id','timestamp'],how='left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = reduce_mem_usage(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"[var for var in dir() if isinstance(eval(var), pd.core.frame.DataFrame)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del df_test, df_weather_test, df_building_metadata\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"row_ids = test_df['row_id']\ntest_df = test_df.drop(['timestamp','row_id'],axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df[\"square_feet\"] = np.log1p(test_df[\"square_feet\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.fillna(test_df.median(),inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"results = []\nfor model in models:\n    if  results == []:\n        results = np.expm1(model.predict(test_df, num_iteration=model.best_iteration)) / len(models)\n    else:\n        results += np.expm1(model.predict(test_df, num_iteration=model.best_iteration)) / len(models)\n    del model\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del test_df, models\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Predict on Testing Data\n# predicted_test = regression_model.predict(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# submission_df = pd.DataFrame(zip(row_id,predicted_test),columns = ['row_id','meter_reading'])\nresults_df = pd.DataFrame({\"row_id\": row_ids, \"meter_reading\": np.clip(results, 0, a_max=None)})\ndel row_ids,results\ngc.collect()\nresults_df.to_csv(\"submission_lgbm1.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# submission_df = submission_df.sort_values(by = 'row_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# submission_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# submission_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# submission_df.to_csv(\"ashrae_submit.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from IPython.display import FileLink\n# FileLink(r'ashrae_submit.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**!!! PLEASE UPVOTE THE KERNEL IF YOU LIKE IT !!!**"},{"metadata":{},"cell_type":"markdown","source":"STILL IN PROGRESS ...."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 3) Handling NA values"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"### 1. Imputing Year built -"},{"metadata":{},"cell_type":"markdown","source":"Number of NAs - "},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_building_metadata['year_built'].isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there are 774 NA values out of 1449 rows in the `year_built` column. (50% NA)"},{"metadata":{},"cell_type":"markdown","source":"Variable Distribution - "},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.figure(figsize=(17,8))\n# plt.hist(df_building_metadata['year_built'],bins = 20)\n# plt.title(\"Histogram showing the Distribution of the Year in which Builidings were built\")\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Problem : If we impute all NA values with a single value(either mean, median or mode), this is going to significantly distort the distribution.\n\nSolution : From the histograms above, we see that most buildings were built around 1960 - 1975 and 2000 - 2010, so we will impute from this year range."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Making a list of mode years\n# mode_yr1,mode_yr2 = list(range(1960,1975)),  list(range(2000,2010))\n# mode_years = mode_yr1 + mode_yr2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import random\n# random.seed(123)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Replacing NAs by chooosing randomly from the mode years\n# nans = df_building_metadata['year_built'].isna()\n# length = sum(nans)\n# replacement = random.choices(mode_years, k=length)\n# df_building_metadata.loc[nans,'year_built'] = replacement","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.figure(figsize=(17,8))\n# plt.hist(df_building_metadata['year_built'],bins = 20)\n# plt.title(\"Histogram showing the Distribution of the Year in which Builidings were built\")\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.figure(figsize=(17,8))\n# plt.hist(df_building_metadata['year_built'],bins = 20)\n# plt.title(\"Histogram showing the Distribution of the Year in which Builidings were built\")\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note that the distribution has still increased around the mode years as there were lot of rows which were NA. But we can see it is kind of levelled out"},{"metadata":{},"cell_type":"markdown","source":"We can again check if there are any null values left out"},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_building_metadata[df_building_metadata.year_built.isnull()]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2. Imputing Floor Count -"},{"metadata":{},"cell_type":"markdown","source":"Number of NAs -"},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_building_metadata['floor_count'].isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there are 1094 NA values out of 1449 rows in the `floor_count` column. (75% NA)"},{"metadata":{},"cell_type":"markdown","source":"Variable Distribution -"},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.figure(figsize=(17,8))\n# plt.hist(df_building_metadata['floor_count'],bins = 20)\n# plt.title(\"Histogram showing the distribution of number of Floors in Buildings\")\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that this is an asymmetric distribution with most of the values taking 1 to 10 values"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Making a list of mode years\n# mode_floors = range(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Replacing NAs by chooosing randomly from the mode years\n# nans = df_building_metadata['floor_count'].isna()\n# length = sum(nans)\n# replacement = random.choices(mode_floors, k=length)\n# df_building_metadata.loc[nans,'floor_count'] = replacement","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.figure(figsize=(17,8))\n# plt.hist(df_building_metadata['floor_count'],bins = 20)\n# plt.title(\"Histogram showing the distribution of number of Floors in Buildings\")\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# [1,0,np.nan,2].isnull().replace(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# nan_yrs = sum(df_building_metadata.year_built.isnull())\n# rand_mode_year = random.choices(mode_years, k =nan_yrs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_building_metadata['year_built'].replace(np.nan,rand_mode_year)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# list(range(1960,1975))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# mode_years[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_building_metadata['year_built'] = df_building_metadata['year_built'].astype('int16')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#df_train['timestamp'].dt.weekday_name.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#df_train['day_of_weak'] = df_train['timestamp'].dt.day","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#df_train.building_id.nunique() * 24 * 365 ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#df_test = read_and_interpret_data('test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_weather_train = read_and_interpret_data('weather_train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#df_weather_test = read_and_interpret_data('weather_test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#df_meta = read_and_interpret_data('building_metadata.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Step 2) - Reducing Memory Usage\n1) From the information above, we can see that `df_train` and `df_test` consume a huge amount of memory (~1GB). But there is a scope to reduce the memory.\n\n2) We notice that the int and float datatypes are 64 bits, which is the underlying cause for this huge size of the datasets. We have a function available to reduce this size."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reducing dataframe size\n# df_train = reduce_mem_usage(df_train)\n# df_test = reduce_mem_usage(df_test)\n# df_weather_train = reduce_mem_usage(df_weather_train)\n# df_weather_test = reduce_mem_usage(df_weather_test)\n# df_meta = reduce_mem_usage(df_meta)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_df = df_train.join(df_meta.set_index('building_id'), on = 'building_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_df = train_df.join(df_weather_train.set_index('site_id'), on = 'site_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# pr = [0 for x in df_test['row_id'] == df_test.index.tolist()]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# len(pr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_test = df_test.drop('row_id',axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Step 2) - Note that the data is huge, so in order to be in the kernel limit, we append these two datasets into one."},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_test['meter_reading'] = 'NA'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_test_train = pd.concat([df_train,df_test])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# del df_train, df_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_test_train[df_test_train.meter_reading == 'NA'].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df_test_train_meta = df_test_train.join(df_meta.set_index('building_id'), on = 'building_id')\n# df_test_train_meta.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}