{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ASHRAE - Great Energy Predictor III"},{"metadata":{},"cell_type":"markdown","source":"This notebook contains basic exploration of provided data and creation of recurent neural network model with tensorflow. \nThis it was kept in a simple manner in order to provide base line solution for LSTM, GRU and SimpleRNN. \n\nContext\nWith advancements in technology and increasing number of people world wide also the amount of energy is rising. To prevent negative impact of this growth we could find possibilities to optimize the energy consumed by buildings. In order to do that there is required some insights into \"energetic efficiency\" of buildings."},{"metadata":{},"cell_type":"markdown","source":"This implementation is based on the following code: https://www.kaggle.com/khalilo159/ashrae-prediction-using-lstm"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's upload some libraries"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"\n%matplotlib inline\nimport json\nimport os\nimport pandas as pd\nimport seaborn as sns\nimport statsmodels.formula.api as smf\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn import metrics\nfrom sklearn.model_selection  import train_test_split\nimport numpy as np\nimport gc\nfrom scipy.stats import norm # for scientific Computing\nfrom scipy import stats, integrate\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's load the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nASHRAE_train = pd.read_csv('/kaggle/input/ashrae-energy-prediction/train.csv')\nASHRAE_test = pd.read_csv('/kaggle/input/ashrae-energy-prediction/test.csv')\nweather_train = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_train.csv')\nweather_test = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_test.csv')\nbuilding_meta = pd.read_csv('/kaggle/input/ashrae-energy-prediction/building_metadata.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Done!"},{"metadata":{},"cell_type":"markdown","source":"## Reduce Data"},{"metadata":{},"cell_type":"markdown","source":" Function to reduce the DF size"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef reduce_memory_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduce_memory_usage(building_meta)\nreduce_memory_usage(weather_train)\nreduce_memory_usage(ASHRAE_train)\n\nreduce_memory_usage(weather_test)\nreduce_memory_usage(ASHRAE_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Done!"},{"metadata":{},"cell_type":"markdown","source":"## Data Preparation"},{"metadata":{},"cell_type":"markdown","source":"Let's visualize our dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_bar(data, name):\n    fig = plt.figure(figsize=(16, 9))\n    ax = fig.add_subplot(111)\n    data_label = data[name].value_counts()\n    dict_train = dict(zip(data_label.keys(), ((data_label.sort_index())).tolist()))\n    names = list(dict_train.keys())\n    values = list(dict_train.values())\n    plt.bar(names, values, color='green')\n    ax.set_xticklabels(names, rotation=45)\n    plt.grid()\n    plt.show()\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_bar(building_meta, 'primary_use')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"merging tables\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nBuildingTrain = building_meta.merge(ASHRAE_train, left_on='building_id', right_on='building_id' , how='left')\nBuildingTest = building_meta.merge(ASHRAE_test, left_on='building_id', right_on='building_id' , how='left')\nBuildingTrain.shape, BuildingTest.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Regarding the plot, we'll delete some data"},{"metadata":{"trusted":true},"cell_type":"code","source":"del ASHRAE_test\ndel ASHRAE_train\ndel building_meta\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BTW_train=BuildingTrain.merge(weather_train,left_on=['site_id','timestamp'],right_on=['site_id','timestamp'],how='left')\nBTW_test = BuildingTest.merge(weather_test,left_on=['site_id','timestamp'],right_on=['site_id','timestamp'],how='left')\nBTW_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del BuildingTest\ndel BuildingTrain\ndel weather_test\ndel weather_train\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Correlation Plot"},{"metadata":{"trusted":true},"cell_type":"code","source":"corrmat=BTW_train.corr()\nplt.figure(figsize = (20,11))\nsns.heatmap(corrmat,cmap=plt.cm.RdYlBu_r,vmin=-0.25,\n            annot=True,vmax=0.6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Filling out null values"},{"metadata":{"trusted":true},"cell_type":"code","source":"BTW_train = BTW_train.drop(columns=['year_built', 'floor_count', 'wind_direction', 'dew_temperature'])\nBTW_test = BTW_test.drop(columns=['year_built', 'floor_count','wind_direction', 'dew_temperature'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BTW_train ['timestamp'] =  pd.to_datetime(BTW_train['timestamp'])\nBTW_test ['timestamp'] =  pd.to_datetime(BTW_test['timestamp'])\nBTW_train['Month']=pd.DatetimeIndex(BTW_train['timestamp']).month\nBTW_test['Month']=pd.DatetimeIndex(BTW_test['timestamp']).month\nBTW_train['Day']=pd.DatetimeIndex(BTW_train['timestamp']).day\nBTW_test['Day']=pd.DatetimeIndex(BTW_test['timestamp']).day","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nBTW_train= BTW_train.groupby(['meter',BTW_train['building_id'],'primary_use',BTW_train['Month'], BTW_train['Day']]).agg({'meter_reading':'sum', 'air_temperature': 'mean', 'wind_speed': 'mean', 'precip_depth_1_hr': 'mean', 'cloud_coverage': 'mean', 'square_feet': 'mean'})\nBTW_test_1= BTW_test.groupby(['row_id','meter',BTW_test['building_id'],'primary_use',BTW_test['Month'], BTW_test['Day']]).agg({ 'air_temperature': 'mean', 'wind_speed': 'mean', 'precip_depth_1_hr': 'mean', 'cloud_coverage': 'mean', 'square_feet': 'mean'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BTW_train = BTW_train.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BTW_train['wind_speed'] = BTW_train['wind_speed'].astype('float32')\nBTW_train['air_temperature'] = BTW_train['air_temperature'].astype('float32')\nBTW_train['precip_depth_1_hr'] = BTW_train['precip_depth_1_hr'].astype('float32')\nBTW_train['cloud_coverage'] = BTW_train['cloud_coverage'].astype('float32')\nBTW_test['wind_speed'] = BTW_test['wind_speed'].astype('float32')\nBTW_test['air_temperature'] = BTW_test['air_temperature'].astype('float32')\nBTW_test['precip_depth_1_hr'] = BTW_test['precip_depth_1_hr'].astype('float32')\nBTW_test['cloud_coverage'] = BTW_test['cloud_coverage'].astype('float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BTW_train['precip_depth_1_hr'].fillna(method='ffill', inplace = True)\nBTW_train['cloud_coverage'].fillna(method='bfill', inplace = True)\n\nBTW_train['wind_speed'].fillna(BTW_train['wind_speed'].mean(), inplace=True)\nBTW_train['air_temperature'].fillna(BTW_train['air_temperature'].mean(), inplace=True)\n\nBTW_test['precip_depth_1_hr'].fillna(method='ffill', inplace = True)\nBTW_test['cloud_coverage'].fillna(method='bfill', inplace = True)\nBTW_test['precip_depth_1_hr'].fillna(BTW_test['precip_depth_1_hr'].mean(), inplace=True)\nBTW_test['cloud_coverage'].fillna(BTW_test['cloud_coverage'].mean(), inplace=True)\n\nBTW_test['wind_speed'].fillna(BTW_test['wind_speed'].mean(), inplace=True)\nBTW_test['air_temperature'].fillna(BTW_test['air_temperature'].mean(), inplace=True)\nBTW_train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Modelling Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"BTW_train.primary_use.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BTW_encoded = BTW_train[:]\nBTW_test_encoded = BTW_test[:]\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"label encoding "},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom sklearn.preprocessing import LabelEncoder\n\n\nle = LabelEncoder()\n\nBTW_encoded[\"primary_use\"] = le.fit_transform(BTW_encoded[\"primary_use\"])\nBTW_test_encoded[\"primary_use\"] = le.fit_transform(BTW_test_encoded[\"primary_use\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = BTW_encoded[['meter', 'building_id', 'primary_use', 'Month', 'Day','air_temperature', 'wind_speed', 'precip_depth_1_hr', 'cloud_coverage',\n       'square_feet']]\ny = BTW_encoded['meter_reading']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Splitting our data"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train, x_val, y_train, y_val = train_test_split(X,y, test_size = 0.2, random_state= 45)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Deep Learning Models"},{"metadata":{},"cell_type":"markdown","source":"Let's upload some libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom sklearn import preprocessing\nfrom keras import backend as K\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping\nfrom keras.layers import Dense, LSTM, GRU, Dropout, BatchNormalization\nfrom keras.models import Sequential\nfrom keras.optimizers import RMSprop,Adam\nfrom keras import regularizers","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's define our function loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"def root_mean_squared_error(y_true, y_pred):\n  return K.sqrt(K.mean(K.square(y_pred - y_true)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Fitting Models Function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_model(model,x_train,y_train,epochs=50,batch_size=500,verbose=1,validation_data=(x_val,y_val),callbacks =None):\n  x_train = x_train.values[:]\n  x_train= x_train.reshape((x_train.shape[0],1,x_train.shape[-1]))\n  y_train = np.log1p(y_train)\n  if validation_data != None:\n    x_val = validation_data[0].values[:]\n    x_val = x_val.reshape((x_val.shape[0],1,x_val.shape[-1]))\n    y_val = np.log1p(validation_data[-1])\n      \n  return model.fit(x_train,y_train,epochs=epochs,batch_size=batch_size,verbose=verbose,validation_data=(x_val,y_val),callbacks=callbacks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Stopping Condition"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nes = EarlyStopping(monitor='val_root_mean_squared_error', min_delta=0.0001, patience=5, verbose=True, mode='auto')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Simple RNN "},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.layers import SimpleRNN","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Architecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_model_rnn(input_dim=10,metrics=root_mean_squared_error,loss='mse', optimizer=\"rmsprop\",drop_rate=0.2):\n\n  model = Sequential()\n  model.add(SimpleRNN(128,return_sequences=True, input_shape=(None,input_dim)))\n  model.add(Dropout(drop_rate))\n  model.add(BatchNormalization())\n  model.add(SimpleRNN(128,return_sequences=False))\n  model.add(BatchNormalization())\n  model.add(Dropout(drop_rate))\n  model.add(Dense(1))\n  model.compile(optimizer=optimizer, loss=loss, metrics=[metrics, 'accuracy'])\n  \n  return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"simple_rnn_model = make_model_rnn(input_dim=x_train.shape[-1],drop_rate=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"simple_rnn_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = run_model(simple_rnn_model,x_train,y_train,epochs=30,batch_size=500,verbose=1,validation_data=(x_val,y_val), callbacks =[es]) # callbacks =[mc, es]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loss = history.history\nloss.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ploting rmse loss\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"rmse_loss_train = loss['root_mean_squared_error']\nrmse_loss_val = loss['val_root_mean_squared_error']\nepochs_stops = es.stopped_epoch +1 # epochs number from early stopping\nepochs = range(1,epochs_stops + 1)  #len(loss_train)\nplt.figure(figsize=(12,6))\nplt.plot(epochs,rmse_loss_train,'r', label='RMSE train loss')\nplt.plot(epochs,rmse_loss_val,'b',label='RMSE val loss')\nplt.title(' root mean square error loss')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## GRU Model"},{"metadata":{},"cell_type":"markdown","source":"Architecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_model(input_dim=10,metrics=root_mean_squared_error,loss='mse', optimizer=\"rmsprop\",drop_rate=0.2):\n\n  model = Sequential()\n  model.add(GRU(128,return_sequences=True, input_shape=(None,input_dim)))\n  model.add(Dropout(drop_rate))\n  model.add(BatchNormalization())\n  model.add(GRU(128,return_sequences=False))\n  model.add(BatchNormalization())\n  model.add(Dropout(drop_rate))\n  model.add(Dense(1))\n  model.compile(optimizer=optimizer, loss=loss, metrics=[metrics, 'accuracy'])\n  \n  return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gru_rnn_model = make_model_rnn(input_dim=x_train.shape[-1],drop_rate=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gru_rnn_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = run_model(gru_rnn_model,x_train,y_train,epochs=30,batch_size=500,verbose=1,validation_data=(x_val,y_val), callbacks =[es]) # callbacks =[mc, es]G","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loss = history.history\nloss.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ploting rmse loss\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"rmse_loss_train = loss['root_mean_squared_error']\nrmse_loss_val = loss['val_root_mean_squared_error']\nepochs_stops = es.stopped_epoch +1 # epochs number from early stopping\nepochs = range(1,epochs_stops + 1)  #len(loss_train)\nplt.figure(figsize=(12,6))\nplt.plot(epochs,rmse_loss_train,'r', label='RMSE train loss')\nplt.plot(epochs,rmse_loss_val,'b',label='RMSE val loss')\nplt.title(' root mean square error loss')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM Model"},{"metadata":{},"cell_type":"markdown","source":"LSTM Architecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_model(input_dim=10,metrics=root_mean_squared_error,loss='mse', optimizer=\"rmsprop\",drop_rate=0.2):\n\n  model = Sequential()\n  model.add(LSTM(128,return_sequences=True, input_shape=(None,input_dim)))\n  model.add(Dropout(drop_rate))\n  model.add(BatchNormalization())\n  model.add(LSTM(128,return_sequences=False))\n  model.add(BatchNormalization())\n  model.add(Dropout(drop_rate))\n  model.add(Dense(1))\n  model.compile(optimizer=optimizer, loss=loss, metrics=[metrics, 'accuracy'])\n  \n  return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = make_model(input_dim=x_train.shape[-1],drop_rate=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = run_model(model,x_train,y_train,epochs=30,batch_size=500,verbose=1,validation_data=(x_val,y_val), callbacks =[es]) # callbacks =[mc, es]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loss = history.history\nloss.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ploting rmse loss\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"rmse_loss_train = loss['root_mean_squared_error']\nrmse_loss_val = loss['val_root_mean_squared_error']\nepochs_stops = es.stopped_epoch +1 # epochs number from early stopping\nepochs = range(1,epochs_stops + 1)  #len(loss_train)\nplt.figure(figsize=(12,6))\nplt.plot(epochs,rmse_loss_train,'r', label='RMSE train loss')\nplt.plot(epochs,rmse_loss_val,'b',label='RMSE val loss')\nplt.title(' root mean square error loss')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# It is easy to deduce that the GRU model is the best performing one !"},{"metadata":{},"cell_type":"markdown","source":"# Submitting GRU Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# submit = pd.read_csv('/kaggle/input/ashrae-energy-prediction/sample_submission.csv')\n# x_test = BTW_test[['meter', 'building_id', 'primary_use', 'Month', 'Day','air_temperature', 'wind_speed', 'precip_depth_1_hr', 'cloud_coverage',#\n#        'square_feet']]\n# x_test = x_test.values[:]\n# x_test = x_test.reshape((x_test.shape[0],1,x_test.shape[-1]))\n# prediction = historyG.predict(x_test)\n# # We proceed with expo function\n# prediction = np.expm1(prediction)\n# submit['meter_reading'] = prediction\n# submit.to_csv('submission.csv', index=False,float_format='%.4f')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Thank you!!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}