{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Project 11: Building Energy Consumption\n*Xinchang Li <br>\nNovember 29, 2020*\n## 0. Load Modules and Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load useful modules\nimport os\nimport gc\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns; sns.set()\nimport tensorflow as tf\nimport sklearn.metrics\nfrom sklearn.preprocessing import LabelEncoder","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Print all files in the input directory (auto-generated code from Kaggle)\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read in the .csv files as Pandas DataFrames\nbldg_meta = pd.read_csv('/kaggle/input/ashrae-energy-prediction/building_metadata.csv')\n\ntrain = pd.read_csv('/kaggle/input/ashrae-energy-prediction/train.csv')\nweather_train = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_train.csv')\n\n# To conserve RAM, the following files are loaded later before they are used.\n\n# test = pd.read_csv('/kaggle/input/ashrae-energy-prediction/test.csv')\n# weather_test = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_test.csv')\n# sample_sub = pd.read_csv('/kaggle/input/ashrae-energy-prediction/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Exploratory Data Analysis Recap\nThe full Exploratory Data Analysis (EDA) available here: https://www.kaggle.com/xinchangli/cee-498ds-project-11-eda\n\nThe [exploratory data analysis (EDA)](https://www.kaggle.com/xinchangli/cee-498ds-project-11-eda) on the ASHRAE Great Energy Predictor III dataset shows that: <br>\n1. **Target variable**: hourly `meter_reading` for 4 types of meters (`{0: electricity, 1: chilledwater, 2: steam, 3: hotwater}`) in 1449 building from 16 sites. <br>\n2. **Independent variables (features)** from dataset: <br>\n       `site_id`, `building_id`, `primary_use`, `square_feet`, `year_built`,\n       `floor_count`, `meter`, `air_temperature`,\n       `cloud_coverage`, `dew_temperature`, `precip_depth_1_hr`,\n       `sea_level_pressure`, `wind_direction`, `wind_speed`\n3. **Potential engineered features**: <br>\n       `month`, `day_of_week`, `hour`\n4. **Missing data in bldg_meta**: <br>\n    * primary_use: \t0 (0.0%)\n    * year_built: \t774 (53.4%)\n    * square_feet: \t0 (0.0%)\n    * floor_count: \t1094 (75.5%) <br>\n5. **Missing data in weather_train**:\n    * air_temperature: \t55 (0.0%)\n    * cloud_coverage: \t69173 (49.5%)\n    * dew_temperature: \t113 (0.1%)\n    * precip_depth_1_hr: \t50289 (36.0%)\n    * sea_level_pressure: \t10618 (7.6%)\n    * wind_direction: \t6268 (4.5%)\n    * wind_speed: \t304 (0.2%) <br>\n6. **Missing data in weather_test**:\n    * air_temperature: \t104 (0.0%)\n    * cloud_coverage: \t140448 (50.7%)\n    * dew_temperature: \t327 (0.1%)\n    * precip_depth_1_hr: \t95588 (34.5%)\n    * sea_level_pressure: \t21265 (7.7%)\n    * wind_direction: \t12370 (4.5%)\n    * wind_speed: \t460 (0.2%)\n<br>\n7. **Two outliers' `buildings_id`**: 740 and 1099 <br>\n8. **Independent variables that are correlated with each other** (correlation coefficient > 0.4):\n    * `square_feet` and `floor_count` \n    * `air_temperature` and `dew_temperature`\n    * `wind_direction` and `wind_speed`\n    \n\n## 2. Model Preparation\n### 2.1 Choossing the Model: Recurrent Neural Network with Long Short Term Memory (RNN-LSTM)\nThis dataset is in its essense a time-series dataset, which is what RNN is designed at handling. LSTM is one of the most effective and commonly used RNN that improves on RNN's diminishing gradient problem. The advantage of using RNN-LSTM is that instead of using engineer features to account for the time information, the model architecture inherently carries this info and learns the relationship between times, reducing the number of features needed. \n\n### 2.2 Training Data Preprocessing\n#### 2.2.1 Building Metadata\nWe will first treat the building meta data as it is used in both training and testing. `year_built` and `floor_count` are two features containing missing data. Since one site likely has buildings built around the same time, we will use the average `year_built` in one site to impute the missing values. Similarly, similar `primary_use` may mean buildings have similar number of floors, so we use the average `floor_count` of one `primary_use` to impute the missing floor counts."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fill in missing year_built with the mean of other buildings in the same site; \n# if none of the buildings in a site has year_built, then fill with the mean of the entire dataset\nyear_built_gp = bldg_meta.groupby('site_id')['year_built']\nbldg_meta['year_built'] = year_built_gp.transform(lambda x: x.fillna(x.mean()))\nbldg_meta['year_built'].fillna(np.nanmean(bldg_meta['year_built']), inplace=True)\nassert pd.isnull(bldg_meta['year_built']).sum() == 0\n\n\n# Fill in missing floor_count with the mean of other buildings of the same primary_use; \n# if none of the buildings of a primary use has year_built, then fill with the mean of the entire dataset\nfloor_count_gp = bldg_meta.groupby('primary_use')['floor_count']\nbldg_meta['floor_count'] = floor_count_gp.transform(lambda x: x.fillna(x.mean()))\nbldg_meta['floor_count'].fillna(np.nanmean(bldg_meta['floor_count']), inplace=True)\nassert pd.isnull(bldg_meta['floor_count']).sum() == 0\n\n# To conserve RAM:\ndel year_built_gp, floor_count_gp\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 2.2.2 Weather Data\nFor `weather_train`, we notice that there seems to be missing entries in the `weather_train` dataframe, i.e. for some hours in the training data there were not a single weather variable record. Since NaN values cannot be handled by RNN, we need to find the missing hours and fill them in as rows in `weather_train`."},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train.shape\n# this should have 140,544 records (16 x 24 x 366, 2016 is a leap year) -> 771 hours missing","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The following function is inspired by https://www.kaggle.com/aitude/ashrae-missing-weather-data-handling\ndef add_missing_hours(weather_df):\n    import datetime\n    time_format = \"%Y-%m-%d %H:%M:%S\"\n    start_date = datetime.datetime.strptime(weather_df['timestamp'].min(),time_format)\n    end_date = datetime.datetime.strptime(weather_df['timestamp'].max(),time_format)\n    total_hours = int(((end_date - start_date).total_seconds() + 3600) / 3600)\n    hours_list = [(end_date - datetime.timedelta(hours=x)).strftime(time_format) for x in range(total_hours)]\n\n    for site_id in range(16):\n        site_hours = np.array(weather_df[weather_df['site_id'] == site_id]['timestamp'])\n        new_rows = pd.DataFrame(np.setdiff1d(hours_list,site_hours),columns=['timestamp'])\n        new_rows['site_id'] = site_id\n        weather_df = pd.concat([weather_df,new_rows]).reset_index(drop=True)\n    return weather_df\n\nweather_train = add_missing_hours(weather_train)\n\n# Varify the number of entries\nweather_train.shape ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will now deal with the missing data in `weather_train`. Since most of the weather variables have clear seasonalities/follows a annual cycle, for each weather variable, we will imput the missing data with the average of the rest of the data in the respective month. To do that, we need to first convert the `timestamp` column into a datetime format, so later we can extract `month` from it. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def convert_to_datetime(df_list):\n    for df in df_list:\n        df['timestamp'] = pd.to_datetime(df.timestamp)\n    return df_list\n\ntrain, weather_train = convert_to_datetime([train, weather_train])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Columns to impute\nweather_cols = ['air_temperature','cloud_coverage', 'dew_temperature', 'precip_depth_1_hr', \n                'sea_level_pressure', 'wind_direction', 'wind_speed']\n\ndef fill_weather_nan(df):\n    # Create the month column\n    df['month'] = df.timestamp.dt.month\n    # Create groupby object\n    gp = df.groupby(['site_id', 'month'])\n    for col in weather_cols:\n        df[col] = gp[col].transform(lambda x: x.fillna(x.mean()))\n        df[col].fillna(np.nanmean(df[col]), inplace=True)\n    # Delete the month column after use\n    del df['month']\n    return df\n\nweather_train = fill_weather_nan(weather_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, We will merge the dataframes into `train_full`, remove the outlier buildings from the training data, and modify the data types to conserve RAM:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_full = bldg_meta.merge(train, on='building_id').merge(weather_train, on=['site_id', 'timestamp'])\nprint(train_full.shape)\n\nassert train_full.shape[0] == train.shape[0]\n#test_full = bldg_meta.merge(test, on='building_id').merge(weather_test, on=['site_id', 'timestamp'])\n\n# To release RAM:\ndel train, weather_train\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove the outliers from the training data\ntrain_full = train_full[(train_full['building_id']!=740)&(train_full['building_id']!=1099)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The following function is based on https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\ndef reduce_mem_usage(df):\n    start_mem_usg = df.memory_usage().sum() / 1024**2 \n    print(\"Memory usage of properties dataframe is :\",start_mem_usg,\" MB\")\n    NAlist = [] # Keeps track of columns that have missing values filled in. \n    for col in df.columns:\n        if df[col].dtype != object and df[col].dtype != 'datetime64[ns]':  # Exclude strings and datetime         \n            # Print current column type\n            print(\"******************************\")\n            print(\"Column: \",col)\n            print(\"dtype before: \",df[col].dtype)            \n            # make variables for Int, max and min\n            IsInt = False\n            mx = df[col].max()\n            mn = df[col].min()\n            print(\"min for this col: \",mn)\n            print(\"max for this col: \",mx)\n            # Integer does not support NA, therefore, NA needs to be filled\n            if not np.isfinite(df[col]).all(): \n                NAlist.append(col)\n                df[col].fillna(mn-1,inplace=True)  \n                   \n            # test if column can be converted to an integer\n            asint = df[col].fillna(0).astype(np.int64)\n            result = (df[col] - asint)\n            result = result.sum()\n            if result > -0.01 and result < 0.01:\n                IsInt = True            \n            # Make Integer/unsigned Integer datatypes\n            if IsInt:\n                if mn >= 0:\n                    if mx < 255:\n                        df[col] = df[col].astype(np.uint8)\n                    elif mx < 65535:\n                        df[col] = df[col].astype(np.uint16)\n                    elif mx < 4294967295:\n                        df[col] = df[col].astype(np.uint32)\n                    else:\n                        df[col] = df[col].astype(np.uint64)\n                else:\n                    if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:\n                        df[col] = df[col].astype(np.int8)\n                    elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:\n                        df[col] = df[col].astype(np.int16)\n                    elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:\n                        df[col] = df[col].astype(np.int32)\n                    elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:\n                        df[col] = df[col].astype(np.int64)    \n            # Make float datatypes 32 bit\n            else:\n                df[col] = df[col].astype(np.float32)\n            \n            # Print new column type\n            print(\"dtype after: \",df[col].dtype)\n            print(\"******************************\")\n            \n    # Print final result\n    print(\"___MEMORY USAGE AFTER COMPLETION:___\")\n    mem_usg = df.memory_usage().sum() / 1024**2 \n    print(\"Memory usage is: \",mem_usg,\" MB\")\n    print(\"This is \",100*mem_usg/start_mem_usg,\"% of the initial size\")\n    return df, NAlist\n\ntrain_full, _ = reduce_mem_usage(train_full)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 2.2.3 Categorical Column: `primary_use`\nI first tried the one-hot encoding for the 16 primary use types, but it created very sparse data (i.e. every one-hot category column only has a small fraction of ones) and quickly consumed all memory. Hence I chose to use the label encoder from sklearn to convert the categories into integers."},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Convert primary_use into one-hot columns\n# train_full = train_full.join(pd.get_dummies(train_full.primary_use)).drop(columns = 'primary_use')\n# test_full = test_full.join(pd.get_dummies(test_full.primary_use)).drop(columns = 'primary_use')\n\nle = LabelEncoder()\ntrain_full[\"primary_use\"] = le.fit_transform(train_full[\"primary_use\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_full.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Model Training\n### 3.1 Creating Training Data Tensors\nFor RNN, training tensors need to have the following shape:\n    `[number of samples, number of timesteps, number of features]`\n\nEach sample needs to have the same shape. However, not every building has record for the whole of 2016. To handle this, I used the same truncating technique as in Class 12, with two major modifications:\n1. **Each sample is a building-meter pair**: this is to solve the problem that not every building has all meter types, and to conform the number of timesteps;\n2. **Setting a cleaning threshold (`THRES`)**: buildings with number of `meter_readings < THRES` will be discarded;\n3. **The start of record time period is truncated**: instead of truncating the time steps exceeding THRES from the end, I decided to truncate the start, because as observed in EDA, many sites have near-zero meter readings at the start of the training period, which likely is not generalizable and hence should be discarded."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data Cleaning Threshold\nTHRES = 8000 #7000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define feature columns\nfeat_cols = [#'site_id', 'building_id', \n    'square_feet', \n    'year_built', \n    'floor_count', \n    'meter', \n    'air_temperature',\n    'cloud_coverage', \n    'dew_temperature', \n    'precip_depth_1_hr', \n    'sea_level_pressure', \n    'wind_direction', \n    'wind_speed', \n    'primary_use'\n]\n\n# The following function is adapted from Module 12 Class 1 Notebook\n# Tensor: [number of building-meter pairs, number of timesteps, number of independent variables]\ndef prepare_train_data(df):\n    feat_list = []\n    target_list = []\n    for bldg_id in df.building_id.unique():\n        bldg = df[df.building_id == bldg_id]\n        for m in bldg.meter.unique():\n            # Each bldg_id-m pair is one sample (1)\n            met = bldg[bldg.meter == m]\n            # Keep only samples exceeding THRES (2)\n            if len(met.timestamp) >= THRES:\n                # Truncating from the start (3)\n                feat_list.append(np.array(np.float32(met[feat_cols][-THRES:])))\n                target_list.append(np.array(np.float32(met['meter_reading'][-THRES:])))\n            del met\n        del bldg\n    return (np.stack(feat_list), np.array(target_list))\n\ntrain_x, train_y = prepare_train_data(train_full)\n\n# Check the tensors' shapes\ntrain_x.shape, train_y.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.2 Transforming Target Variable Space\nBecause we have many heteorogenous feature varaibles having values of different orders of magnitude, we would like to use a normalization layer in our model architecture to transform the data into having zero means and unit standard deviations. If we can also transform the target variable, projecting the values onto a closer space to the training data, and that will help the model converge faster (I learned this the hard way...).\n\nI chose to do log transformation on the all target values plus one. This way, zero meter readings can also be handled without generating negative infinity, and the transformed data have the same order of magnitude as all feature variables. Moreover, unlike normalization/standardization, this transformation is self-contained, meaning we can transform the testing predictions back without relying on the information from the training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_y = np.log1p(train_y)\n\ntrain_y.mean(), train_y.std()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can combine the training data into a single dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = tf.data.Dataset.from_tensor_slices((train_x, train_y))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.3 Constructing the Model\nI built the RNN-LSTM model with the following architecture:\n1. **Normalization layer**: to transform the feature variables;\n2. **LSTM layer with return_sequence = True**: This will allow LSTM to generate one output at each time step;\n3. **Dense output layer**."},{"metadata":{"trusted":true},"cell_type":"code","source":"num_features = len(feat_cols)\n\n# From class 12\nmodel = tf.keras.Sequential()\n\nnorm = tf.keras.layers.experimental.preprocessing.Normalization()\nnorm.adapt(train_x)\n\n# Add normalization layer\nmodel.add(norm)\n\n# Add RNN: LSTM layer\nmodel.add(\n    tf.keras.layers.LSTM(units=32, # units is the number of hidden states\n                         input_shape = (None, num_features), # None to allow for flexible prediction length\n                         dropout = 0.2, # for regularization\n                         return_sequences = True) # So we get a prediction for each time step\n         ) \n\n# Add output layer\nmodel.add(tf.keras.layers.Dense(1)) # because we only want to predict one value; add 'activation=sigmoid' for classification (broken or not)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4 Compiling and Traing the Model\nWe will start with a relatively large learning rate, and monitor the losses with early stopping:"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=5e-4),\n              loss='mse',\n              metrics=[tf.keras.metrics.RootMeanSquaredError()])#, \n                       #tf.keras.metrics.MeanSquaredLogarithmicError()])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(train_ds.shuffle(50).batch(10), \n          epochs=30, \n          callbacks=tf.keras.callbacks.EarlyStopping(monitor='loss', patience=3))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Model Tuning\n### 4.1 Further Training\nWe will lower the learning rate and continue training more epochs. This two-step manual learning rate scheduling seem to generate better performance than using a constant learning rate, as we can see the decreasing trend slows down and plateaus towards the initial training, which likely suggests that model learning is at capacity.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),\n              loss='mse',\n              metrics=[tf.keras.metrics.RootMeanSquaredError()])#, \n                       #tf.keras.metrics.MeanSquaredLogarithmicError()])\nmodel.summary()\n\nmodel.fit(train_ds.shuffle(100).batch(10), \n          epochs=30, \n          callbacks=tf.keras.callbacks.EarlyStopping(monitor='loss', patience=3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Run this cell to save the model \nmodel.save('rnn_model')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 Hyperparamter Tuning and Other Adjustments\nI have attempted to improve the model performance by adjusting the following elements of the model:\n* Model architecture: whether to have dropout or not at the LSTM layer;\n* `THRES` value: a higher threshold means less samples but more training time steps, and vice versa. \n* Hyperparameters: such as learning rate, number of epochs and number of samples to shuffle.\n\nThe following table summarizes the changes I made for three of the submissions, as well as the scores. Note the score for the competition is Root Mean Squared Logarithmic Error (RMSLE) as defined by the competition.\n\n| Submission | Model Architecture  | `THRES` | Learning Rate                                     | Shuffle, Batch | EarlyStopping | Scores (Training, Testing) |\n|------------|---------------------|-----------------------------------|---------------------------------------------------|----------------|---------------|----------------------------|\n| 1          | LSTM w/o dropout    | 7,000                             | 1e-3 for 14/15 epochs, then 1e-4 for 10/20 epochs | 20, 10         | patience=3    | 1.696. 1.708               |\n| 2          | LSTM w/o dropout    | 8,000                             | 5e-4 for 25/25 epochs, then 1e-4 for 25/25 epochs | 50, 10         | patience=3    | 1.696, 1.681               |\n| 3          | LSTM w. dropout=0.2 | 8,000                             | 5e-4 for 30/30 epochs, then 1e-4 for 30/30 epochs | 100, 10        | patience=3    | 1.651, 1.623               |\n<br>\n\nChanges from 1 to 2 are mainly to test the effect of THRES, and 2 to 3 to test the effect of dropout. learning rate schedules etc. are also adjusted based on observations from other unsubmitted tries. "},{"metadata":{},"cell_type":"markdown","source":"## 5. Model Prediction and Submission\nWe need to start by releasing RAM held by the training data before we have enough memory to store and treat testing data."},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_full, train_x, train_y, train_ds\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we are ready to load and preprocess the testing data, following how we treated the training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load data\ntest = pd.read_csv('/kaggle/input/ashrae-energy-prediction/test.csv')\nweather_test = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fill in the missing hours in weather_test to match test\nweather_test = add_missing_hours(weather_test)\n\n# Convert timestamp in both dataframes into datetime format\ntest, weather_test = convert_to_datetime([test, weather_test])\n\n# Fill in missing weather records\nweather_test = fill_weather_nan(weather_test)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following two cells merge `bldg_meta`, `test` and `weather_test` into one dataframe (splitting into two cells due to memory limitation)."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_full = bldg_meta.merge(test, on='building_id')\ndel bldg_meta\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_full = test_full.merge(weather_test, on=['site_id', 'timestamp'])\ndel weather_test\n\nassert test_full.shape[0] == test.shape[0]\nprint(test_full.shape)\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reduce RAM usage\ntest_full, _ = reduce_mem_usage(test_full)\n\n# Encode primary_use column\ntest_full[\"primary_use\"] = le.transform(test_full[\"primary_use\"])\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We now have data ready to feed into the model. The first cell below is only used when session is forced to restart, to load the saved model back into memory."},{"metadata":{"trusted":true},"cell_type":"code","source":"# model = tf.keras.models.load_model('rnn_model')\n\n# feat_cols = [#'site_id', 'building_id', \n#     'square_feet', \n#     'year_built', \n#     'floor_count', \n#     'meter', \n#     'air_temperature',\n#     'cloud_coverage', \n#     'dew_temperature', \n#     'precip_depth_1_hr', \n#     'sea_level_pressure', \n#     'wind_direction', \n#     'wind_speed', \n#     'primary_use'\n# ]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since we constructed traing dataset by seperating samples based on building_id and meter type, we will also do predictions accordingly, looping through each building and each of its meters. We cannot compile testing data into a single array because 1) it causes too much memory overhead; and 2) each building may have different length of time for predictions.\n\nThe prediction results are first converted back into the original data space (by taking exponential and adding 1), then stored to the corresponding rows in the newly added `meter_reading` column in the original `test` dataframe. Using the original `test` dataframe is becuase we need to match predictions with `row_id`, as required by the submission."},{"metadata":{"trusted":true},"cell_type":"code","source":"test['meter_reading'] = np.zeros(test.shape[0], dtype=np.float32)\n\nfor bldg_id in test_full.building_id.unique():\n    bldg = test_full[test_full.building_id==bldg_id]\n    print(str(bldg_id)+', ', end='')\n    for m in bldg.meter.unique():\n        met = bldg[bldg.meter==m]\n        # adding a dim=1 at axis=0 to match the input layer shape\n        ts = np.expand_dims(met[feat_cols].values, axis=0) \n        del met\n        v = np.float32(np.expm1(model.predict(ts).squeeze()))\n        del ts\n        test.loc[(test.building_id==bldg_id)&(test.meter==m), 'meter_reading'] = v\n        del v\n    del bldg\n    gc.collect()\n\ntest\n# -- takes ~35 mins to run.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del test_full\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To prepare the final submission file, we first load the sample submission, keeping only `row_id` to conserve RAM. Then we will merge `test` and `sample_sub` into one dataframe, matching `row_id`. The merged dataframe is saved as a .csv file and submitted to the competition page for testing score evaluation."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub = pd.read_csv('/kaggle/input/ashrae-energy-prediction/sample_submission.csv',\n                         usecols=['row_id'])\nprint(sample_sub.shape)\nsample_sub","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub = sample_sub.merge(test[['row_id', 'meter_reading']], on='row_id')\nsample_sub","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Discussions\nIn this notebook we explored the use of RNN-LSTM on making time-series prediction for the ASHRAE building energy use dataset. The simple LSTM model is fairly effective, largely outperforming the baseline linear regression model (by Benjamin Smakic in our group) but slightly underperforming the lightgbm model (by Zhiyi Yang in our group). Model tuning I attempted improved the performance marginally but steadily. \n\nOther tuning opportunities I hope to explore if I had more time include: excluding some correlated features; increasing model complexity by adding one or more layers; other ways of handling missing data. \n\nThe most significant challenge has been combating the limited memory resources. A significant amount of time was spent on optimizing the memory usage; which is also helpful as that potentially has also improved the speed of training/predicting, and in the long run building foundations for dealing with larger data and more complex problems and models in the future. Another challenge was the data preprocessing. The trade-off between number of samples and number of timestamps can potentially be viewed as a shortcoming for RNN-LSTM (or rather my way of handling it) because it meant I am forced to leave behind part of the information from the raw data in training. Nevertheless, given the incomplete training data the performance is quite satisfactory, and I am happy to consider my first time building and training an RNN model a success.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}