{"cells":[{"metadata":{},"cell_type":"markdown","source":"My notebook for the [ASHRAE Great Energy Predictor III](https://www.kaggle.com/c/ashrae-energy-prediction) competition."},{"metadata":{},"cell_type":"markdown","source":"# Import libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# General libs\nimport os\nimport pandas as pd\nimport numpy as np\n\n# Data visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# ML/DL libs\n# from sklearn.model_selection import train_test_split \nfrom sklearn.linear_model import LinearRegression, SGDRegressor\n# from sklearn import metrics\nimport tensorflow as tf\nfrom tensorflow import keras\n\n# Other stuff\nimport pickle\n\n# Constants\nroot = '../input/ashrae-energy-prediction/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Exploration and Imputation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TODO: compartmentalize this code so it's callable for both test and train data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training data"},{"metadata":{"trusted":true},"cell_type":"code","source":"type_dict_meter = {  # Change data types to reduce size and training time.\n    'building_id': 'int16',\n    'meter': 'int8',\n    'meter_reading': 'float32'}\ntrain_df = pd.read_csv(\n    os.path.join(root, 'train.csv'),\n    dtype=type_dict_meter,\n    parse_dates=['timestamp'])\ntrain_df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is an additional column called timestamp. None of the columns are missing data."},{"metadata":{},"cell_type":"markdown","source":"### Weather training data"},{"metadata":{"trusted":true},"cell_type":"code","source":"type_dict_weather = {  # Change data types to reduce size and training time.\n    'site_id': 'int8',\n    'air_temperature': 'float32',\n    'cloud_coverage': 'float32',\n    'dew_temperature': 'float32',\n    'precip_depth_1_hr': 'float32',\n    'sea_level_pressure': 'float32',\n    'wind_direction': 'float32',\n    'wind_speed': 'float32'}\nwtrain_df = pd.read_csv(\n    os.path.join(root, 'weather_train.csv'),\n    dtype=type_dict_weather,\n    parse_dates=['timestamp'])\nwtrain_df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like there is a significant amount of data missing from the weather training set. We'll have to perform some inference to fill this in. Ignoring the missing data is not useful as it comprises a significant portion of the dataset. We'll need to infer using time-series methods. Use this: https://www.kaggle.com/juejuewang/handle-missing-values-in-time-series-for-beginners. Fortunately, this data is already sorted by site, then timestamp, meaning that we can easily interpolate without sorting.\n\nNote: cloud_coverage is in [oktas](https://en.wikipedia.org/wiki/Okta) which notate 1/8ths of the sky covered. A \"9\" means no data/sky not observable, which is \"usually due to dense fog or snow\", which implies heavy cloud coverage).\n\nWe also have a timestamp column, which is not missing any data.\n\nData that is missing completely by site:\n* 1: precip_depth_1_hr\n* 5: precip_depth_1_hr, sea_level_pressure\n* 7: cloud_coverage\n* 11: cloud_coverage\n* 12: precip_depth_1_hr"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fill in data that's entirely missing from some sites.\n# cloud_coverage as -1 as it's categorical\n# precip_depth_1_hr and sea_level_pressure as column mean\nwtrain_df.loc[wtrain_df['site_id'].isin([7,11]), 'cloud_coverage'] = wtrain_df.loc[wtrain_df['site_id'].isin([7,11]), 'cloud_coverage'].fillna(-1)\nwtrain_df.loc[wtrain_df['site_id'].isin([1,5,12]), 'precip_depth_1_hr'] = wtrain_df.loc[wtrain_df['site_id'].isin([1,5,12]), 'precip_depth_1_hr'].fillna(wtrain_df['precip_depth_1_hr'].mean())\nwtrain_df.loc[wtrain_df['site_id'].isin([5]), 'sea_level_pressure'] = wtrain_df.loc[wtrain_df['site_id'].isin([5]), 'sea_level_pressure'].fillna(wtrain_df['sea_level_pressure'].mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Perform linear interpolation for all items.\n# Manual review of the data looks like several sites just measure cloud coverage\n# once every few hours, which indicates that it probably doesn't change that fast.\nwtrain_df.interpolate(method='linear', inplace=True)\nwtrain_df.fillna(-1, inplace=True)  # linear method doesn't fill start/end values\n# Reference: https://medium.com/@drnesr/filling-gaps-of-a-time-series-using-python-d4bfddd8c460","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert cloud_coverage to int as it should be a category.\nwtrain_df['cloud_coverage'] = wtrain_df['cloud_coverage'].astype('int8')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Building Metadata"},{"metadata":{"trusted":true},"cell_type":"code","source":"type_dict_bm = {  # Change data types to reduce size and training time.\n    'site_id': 'int8',\n    'building_id': 'int16',\n    'square_feet': 'float32',\n    'year_built': 'float32',\n    'floor_count': 'float32'}\nbm_df = pd.read_csv(\n    os.path.join(root, 'building_metadata.csv'),\n    dtype=type_dict_bm)\nbm_df['primary_use'] = pd.Categorical(bm_df['primary_use']).codes\nbm_df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This set is also missing a lot of data. Looking at the data externally, floor_count seems to toggle by site (either all buildings at a site_id have a floor_count or none do), except for site 12. I'm not sure if that's significant. However, I can predict the floor_count using the square_feet since they are correlated. If I wanted to be more precise, I could also group by primary_use.\n\nSimilarly, the year_built is missing for several rows. However, its presence does not seem to toggle with other columns. There are no years listed before 1900, though it seems unlikely that 774 of the 1449 buildings were built before that year. This data also seems too important to ignore as more modern building methods should influence the overall power usage. I will likely just fill with the mean."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Perform floor_count imputation using linear regression model based on average of each floor_count category.\navg_sq_feet = bm_df.groupby(['floor_count']).mean()  # Will contain average square_feet for each floor_count\nsns.regplot(x=avg_sq_feet['square_feet'], y=avg_sq_feet.index.to_series())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dropping these two floor_counts as they only have 1 or 2 buildings each, and\n# regressing with them causes predictions for low square_feet values to be off.\navg_sq_feet.drop([13, 16], inplace=True)\n\n# Create regression model.\nfloor_reg = LinearRegression()\nfloor_reg.fit(\n    avg_sq_feet['square_feet'].values.reshape(-1, 1),  # x\n    avg_sq_feet.index.values.reshape(-1, 1))  # y\n\n# Fill df with predicted values.\nfor index, row in bm_df.iterrows():\n    if np.isnan(row['floor_count']):\n        bm_df.at[index, 'floor_count'] = round(floor_reg.predict(np.array([[row['square_feet']]]))[0][0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at values after filling - should be pretty linear.\navg_sq_feet = bm_df.groupby(['floor_count']).mean()\nsns.scatterplot(x=avg_sq_feet['square_feet'], y=avg_sq_feet.index.to_series())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Lazy imputation for year_built - we'll use the overall mean.\nbm_df['year_built'].fillna(round(bm_df['year_built'].mean()), inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Modify site 0 electric meter readings due to\n# https://www.kaggle.com/c/ashrae-energy-prediction/discussion/119261#latest-682635\n# meter 0 is the electricity\nsite_0_buildings = bm_df[bm_df['site_id']==0]['building_id']\ntrain_df.loc[(train_df['building_id'].isin(site_0_buildings)) & (train_df['meter']==0), 'meter_reading'] *= 0.2931","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature Engineering"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TODO\n# Align temp by (hour) timestamps to local time: https://www.kaggle.com/nz0722/aligned-timestamp-lgbm-by-meter-type\n# Do for weather data, do not for meter data (check each first)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add derived columns from timestamp.\n# year not useful as all data is in 2016, and day not as useful as month for seasonal trends.\nwtrain_df['month'] = wtrain_df['timestamp'].map(lambda x: x.month).astype('int8')\nwtrain_df['hour'] = wtrain_df['timestamp'].map(lambda x: x.hour).astype('int8')\nwtrain_df['weekday'] = wtrain_df['timestamp'].map(lambda x: x.dayofweek).astype('int8')\nwtrain_df['weekend'] = wtrain_df['weekday'].map(lambda x: 1 if x in [5,6] else 0).astype('bool')  # 5/6 are Sat/Sun","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Merge Datasets\n\nWe need to merge the weather, meter, and building data into a form usable for training. We need to join the train_df (meter data) with bm_df by building_id, and join the train_df with wtrain_df by timestamp and site_id. We also need to extract the meter readings as our output variable, and delete our unused dataframes."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.merge(left=train_df, right=bm_df, on='building_id')\ntrain_df = pd.merge(left=train_df, right=wtrain_df, on=['site_id', 'timestamp'])\ntrain_df.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"meter_reading = train_df.pop('meter_reading')  # Pop meter_reading off for training.\ntrain_df.drop('timestamp', axis=1, inplace=True)  # Remove timestamp.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del wtrain_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Selection and Training"},{"metadata":{},"cell_type":"markdown","source":"### Model Selection\n\nWhat type of model will work best for this data? We have a millions of data points, and a few dozen features at most. Predicted outputs must be quantitative, not categorical.\n\nGiven these requirements, [my favorite lazy method to decide on a model](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html) suggests using a [simple SGD regressor](https://scikit-learn.org/stable/modules/sgd.html). If this model gives poor performance, then I'll try a neural network with simple structure."},{"metadata":{},"cell_type":"markdown","source":"### Create Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Perform feature scaling prior to training SGD regressor.\nfeat_scal_dict = {}\nfor col in train_df.columns:\n    \n    if train_df[col].dtypes in ['int8', 'int16', 'int32', 'bool']:\n        continue\n    \n    t_max = train_df[col].max()\n    t_min = train_df[col].min()\n    diff = t_max - t_min\n    feat_scal_dict[col] = (t_max, t_min, diff)\n    train_df[col] -= t_min\n    train_df[col] /= diff","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_sgd = SGDRegressor(  # Currently set to library defaults.\n    loss='squared_loss',  # options: [squared_loss, huber, epsilon_insensitive, squared_epsilon_insensitive]\n    # epsilon=1  # for [huber, epsilon_insensitive, squared_epsilon_insensitive] models\n    max_iter=1000,\n    alpha=1e-4,\n    tol=1e-3,\n    # learning_rate='constant',  # Default is invscaling\n    eta0=1e0,  # Default is 1e-2\n    # power_t=0.9,  # Default is 0.5\n    early_stopping=False,\n    verbose=1)  # Default is 0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_sgd.fit(train_df, meter_reading)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Save Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"pickle.dump(model_sgd, open('model_sgd.model', 'wb'))\n# To load model:\n# model_sgd = pickle.load(open(filename, 'rb'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load and Modify Test Data"},{"metadata":{},"cell_type":"markdown","source":"### Test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TODO","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Weather test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TODO","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TODO","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# TODO: for SGD model, undo feature scaling.\n# use feat_scal_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Modify site 0 electric meter readings due to\n# https://www.kaggle.com/c/ashrae-energy-prediction/discussion/119261#latest-682635\n# meter 0 is the electricity\n# TODO: convert back for submission/scoring (multiply by 3.4118)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# References\n\nSome other notebooks I've referenced:\n* https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113338#latest-667617\n    * site_id likely corresponds to ASHRAE regions (there are 16); however may not be useful as some sites cover very large geographic areas\n    * ASHRAE regions: https://www.ashrae.org/communities/regions\n* https://www.kaggle.com/c/ashrae-energy-prediction/discussion/114483#latest-670519\n    * this guy is smart\n* https://www.kaggle.com/patrick0302/locate-cities-according-weather-temperature\n    * potential site locations\n* https://www.kaggle.com/hamditarek/reducing-memory-size-for-great-energy-predictor\n    * reducing memory size by 60%"},{"metadata":{"trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# Input data files are available in the \"../input/\" directory.\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}