{"cells":[{"metadata":{},"cell_type":"markdown","source":" # Baseline Median Prediction"},{"metadata":{},"cell_type":"markdown","source":"\nThe below solution is a simple baseline created by calculating median meter reading values for each building and meter type and then appending them directly to the test set. The idea comes from the assumption that the future meter readings will be similar to the meter readings in the past for each building and meter type.\n\nInspiration for this solution comes from the past kaggle competition https://www.kaggle.com/c/web-traffic-time-series-forecasting which was also a time series forecasting problem. There were more than 100,000 separate time series that had to be used to predict future web traffic per page. A few solutions used the median visits over week time periods and one of the high placing solutions used the median predictions as features for modeling.\n\nhttps://www.kaggle.com/safavieh/median-estimation-by-fibonacci-et-al-lb-44-9\nhttps://www.kaggle.com/chechir/weekend-flag-median-with-wiggle\nhttps://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/44729#latest-305761\n"},{"metadata":{},"cell_type":"markdown","source":" ## What's Old is New Again  \n \nThe solution achieves a public LB of 1.61. This solution is certainly not high performing in the long run but it proves that a simple baseline is a good place to start. For time series forecasting, the most naive assumption for a baseline is that your future values will be the same as what you have seen previously. \n\nData cleaning, feature engineering, feature selection, cross-validation and model selection are as important as ever in this competition but if your model cannot outperform this naive method, you need to rethink your approach.\n"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# No modeling packages required\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---"},{"metadata":{},"cell_type":"markdown","source":"### 1. Read in the Data\nIn this solution, we will only need the training csv and the test csv as we are only interested in the meter reading values themselves. I will not read in the weather data or building metadata at all."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"PATH = '/kaggle/input/ashrae-energy-prediction/train.csv'\ndf = pd.read_csv(PATH)\ndf.head().append(df.tail())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From looking at the training data, we can see that we have a year of data in hour increments from 00:00 01/01/2016 to 23:00 12/31/2016. The data represents separate time series for each of the 1448 buildings and the 4 meter types."},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = '/kaggle/input/ashrae-energy-prediction/test.csv'\ntest = pd.read_csv(PATH)\ntest.head().append(test.tail())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From looking at the testing data, we can see that we are predicting nearly a year and a half of data in hour increments from 00:00 01/01/2017 to 23:00 05/09/2018. In this problem, we are predicting more in the future than we have data for."},{"metadata":{},"cell_type":"markdown","source":"---"},{"metadata":{},"cell_type":"markdown","source":"### 2. Visualize Each Meter Type\nIn addition to the 1448 buildings, the dataset has 4 different types of energy meter: electric, chilled water, steam and hot water. Not every building has every meter type but each building that has more than one type has a different time series for each type of meter. Below are representative visualizations of each of the meter types. Note that each meter type time series has a different pattern, measurement scale and seasonality. This means that we will have to account for the meter type in addition to the building number for our calculation.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Meter 0: Electric \nts = df.loc[df['building_id']== 206]\nts = ts.loc[ts['meter']== 0]\nN = len(ts)\nts['t'] = range(N)\nplt.figure(figsize=(30,5))\nsns.lineplot(x='t', y='meter_reading', data=ts)\nplt.xlabel(\"Time: 2016-01-01 00:00 - 2016-12-31 23:00 (hr)\", size = 12)\nplt.ylabel(\"Meter Reading (kWh)\", size = 12)\nplt.title('Building 206, Electric Meter', size = 16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Meter 1: Chilled Water\nts = df.loc[df['building_id']== 206]\nts = ts.loc[ts['meter']== 1]\nN = len(ts)\nts['t'] = range(N)\nplt.figure(figsize=(30,5))\nsns.lineplot(x='t', y='meter_reading', data=ts)\nplt.xlabel(\"Time: 2016-01-01 00:00 - 2016-12-31 23:00(hr)\", size = 12)\nplt.ylabel(\"Meter Reading (kWh)\", size = 12)\nplt.title('Building 206, Chilled Water Meter', size = 16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Meter 2: Steam\nts = df.loc[df['building_id']== 750]\nts = ts.loc[ts['meter']== 2]\nN = len(ts)\nts['t'] = range(N)\nplt.figure(figsize=(30,5))\nsns.lineplot(x='t', y='meter_reading', data=ts)\nplt.xlabel(\"Time: 2016-01-01 00:00 - 2016-12-31 23:00(hr)\", size = 12)\nplt.ylabel(\"Meter Reading (kWh)\", size = 12)\nplt.title('Building 750, Steam Meter', size = 16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Meter 3: Hot Water\nts = df.loc[df['building_id']== 206]\nts = ts.loc[ts['meter']== 3]\nN = len(ts)\nts['t'] = range(N)\nplt.figure(figsize=(30,5))\nsns.lineplot(x='t', y='meter_reading', data=ts)\nplt.xlabel(\"Time: 2016-01-01 00:00 - 2016-12-31 23:00 \", size = 12)\nplt.ylabel(\"Meter Reading (kWh)\", size = 12)\nplt.title('Building 206, Hot Water Meter', size = 16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---"},{"metadata":{},"cell_type":"markdown","source":"### 3. Calculate Median Meter Readings of Training Set\n\nWe will be creating two new features from the timestamp: week and hour. This will allow us to group by building, meter, week and hour to calculate the median."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set up variables of column names\ntime = 'timestamp'\ntarget = 'meter_reading'\nbuilding = 'building_id'\nmeter = 'meter'\nweek = 'week'\nhr = 'hour'\npred = 'prediction'\ntarget = 'meter_reading'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert time stamp to datetime \ndf[time] = pd.to_datetime(df[time])\ntest[time] = pd.to_datetime(test[time])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split time stamp into week and hour features\ndf[week] = df[time].dt.week\ndf[hr] = df[time].dt.hour\n\ntest[week] = test[time].dt.week\ntest[hr] = test[time].dt.hour\n\nprint(df.head())\nprint(test.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Group by building id, meter type, week and hour. Then calculate median value for each group and make a new series called prediction from this. \n\ngroup = [building, meter, week, hr]\ngp = df.groupby(group)[[target]].median().rename({target:pred},axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can do this because we only have one year of data so we are assuming that the predicted data will be the same as the old data for each week and hour. \nHowever, since we are predicting January - May twice in the test set, we will be repeating some of the predicted values."},{"metadata":{"trusted":true},"cell_type":"code","source":"gp.head().append(gp.tail())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Merge the predictions to the test set to align timestamps again\ntest = test.merge(gp, on=group, how='left')\ntest","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4. Read in Submission File and Append Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read in submission file\nPATH = '/kaggle/input/ashrae-energy-prediction/sample_submission.csv'\nsub = pd.read_csv(PATH)\nsub.head().append(sub.tail())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check that submission ids match test set ids \n# Add in median predictions to submission as target\nprint(len(sub))\nprint(len(test))\nprint((sub['row_id'] == test['row_id']).all())\n\n# Forward fill and back fill nans as a precaution\nsub[target] = test[pred].ffill().bfill()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub[[target]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Write csv for submission\nPATH = 'submission.csv'\nsub.to_csv(PATH, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.head().append(sub.tail())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Final check for nan values in submission\nsub.isna().any()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.isinf(sub).any()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"}},"nbformat":4,"nbformat_minor":1}