{"cells":[{"metadata":{},"cell_type":"markdown","source":" **Introduction**\n \n This notebook assesses the main characteristics of the training dataset and shows some basic relationships among the main variables"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\n#from time import time\nimport datetime as dt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Load data**"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_folder = '/kaggle/input/ashrae-energy-prediction/'\ntrain_df = pd.read_csv(data_folder + 'train.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I added the 1+log of the target variable (meter_reading). This metric is the same as the one used for evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['log_meter_reading'] = np.log(1+train_df['meter_reading'])\n# convert timestamp into datetime object\ntrain_df['timestamp'] = pd.to_datetime(train_df['timestamp'],format='%Y-%m-%d %H:%M:%S')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.tail()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training dataset contains data for 1448 different buildings.\n\nHowever, each building can have more than one meter type."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.groupby('building_id')['meter'].nunique().hist(bins= np.arange(0.5,5.5,1));\nplt.xticks(np.arange(1,5));\nplt.ylabel('# of buildings')\nplt.xlabel('# of different meter values per building');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most buildings have only one meter type, but the fraction of those with more than one is not negligible.\n\nOnly 13 buildings have all 4 meter types.\n\nAs shown by other participants, the meter type influences the time trend of the target variable.\n\nSo it makes sense to consider the combinations 'building/meter'"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_meter_datapoints = train_df.groupby(['building_id','meter'])['meter_reading'].count().to_frame()\nbuilding_meter_datapoints.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 2380 combinations of building/meter type"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_meter_datapoints.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As shown earlier, the training dataset covers for the whole year 2016, with data sampled every hour.\n\nSince 2016 was a leap year, the max number of data points can be 366*24=8784\n\nAnd in fact this is the max number of datapoints found."},{"metadata":{"trusted":true},"cell_type":"code","source":"building_meter_datapoints.meter_reading.max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max_data_points = 366*24\nsum(building_meter_datapoints.meter_reading==max_data_points)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bins = np.concatenate( ( np.arange(0,9000,1000), np.array([max_data_points-.5]) ) )\nmax_bin = np.array([max_data_points-.5, max_data_points+100])\n\nfig, ax = plt.subplots(figsize=(8,6))\nax.hist(building_meter_datapoints.meter_reading, bins=bins,histtype='bar')\nax.hist(building_meter_datapoints.meter_reading, bins=max_bin)\nplt.grid();\nplt.yscale('log');\nplt.xlabel('# of data points');\nplt.ylabel('building-meter occurrences');\n# The red bar represents cases with complete dataset\n# NOTE: the plot is in log-scale","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's find the building with fewest datapoints"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_meter_datapoints.loc[building_meter_datapoints.idxmin()]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b_403_0 = train_df[(train_df['building_id']==403) & (train_df['meter']==0)]\nb_403_0.plot(x='timestamp',y='meter_reading', figsize=(8,6));\nplt.grid();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Only 19 days of data collection. \n\nGood luck with predicting how the building behaves in the next one and a half years..."},{"metadata":{},"cell_type":"markdown","source":"**Correlation between quantities**\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read building metadata\nbuilding_df = pd.read_csv(data_folder + 'building_metadata.csv')\nbuilding_df['log_square_feet'] = np.log(building_df['square_feet'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(np.log10(1+train_df['meter_reading']))\nplt.yscale('log');\nplt.grid();\nplt.xlabel('log10[meter_reading]')\nplt.ylabel('datapoints');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The range of values for the target variable spans several order of magnitude\n\nInteresting to see a spike on the right end of the graph. Could they by outliers?"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(np.log10(1+building_df['square_feet']))\nplt.yscale('log');\nplt.grid();\nplt.xlabel('log10[square_feet]')\nplt.ylabel('# of buildings');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Building sizes vary across several orders of magnitude too.\n\nIgnoring all other variables, the correlation coefficient between building size and meter reading is very weak"},{"metadata":{"trusted":true},"cell_type":"code","source":"trainbuilding_df = train_df.join(building_df, on='building_id', rsuffix = 'r')\n\ntrainbuilding_df[['meter_reading','square_feet']].corr()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"However, in log scale, the correlation becomes moderate.\n\nNow consider that this correlation does not consider building primary use, weather, meter type and reading variation over time !!"},{"metadata":{"trusted":true},"cell_type":"code","source":"trainbuilding_df[['log_meter_reading','log_square_feet']].corr()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we restrict the choice to all buildings with a given primary use (eg. Education) and meter type (eg. 0), the correlation becomes larger"},{"metadata":{"trusted":true},"cell_type":"code","source":"trainbuilding_edu0_df = trainbuilding_df[(trainbuilding_df['primary_use']=='Education') & (trainbuilding_df['meter']==0)]\ntrainbuilding_edu0_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainbuilding_edu0_df[['log_meter_reading','log_square_feet']].corr()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To be continued with other variables..."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}