{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport datetime\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Import Data**"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata = pd.read_csv('/kaggle/input/ashrae-energy-prediction/building_metadata.csv')\ntrain = pd.read_csv('/kaggle/input/ashrae-energy-prediction/train.csv')\ntest = pd.read_csv('/kaggle/input/ashrae-energy-prediction/test.csv')\nweather_train = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_train.csv')\nweather_test = pd.read_csv('/kaggle/input/ashrae-energy-prediction/weather_test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Size of the building_metadata dataset is', building_metadata.shape)\nprint('Size of the weather_train dataset is', weather_train.shape)\nprint('Size of the train dataset is', train.shape)\nprint('Size of the weather_test dataset is', weather_test.shape)\nprint('Size of the test dataset is', test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Explore Train Data**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The start date is', train.timestamp[0])\nprint('The last date is', train.timestamp[len(train) -1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observations from train dataset:\n\n1. The data is taken from Jan 1 2016 to Dec 31 2016\n2. There are 2,021,6100 data points in the  dataset\n3. The meter_reading is the target value and ranges from 0 to 21 million\n4. The 'meter' column is categorical, refers to the meter type, and is assigned an integer value from 0 to 3\n5. From ASHRAE EDA, we have the following id assignments for meter type: {0: electricity, 1: chilledwater, 2: steam, hotwater: 3} \n6. There are no missing data values "},{"metadata":{},"cell_type":"markdown","source":" **Explore Test Data**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The start date is', test.timestamp[0])\nprint('The last date is', test.timestamp[len(test) -1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observations from the test dataset\n\n1. There are 41,697,600 test data points.\n2. The test data is taken from Jan 1 2017 to May 9 2018 \n3. There are no missing values in the dataset"},{"metadata":{},"cell_type":"markdown","source":"**Explore Building Metadata**"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Also, each of the 1449 buildings has a site_id. We can determine the actual number of sites:\nprint(building_metadata.site_id.unique())\nprint(building_metadata.site_id.value_counts().sort_index())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observations from building_metadata\n\n1. There are a total of 1449 buildings/data points\n2. There are 16 different sites for the 1449 buildings\n3. Columns 'year_built' and 'floor_count' are missing values about 50% of values\n4. The 'primary use' column is categorical  "},{"metadata":{},"cell_type":"markdown","source":"**Explore weather_train**"},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The start date is', weather_train.timestamp[0])\nprint('The last date is', weather_train.timestamp[len(weather_train) -1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observations from weather_train\n\n1. Data is taken hourly from Jan 1 2016 to Dec 31 2016\n2. There are a total of 139,773 data points. Since there are 16 sites and the data is taken hourly in the year 2016, there should be (16)(24)(366) = 140,544 data points. Yet, there are only 139,733 data points, which means we are missing hourly data. \n3. There is also missing data from every weather feature\n4. The following units are given from the ASHRAE EDA:\n    * air_temperature - Degrees Celsius\n    * cloud_coverage - Portion of the sky covered in clouds, in oktas\n    * dew_temperature - Degrees Celsius\n    * precip_depth_1_hr - Millimeters\n    * sea_level_pressure - Millibar/hectopascals\n    * wind_direction - Compass direction (0-360)\n    * wind_speed - Meters per second\n"},{"metadata":{},"cell_type":"markdown","source":"**Explore Weather_test**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The start date is', weather_test.timestamp[0])\nprint('The last date is', weather_test.timestamp[len(weather_test) -1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_test.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observations from weather_test\n\n1. Data is taken from Jan 1 2017 to Dec 31 2018\n2. There are 277243 data points and there should be 280,320 data points, so there is missing hourly data\n3. There is also missing data for every weather feature"},{"metadata":{},"cell_type":"markdown","source":"**Data Cleaning**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Elminiate missing values from weather datasets by replacing NaN values with mean values taken for a month. \n# Source code taken from, https://www.kaggle.com/aitude/ashrae-missing-weather-data-handling#How-to-Use\n\ndef missing_statistics(df):    \n    statitics = pd.DataFrame(df.isnull().sum()).reset_index()\n    statitics.columns=['COLUMN NAME',\"MISSING VALUES\"]\n    statitics['TOTAL ROWS'] = df.shape[0]\n    statitics['% MISSING'] = round((statitics['MISSING VALUES']/statitics['TOTAL ROWS'])*100,2)\n    return statitics\n    \ndef fill_weather_dataset(weather_df):\n    \n    # Find Missing Dates\n    time_format = \"%Y-%m-%d %H:%M:%S\"\n    start_date = datetime.datetime.strptime(weather_df['timestamp'].min(),time_format)\n    end_date = datetime.datetime.strptime(weather_df['timestamp'].max(),time_format)\n    total_hours = int(((end_date - start_date).total_seconds() + 3600) / 3600)\n    hours_list = [(end_date - datetime.timedelta(hours=x)).strftime(time_format) for x in range(total_hours)]\n\n    missing_hours = []\n    for site_id in range(16):\n        site_hours = np.array(weather_df[weather_df['site_id'] == site_id]['timestamp'])\n        new_rows = pd.DataFrame(np.setdiff1d(hours_list,site_hours),columns=['timestamp'])\n        new_rows['site_id'] = site_id\n        weather_df = pd.concat([weather_df,new_rows])\n\n        weather_df = weather_df.reset_index(drop=True)           \n\n    # Add new Features\n    weather_df[\"datetime\"] = pd.to_datetime(weather_df[\"timestamp\"])\n    weather_df[\"day\"] = weather_df[\"datetime\"].dt.day\n    weather_df[\"week\"] = weather_df[\"datetime\"].dt.week\n    weather_df[\"month\"] = weather_df[\"datetime\"].dt.month\n    \n    # Reset Index for Fast Update\n    weather_df = weather_df.set_index(['site_id','day','month'])\n\n    air_temperature_filler = pd.DataFrame(weather_df.groupby(['site_id','day','month'])['air_temperature'].mean(),columns=[\"air_temperature\"])\n    weather_df.update(air_temperature_filler,overwrite=False)\n\n    # Step 1\n    cloud_coverage_filler = weather_df.groupby(['site_id','day','month'])['cloud_coverage'].mean()\n    # Step 2\n    cloud_coverage_filler = pd.DataFrame(cloud_coverage_filler.fillna(method='ffill'),columns=[\"cloud_coverage\"])\n\n    weather_df.update(cloud_coverage_filler,overwrite=False)\n\n    due_temperature_filler = pd.DataFrame(weather_df.groupby(['site_id','day','month'])['dew_temperature'].mean(),columns=[\"dew_temperature\"])\n    weather_df.update(due_temperature_filler,overwrite=False)\n\n    # Step 1\n    sea_level_filler = weather_df.groupby(['site_id','day','month'])['sea_level_pressure'].mean()\n    # Step 2\n    sea_level_filler = pd.DataFrame(sea_level_filler.fillna(method='ffill'),columns=['sea_level_pressure'])\n\n    weather_df.update(sea_level_filler,overwrite=False)\n\n    wind_direction_filler =  pd.DataFrame(weather_df.groupby(['site_id','day','month'])['wind_direction'].mean(),columns=['wind_direction'])\n    weather_df.update(wind_direction_filler,overwrite=False)\n\n    wind_speed_filler =  pd.DataFrame(weather_df.groupby(['site_id','day','month'])['wind_speed'].mean(),columns=['wind_speed'])\n    weather_df.update(wind_speed_filler,overwrite=False)\n\n    # Step 1\n    precip_depth_filler = weather_df.groupby(['site_id','day','month'])['precip_depth_1_hr'].mean()\n    # Step 2\n    precip_depth_filler = pd.DataFrame(precip_depth_filler.fillna(method='ffill'),columns=['precip_depth_1_hr'])\n\n    weather_df.update(precip_depth_filler,overwrite=False)\n\n    weather_df = weather_df.reset_index()\n    weather_df = weather_df.drop(['datetime','day','week','month'],axis=1)\n        \n    return weather_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_statistics(weather_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_statistics(weather_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_train = fill_weather_dataset(weather_train)\nweather_test = fill_weather_dataset(weather_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Missing Data from building_metadata\nmissing_statistics(building_metadata)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# floor count is missing too many values, probably best to delete it all together\n\ndel building_metadata['floor_count']\ndel building_metadata['year_built']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Reduce Memory**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Since the dataset sizes are quite large, it is important to reduce memory usage before proceeding\n# Source code taken from https://www.kaggle.com/cereniyim/save-the-energy-for-the-future-1-detailed-eda @ \n  \nfrom pandas.api.types import is_datetime64_any_dtype as is_datetime\nfrom pandas.api.types import is_categorical_dtype\n\n\ndef reduce_mem_usage(df, use_float16=False):\n    \"\"\"\n    Iterate through all the columns of a dataframe and modify the data type to reduce memory usage.        \n    \"\"\"\n    \n    start_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage of dataframe is {:.2f} MB\".format(start_mem))\n    \n    for col in df.columns:\n        if is_datetime(df[col]) or is_categorical_dtype(df[col]):\n            continue\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == \"int\":\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if use_float16 and c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype(\"category\")\n    \n    end_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage after optimization is: {:.2f} MB\".format(end_mem))\n    print(\"Decreased by {:.1f}%\".format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_metadata = reduce_mem_usage(building_metadata,use_float16=True)\nweather_train = reduce_mem_usage(weather_train,use_float16=True)\ntrain = reduce_mem_usage(train,use_float16=True)\n\nweather_test = reduce_mem_usage(weather_test,use_float16=True)\ntest = reduce_mem_usage(test,use_float16=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Prepare Training and Test Dataset**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Merge training dataset\ntrain = train.merge(building_metadata, left_on='building_id',right_on='building_id',how='left')\ntrain = train.merge(weather_train,how='left',left_on=['site_id','timestamp'],right_on=['site_id','timestamp'])\ndel weather_train\n\n# Merge test dataset\ntest = test.merge(building_metadata, left_on='building_id',right_on='building_id',how='left')\ntest = test.merge(weather_test,how='left',left_on=['site_id','timestamp'],right_on=['site_id','timestamp'])\ndel weather_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Replace categorical values \n\nfrom sklearn.preprocessing import LabelEncoder\n\nlb_make = LabelEncoder()\n\ntrain[\"primary_use\"] = lb_make.fit_transform(train[\"primary_use\"])\ntest['primary_use']= lb_make.fit_transform(test[\"primary_use\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Drop columns\ntrain = train.drop(['timestamp'], axis = 1)\ntest = test.drop(['timestamp'], axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split train labels and training \n\ny_train = np.log1p(train[\"meter_reading\"])\nx_train = train.drop(['meter_reading'], axis = 1)\ndel train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Replace NaN values\n# from sklearn.impute import SimpleImputer\n\n# imputer  = SimpleImputer(missing_values=np.nan, strategy='mean')\n\n# x_train = imputer.fit_transform(x_train.values)\n# test = imputer.fit_transform(test.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(np.isnan(x_train).sum())\nprint(np.isnan(test).sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Scale Data\n\n# from sklearn.preprocessing import MinMaxScaler\n\n# scaler = MinMaxScaler()\n# scaler.fit(x_train.values)\n# MinMaxScaler(copy=True, feature_range=(0, 1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Check Distribution of Data**"},{"metadata":{},"cell_type":"markdown","source":"The target values is in a normal distribution, this is due to the log transformation that was performed earlier. "},{"metadata":{},"cell_type":"markdown","source":"**Create Baseline Model**"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\n\nmodel = keras.Sequential([\n    keras.layers.Dense(12, activation='relu'),\n    keras.layers.Dense(1, kernel_initializer='normal')\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(loss='mean_squared_error',optimizer=optimizer)\nmodel.fit(x_train,y_train.values,epochs=10, batch_size = 100,validation_split = 1/5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import mean_squared_log_error\n\npredictions = model.predict(x_train)\nnp.sqrt(mean_squared_log_error( y_train, predictions ))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}