{"cells":[{"metadata":{},"cell_type":"markdown","source":"## <center>Team Kullipojad<center>"},{"metadata":{},"cell_type":"markdown","source":" Team members: Ainika Adamson, Karl-Gustav Kallasmaa, Siim Kurvits, Laura Katrin Leman, Galina Avital Pass"},{"metadata":{},"cell_type":"markdown","source":"For analysing the data, we used an introductory notebook provided by the kind **CaesarLupum**. \nThe notebook is available here: https://www.kaggle.com/caesarlupum/ashrae-start-here-a-gentle-introduction"},{"metadata":{},"cell_type":"markdown","source":"Using CaesarLupum's notebook, we read the data in and reduced its memory usage with the following code:"},{"metadata":{"trusted":false},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport gc\n\n# matplotlib and seaborn for plotting\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns\nimport matplotlib.patches as patches\n\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\npd.set_option('max_columns', 150)\n\npy.init_notebook_mode(connected=True)\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\nimport os,random, math, psutil, pickle    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\nroot = 'ashrae-energy-prediction/'\ntrain_df = pd.read_csv(root + 'train.csv')\ntrain_df[\"timestamp\"] = pd.to_datetime(train_df[\"timestamp\"], format='%Y-%m-%d %H:%M:%S')\n\nweather_train_df = pd.read_csv(root + 'weather_train.csv')\ntest_df = pd.read_csv(root + 'test.csv')\nweather_test_df = pd.read_csv(root + 'weather_test.csv')\nbuilding_meta_df = pd.read_csv(root + 'building_metadata.csv')\nsample_submission = pd.read_csv(root + 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print('Size of train_df data', train_df.shape)\nprint('Size of weather_train_df data', weather_train_df.shape)\nprint('Size of weather_test_df data', weather_test_df.shape)\nprint('Size of building_meta_df data', building_meta_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"## Function to reduce the DF size\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df = reduce_mem_usage(train_df)\ntest_df = reduce_mem_usage(test_df)\n\nweather_train_df = reduce_mem_usage(weather_train_df)\nweather_test_df = reduce_mem_usage(weather_test_df)\nbuilding_meta_df = reduce_mem_usage(building_meta_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Merging, filling in NaNs etc"},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df['timestamp'] = pd.to_datetime(train_df['timestamp'])\ntest_df['timestamp'] = pd.to_datetime(test_df['timestamp'])\nweather_train_df['timestamp'] = pd.to_datetime(weather_train_df['timestamp'])\nweather_test_df['timestamp'] = pd.to_datetime(weather_test_df['timestamp'])\n    \nbuilding_meta_df['primary_use'] = building_meta_df['primary_use'].astype('category')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"   \ntemp_df = train_df[['building_id']]\ntemp_df = temp_df.merge(building_meta_df, on=['building_id'], how='left')\ndel temp_df['building_id']\ntrain_df = pd.concat([train_df, temp_df], axis=1)\n\ntemp_df = test_df[['building_id']]\ntemp_df = temp_df.merge(building_meta_df, on=['building_id'], how='left')\n\ndel temp_df['building_id']\ntest_df = pd.concat([test_df, temp_df], axis=1)\ndel temp_df, building_meta_df\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"temp_df = train_df[['site_id','timestamp']]\ntemp_df = temp_df.merge(weather_train_df, on=['site_id','timestamp'], how='left')\n\ndel temp_df['site_id'], temp_df['timestamp']\ntrain_df = pd.concat([train_df, temp_df], axis=1)\n\ntemp_df = test_df[['site_id','timestamp']]\ntemp_df = temp_df.merge(weather_test_df, on=['site_id','timestamp'], how='left')\n\ndel temp_df['site_id'], temp_df['timestamp']\ntest_df = pd.concat([test_df, temp_df], axis=1)\n\ndel temp_df, weather_train_df, weather_test_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df.to_pickle('train_df.pkl')\ntest_df.to_pickle('test_df.pkl')\n   \ndel train_df, test_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df = pd.read_pickle('train_df.pkl')\ntest_df = pd.read_pickle('test_df.pkl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df['age'] = train_df['year_built'].max() - train_df['year_built'] + 1\ntest_df['age'] = test_df['year_built'].max() - test_df['year_built'] + 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"\ntrain_df['floor_count'] = train_df['floor_count'].fillna(-999).astype(np.int16)\ntest_df['floor_count'] = test_df['floor_count'].fillna(-999).astype(np.int16)\n\ntrain_df['year_built'] = train_df['year_built'].fillna(-999).astype(np.int16)\ntest_df['year_built'] = test_df['year_built'].fillna(-999).astype(np.int16)\n\ntrain_df['age'] = train_df['age'].fillna(-999).astype(np.int16)\ntest_df['age'] = test_df['age'].fillna(-999).astype(np.int16)\n\ntrain_df['cloud_coverage'] = train_df['cloud_coverage'].fillna(-999).astype(np.int16)\ntest_df['cloud_coverage'] = test_df['cloud_coverage'].fillna(-999).astype(np.int16) \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df['month_datetime'] = train_df['timestamp'].dt.month.astype(np.int8)\ntrain_df['weekofyear_datetime'] = train_df['timestamp'].dt.weekofyear.astype(np.int8)\ntrain_df['dayofyear_datetime'] = train_df['timestamp'].dt.dayofyear.astype(np.int16)\n    \ntrain_df['hour_datetime'] = train_df['timestamp'].dt.hour.astype(np.int8)  \ntrain_df['day_week'] = train_df['timestamp'].dt.dayofweek.astype(np.int8)\ntrain_df['day_month_datetime'] = train_df['timestamp'].dt.day.astype(np.int8)\ntrain_df['week_month_datetime'] = train_df['timestamp'].dt.day/7\ntrain_df['week_month_datetime'] = train_df['week_month_datetime'].apply(lambda x: math.ceil(x)).astype(np.int8)\n    \ntrain_df['year_built'] = train_df['year_built']-1900\ntrain_df['square_feet'] = np.log(train_df['square_feet'])\n    \ntest_df['month_datetime'] = test_df['timestamp'].dt.month.astype(np.int8)\ntest_df['weekofyear_datetime'] = test_df['timestamp'].dt.weekofyear.astype(np.int8)\ntest_df['dayofyear_datetime'] = test_df['timestamp'].dt.dayofyear.astype(np.int16)\n    \ntest_df['hour_datetime'] = test_df['timestamp'].dt.hour.astype(np.int8)\ntest_df['day_week'] = test_df['timestamp'].dt.dayofweek.astype(np.int8)\ntest_df['day_month_datetime'] = test_df['timestamp'].dt.day.astype(np.int8)\ntest_df['week_month_datetime'] = test_df['timestamp'].dt.day/7\ntest_df['week_month_datetime'] = test_df['week_month_datetime'].apply(lambda x: math.ceil(x)).astype(np.int8)\n    \ntest_df['year_built'] = test_df['year_built']-1900\ntest_df['square_feet'] = np.log(test_df['square_feet'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Our contribution to feature engineering"},{"metadata":{},"cell_type":"markdown","source":"CaesarLupum's notebook provides an overview of what buildings were responsible for the max values in all four metrics. Because only 22 buildings made up the max values over dozens of days, we decided to eliminate these, as there may have been issues with the meters or some other explanation for why some values were unreasonably high, and we did not want these few outliers to cause changes in the results for the millions of other instances."},{"metadata":{"trusted":false},"cell_type":"code","source":"# The IDs of the notorious 22 buildings, taken from the graphs in  CaesarLupum's notebook \nids = [803,801,799,1088,993,794,1284,76,1258,1289,778,29,1156,29,60,50,1099,1197,1168,1148,1021,1221]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# copy of the train and test dataframes \ntrain_copy = train_df.copy(deep=True)\ntest_copy = test_df.copy(deep=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at the correlations between features, we found many that seemed to be worthless when determining the value of the meter, so we decided to drop these."},{"metadata":{"trusted":false},"cell_type":"code","source":"to_drop = ['precip_depth_1_hr',\n 'cloud_coverage',\n 'wind_direction',\n'primary_use' ,                            \n'year_built',       \n'weekofyear_datetime' ,\n'month_datetime' ,   \n'floor_count',          \n'sea_level_pressure',   \n'dew_temperature',       \n'cloud_coverage',        \n'day_week',              \n'wind_direction',        \n'day_month_datetime',    \n'week_month_datetime',  \n ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# We drop the previous columns and remove the 22 buildings\ntrain_copy = train_copy.drop(columns=to_drop,axis=1)\ntrain_copy = train_copy[~train_copy1.building_id.isin(ids)]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# It was a headache to feed this value to the model so we dropped it and prayed that the day and the hour would be enough\ntrain_copy = train_copy.drop(['timestamp'],axis=1)\ntest_copy = test_copy.drop(['timestamp'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We normalise some of the features with min-max normalisation, as they have very varying ranges and values."},{"metadata":{"trusted":false},"cell_type":"code","source":"train_copy['square_feet']=(train_copy['square_feet']-train_copy['square_feet'].min())/(train_copy['square_feet'].max()-train_copy['square_feet'].min())\ntrain_copy['air_temperature']=(train_copy['air_temperature']-train_copy['air_temperature'].min())/(train_copy['air_temperature'].max()-train_copy['air_temperature'].min())\ntrain_copy['age']=(train_copy['age']-train_copy['age'].min())/(train_copy['age'].max()-train_copy['age'].min())\ntrain_copy['wind_speed']=(train_copy['wind_speed']-train_copy['wind_speed'].min())/(train_copy['wind_speed'].max()-train_copy['wind_speed'].min())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Shuffle the train set and take out the value we want to predict."},{"metadata":{"trusted":false},"cell_type":"code","source":"shuffled_train = train_copy.sample(frac=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"y_train=shuffled_train['meter_reading']\nshuffled_train= shuffled_train.drop(['meter_reading'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Same goes for the test data, i.e drop useless columns and normalise."},{"metadata":{"trusted":false},"cell_type":"code","source":"test_copy = test_copy.drop(columns=to_drop,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# save the row id separately\nrow_id = test_copy['row_id']\ntest_copy = test_copy.drop(['row_id'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"test_copy['square_feet']=(test_copy['square_feet']-test_copy['square_feet'].min())/(test_copy['square_feet'].max()-test_copy['square_feet'].min())\ntest_copy['air_temperature']=(test_copy['air_temperature']-test_copy['air_temperature'].min())/(test_copy['air_temperature'].max()-test_copy['air_temperature'].min())\ntest_copy['age']=(test_copy['age']-test_copy['age'].min())/(test_copy['age'].max()-test_copy['age'].min())\ntest_copy['wind_speed']=(test_copy['wind_speed']-test_copy['wind_speed'].min())/(test_copy['wind_speed'].max()-test_copy['wind_speed'].min())\ntest_copy['dayofyear_datetime']=(test_copy['dayofyear_datetime']-test_copy['dayofyear_datetime'].min())/(test_copy['dayofyear_datetime'].max()-test_copy['dayofyear_datetime'].min())\ntest_copy['hour_datetime']=(test_copy['hour_datetime']-test_copy['hour_datetime'].min())/(test_copy['hour_datetime'].max()-test_copy['hour_datetime'].min())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model "},{"metadata":{},"cell_type":"markdown","source":"We used an entirely new model we hadn't yet seen in neither the introductory nor the baseline notebook. We thought using some gradient boosting models would be a good idea, as the instructors went for the same idea and it seems to be a very popular thing in Kaggle competitions. We wanted to make use of something else besides LightGBM, so we chose XGBoost instead. This is a fairly common and powerful optimised and distributed gradient boosting library that is supposed to be efficient, fast and flexible as well as reasonably accurate. You can read more about XGBoost here: https://xgboost.readthedocs.io/en/latest/"},{"metadata":{"trusted":false},"cell_type":"code","source":"import xgboost as xgb\nfrom sklearn.metrics import mean_squared_error","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We wanted to use the RMSLE function as the objective, but for some reason the version of XGBoost _pip_ automatically installed did not have this function included, so we made use of other ones. We chose a fairly general learning rate, and a max depth of 5 (the maximum is supposed to be 6). The documentation of XGBoost was massive and at times difficult to understand, or did not contain as much information as we would have liked, and we did not have much time to experiment with different parameters, so perhaps using a different learning rate, alpha, estimators etc would have yielded better results. Perhaps LightGBM would've been even better, but we wanted to try out an entirely new model instead of imitating the one chosen by the instructors. "},{"metadata":{"trusted":false},"cell_type":"code","source":"xgboost_reg = xgb.XGBRegressor(objective ='reg:squarederror', colsample_bytree = 0.3, learning_rate = 0.1,\n                max_depth = 5, alpha = 10, n_estimators = 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Fitting\nxgboost_reg.fit(shuffled_train,y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Predicting\npreds = xgboost_reg.predict(test_copy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Writing the results into a file \ndt = pd.DataFrame(data=preds)\ndt.to_csv('test_try_xgb.csv',  index=True)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.9"}},"nbformat":4,"nbformat_minor":1}