{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Welcome to the ASHRAE - Great Energy Predictor Competition\nThis notebook is a starter code for all beginners and easy to understand.<br>\nWe focus on\n* a simple analysis of the data,\n* create new features,\n* hanlde missing data,\n* encoding and \n* scale data. <br>\n\nWe use categorical feature encoding techniques, compare<br>\nhttps://www.kaggle.com/drcapa/categorical-feature-encoding-challenge-xgb\n\nIn this kernel we consider the train data. For prediction we must repeate all operations also for the test data. <br>\nFinally we merge the train data with weahter and building date. After that we define X_train and y_train."},{"metadata":{},"cell_type":"markdown","source":"# Load Libraries\nWe need the standard python libraries and some libraries of sklearn."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy.special\nimport matplotlib.pyplot as plt\nimport os","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from sklearn.impute import SimpleImputer\nfrom sklearn.preprocessing import LabelEncoder","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"path_in = '../input/ashrae-energy-prediction/'\nprint(os.listdir(path_in))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = pd.read_csv(path_in+'train.csv', parse_dates=['timestamp'])\ntrain_weather = pd.read_csv(path_in+'weather_train.csv', parse_dates=['timestamp'])\nbuilding_data = pd.read_csv(path_in+'building_metadata.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Help functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_bar(data, name):\n    fig = plt.figure(figsize=(16, 9))\n    ax = fig.add_subplot(111)\n    data_label = data[name].value_counts()\n    dict_train = dict(zip(data_label.keys(), ((data_label.sort_index())).tolist()))\n    names = list(dict_train.keys())\n    values = list(dict_train.values())\n    plt.bar(names, values)\n    ax.set_xticklabels(names, rotation=45)\n    plt.grid()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Analysis\nFirst we do a simple analysis."},{"metadata":{},"cell_type":"markdown","source":"## Have a look on the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('# samples train_data:', len(train_data))\nprint('# samples train_weather:', len(train_weather))\nprint('# samples building_data:', len(building_data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_weather.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Extract missing data\n* train_data: no missing values\n* train_weather: there are some missing values we have to deal with\n* builing_data: there are missing values for the features year_build and floor_count\n\nThe missing data are numerical values."},{"metadata":{"trusted":true},"cell_type":"code","source":"cols_with_missing_train_data = [col for col in train_data.columns if train_data[col].isnull().any()]\ncols_with_missing_train_weather = [col for col in train_weather.columns if train_weather[col].isnull().any()]\ncols_with_missing_building = [col for col in building_data.columns if building_data[col].isnull().any()]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(cols_with_missing_train_data)\nprint(cols_with_missing_train_weather)\nprint(cols_with_missing_building)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature engineering\nWe handle the missing values, create new features and use encoding techniques based on<br>\nhttps://www.kaggle.com/drcapa/categorical-feature-encoding-challenge-xgb\n"},{"metadata":{},"cell_type":"markdown","source":"## Train data\n### New features\nBased on the timestamp we create new features for the month, the day the hour and the year. These are cyclic features. "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['month'] = train_data['timestamp'].dt.month\ntrain_data['day'] = train_data['timestamp'].dt.weekday\ntrain_data['hour'] = train_data['timestamp'].dt.hour","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Additionally we create the feature weekend: 5 = saturday and 6 = sunday."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['weekend'] = np.where((train_data['day'] == 5) | (train_data['day'] == 6), 1, 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Encoding\nWe created the features month, day and hour which are cyclic. "},{"metadata":{"trusted":true},"cell_type":"code","source":"features_cyc = {'month' : 12, 'day' : 7, 'hour' : 24}\nfor feature in features_cyc.keys():\n    train_data[feature+'_sin'] = np.sin((2*np.pi*train_data[feature])/features_cyc[feature])\n    train_data[feature+'_cos'] = np.cos((2*np.pi*train_data[feature])/features_cyc[feature])\ntrain_data = train_data.drop(features_cyc.keys(), axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 4 types of meters: <br>\n0 = electricity, 1 = chilledwater, 2 = steam, 3 = hotwater <br>\nWe use the one hot encoding for this 4 feature."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = pd.get_dummies(train_data, columns=['meter'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Building data\n### Handle missing data"},{"metadata":{"trusted":true},"cell_type":"code","source":"imp_most = SimpleImputer(strategy='most_frequent')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_data[cols_with_missing_building] = imp_most.fit_transform(building_data[cols_with_missing_building])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Encoding\nThe feature primary_use is a categorical feature with 16 categories. For the first we use a simple mapping."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_bar(building_data, 'primary_use')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"map_use = dict(zip(building_data['primary_use'].value_counts().sort_index().keys(),\n                     range(1, len(building_data['primary_use'].value_counts())+1)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_data['primary_use'] = building_data['primary_use'].replace(map_use)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_data = pd.get_dummies(building_data, columns=['primary_use'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Scale"},{"metadata":{"trusted":true},"cell_type":"code","source":"building_scale = ['square_feet', 'year_built', 'floor_count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mean = building_data[building_scale].mean(axis=0)\nbuilding_data[building_scale] = building_data[building_scale].astype('float32')\nbuilding_data[building_scale] -= building_data[building_scale].mean(axis=0)\nstd = building_data[building_scale].std(axis=0)\nbuilding_data[building_scale] /= building_data[building_scale].std(axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"building_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Weather data\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"weather_int = ['cloud_coverage']\nweather_cyc = ['wind_direction']\nweather_scale = ['air_temperature', 'dew_temperature', 'precip_depth_1_hr', 'sea_level_pressure', 'wind_speed']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Handle missing data "},{"metadata":{"trusted":true},"cell_type":"code","source":"imp_most = SimpleImputer(strategy='most_frequent')\ntrain_weather[cols_with_missing_train_weather] = imp_most.fit_transform(train_weather[cols_with_missing_train_weather])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Encoding\nThe feature wind_direction is the compass direction (0-360) and cyclic."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_weather['wind_direction'+'_sin'] = np.sin((2*np.pi*train_weather['wind_direction'])/360)\ntrain_weather['wind_direction'+'_cos'] = np.cos((2*np.pi*train_weather['wind_direction'])/360)\ntrain_weather = train_weather.drop(['wind_direction'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Scale"},{"metadata":{"trusted":true},"cell_type":"code","source":"mean = train_weather[weather_scale].mean(axis=0)\ntrain_weather[weather_scale] = train_weather[weather_scale].astype('float32')\ntrain_weather[weather_scale] -= train_weather[weather_scale].mean(axis=0)\nstd = train_weather[weather_scale].std(axis=0)\ntrain_weather[weather_scale] /= train_weather[weather_scale].std(axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_weather.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Merge the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = pd.merge(train_data, building_data, on='building_id', right_index=True)\ntrain_data = train_data.sort_values(['timestamp'])\ntrain_data = pd.merge_asof(train_data, train_weather, on='timestamp', by='site_id', right_index=True)\ndel building_data\ndel train_weather","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = train_data.sort_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define X_train and y_train"},{"metadata":{"trusted":true},"cell_type":"code","source":"no_feature = ['building_id', 'timestamp', 'meter_reading', 'site_id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = train_data[train_data.columns.difference(no_feature)].copy(deep=False)\ny_train = train_data['meter_reading']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Scale and rescale y_train\nThe target value ist energy consumption in kWh (or equivalent). Note that this is real data with measurement error, which we expect will impose a baseline level of modeling error.\nTo train we recommend to scale y_train and rescale the predicted y_test."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train_scaled = np.log1p(y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train_scaled.hist(bins=50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train_scaled[110:115]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test = np.expm1(y_train_scaled)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test[110:115]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation Metric\nThe evaluation metric for this competition is Root Mean Squared Logarithmic Error. <br>\nhttps://www.kaggle.com/c/ashrae-energy-prediction/overview/evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def rmse(y_true, y_pred):\n    \"\"\" root_mean_squared_error \"\"\"\n    return K.sqrt(K.mean(K.square(y_pred - y_true)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model\n**Current score: 3.09**\n\nTo create and train a model you can use for example a simple neural network like <br>\nhttps://www.kaggle.com/drcapa/ashrae-datagenerator-neuralnetwork\n\nThere is also used a DataGenerator. Because of the big data we need a lot of time to train end predict. We recommend to download the kernel and calculate local. So you have more than 9 hours and can reach good results. "},{"metadata":{},"cell_type":"markdown","source":"# Next Steps\n\nFurther the feature enginneering can be extended:\n* Are there any holidays?\n* Is there any realationship between the features?"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}