{"cells":[{"metadata":{},"cell_type":"markdown","source":"5.  [Modeling](#section-five)\n    - [Model Validation](#subsection-v)\n    - [Test Set](#subsection-test)\n    - [Testing Model](#subsection-pred)\n    \n6. [Conclusion](#section-six)\n    - [Model Validation](#subsection-future)"},{"metadata":{},"cell_type":"markdown","source":"# Data Preparation"},{"metadata":{},"cell_type":"markdown","source":"This notebook is the continuation of [ASHRAE - Energy Prediction1](https://www.kaggle.com/fatmanuranl/ashrae-energy-prediction1). Data is preapred according to EDA on that notebook. Run the cells below to prepare data for modeling."},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport gc\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Function for reducing size."},{"metadata":{"trusted":true},"cell_type":"code","source":"## Function to reduce the DF size\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Function to fill missing meterological data"},{"metadata":{"trusted":true},"cell_type":"code","source":"#function to fill missing meterological data\ndef site_mean_weather(table):\n    for col in list(table.columns[table.isnull().any()]):\n        imputaion = table.groupby(['site_id','hour','month'])[col].transform('mean')\n        table[col].fillna(imputaion,inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Data preparation function."},{"metadata":{"trusted":true},"cell_type":"code","source":"def prep_func(df):\n    # Drop unnecessary columns\n    df.drop(['floor_count','year_built', 'cloud_coverage','precip_depth_1_hr','sea_level_pressure'],axis=1,inplace=True)\n    gc.collect()\n    df['timestamp'] = pd.to_datetime(df['timestamp'])\n    # Encode meter types\n    df['meter'] = pd.Categorical(df['meter']).rename_categories({0: 'electricity', 1: 'chilledwater', 2: 'steam', 3: 'hotwater'})\n    gc.collect()\n    #Create time related features\n    df['hour'] = df.timestamp.dt.hour\n    df['month'] = df.timestamp.dt.month\n    # Fill missing data\n    site_mean_weather(df)\n    gc.collect()\n    #Create time related features\n    df['day'] = df.timestamp.dt.day\n    df[\"weekday\"] = df.timestamp.dt.weekday \n    gc.collect()\n    #Change column types to category\n    df[['primary_use','hour','month','site_id',\n    'building_id','wind_direction','weekday','day']] = df[['primary_use','hour','month','site_id','building_id','wind_direction','weekday','day']].astype('category')\n    gc.collect()\n    #Sort Data chronologically\n    df['timestamp'].sort_values().reset_index(drop=True)\n    gc.collect()\n    df.drop(['timestamp'],axis = 1, inplace = True)\n    gc.collect()\n    # Create weekend feature\n    df.loc[df['weekday'].isin([5, 6]), 'Weekend'] = 1\n    df['Weekend'].fillna(0,inplace = True)\n    df['Weekend'] = df['Weekend'].astype('bool')\n    gc.collect()\n    \n    df = reduce_mem_usage(df)\n    gc.collect()\n    \n    print('Data is ready')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run cell to import data."},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/ashrae-energy-prediction/train.csv')\nweather_train = pd.read_csv('../input/ashrae-energy-prediction/weather_train.csv')\nbuilding = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv')\ndf = pd.merge(train,building, on=\"building_id\", how=\"left\")\ndf = df.merge(weather_train, on=[\"site_id\", \"timestamp\"], how=\"left\")\ndel train, weather_train,building\ngc.collect()\nprint('Data is imported')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prep_func(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.loc[(df['site_id'] == 0) & (df['meter'] == 'electricity'), 'meter_reading'] = df[(df['site_id'] == 0) & (df['meter'] == 'electricity')]['meter_reading'].apply(lambda x: x* 0.2931 )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['cons/sqft'] = df['meter_reading'] / df['square_feet']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduce_mem_usage(df)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"section-five\"></a>\n# 1. Modeling"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Light Gradient Boosted Machine (LightGBM) model is chosen. \n* Faster training speed and higher efficiency.\n* Lower memory usage.\n* Better accuracy.\n* Support of parallel and GPU learning.\n* Capable of handling large-scale data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfrom sklearn import metrics\nfrom sklearn.metrics import mean_squared_error\nimport lightgbm as lgb\nfrom sklearn.model_selection import TimeSeriesSplit","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"subsection-v\"></a>\n# Model Validation"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = df.drop(['meter_reading','cons/sqft'], axis = 1) #Features\ngc.collect()\ny = df['cons/sqft'] #target\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For all model following parameters are used."},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical_columns = ['primary_use','hour','month','site_id','building_id','wind_direction','weekday','day','meter']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {\n    \"objective\": \"regression\",\n    \"boosting\": \"gbdt\", # gradient boosting\n    \"learning_rate\": 0.15,\n    \"num_leaves\": 30,\n    \"feature_fraction\": 0.6,\n    \"reg_lambda\": 2,\n    \"metric\": \"rmse\"}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run either time series  or kfold model. The predictipn part is same for both"},{"metadata":{},"cell_type":"markdown","source":"### Time Series Split Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Choose number of splits\ntss = TimeSeriesSplit(2)\ntss.split(X) \nfolds = tss.split(X) \nmodels = []\n#Spliting data\nfor train_index, test_index in folds:\n    X_train, X_test = X.iloc[train_index, :], X.iloc[test_index,:]\n    y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n    #print('TRAIN:', train_index, 'TEST:', test_index)\n#defining train and validation sets\n    lgb_train = lgb.Dataset(X_train, y_train)\n    lgb_valid = lgb.Dataset(X_test, y_test)\n    dftrainLGB = lgb.Dataset(data = X_train, label = y_train, feature_name = list(X_train))\n    del X_test, y_train, y_test\n    gc.collect()\n#model training\n    model = lgb.train(params, train_set=dftrainLGB, num_boost_round=1000, valid_sets=(lgb_train, lgb_valid), \n                      verbose_eval=75, early_stopping_rounds=200,categorical_feature = categorical_columns)\n    models.append(model)\n    del lgb_train, lgb_valid, dftrainLGB\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del X, y\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del tss, folds\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for model in models:\n    lgb.plot_importance(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del X_train\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#cv_results = lgb.cv(params, dftrainLGB, num_boost_round=200, nfold=4, \n                    #verbose_eval=10, early_stopping_rounds=40)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Kfold Model"},{"metadata":{},"cell_type":"markdown","source":"#Choose number of splits\nkf = KFold(n_splits=3)\nmodels = []\n#Spliting data\nfor train_index, test_index in  kf.split(X):\n    X_train, X_test = X.iloc[train_index, :], X.iloc[test_index,:]\n    y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n#defining train and validation sets\n    lgb_train = lgb.Dataset(X_train, y_train)\n    lgb_valid = lgb.Dataset(X_test, y_test)\n    dftrainLGB = lgb.Dataset(data = X_train, label = y_train, feature_name = list(X_train))\n    del X_test, y_train, y_test\n    gc.collect()\n#model training\n    model = lgb.train(params, train_set=dftrainLGB, num_boost_round=1000, valid_sets=(lgb_train, lgb_valid), \n                      verbose_eval=75, early_stopping_rounds=250,categorical_feature = categorical_columns)\n    models.append(model)\n    del lgb_train, lgb_valid, dftrainLGB\n    gc.collect()"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"subsection-test\"></a>\n## Test Set"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test = pd.read_csv('../input/ashrae-energy-prediction/test.csv')\nweather_test = pd.read_csv('../input/ashrae-energy-prediction/weather_test.csv')\nbuilding = pd.read_csv('../input/ashrae-energy-prediction/building_metadata.csv')\ntest = reduce_mem_usage(test)\nweather_test = reduce_mem_usage(weather_test)\nbuilding = reduce_mem_usage(building)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.merge(test,building, on=\"building_id\", how=\"left\")\ngc.collect()\ntest_df = test_df.merge(weather_test, on=[\"site_id\", \"timestamp\"], how=\"left\")\ngc.collect()\ndel test, weather_test, building\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"prep_func(test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"subsection-pred\"></a>\n## Testing Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"half1 = test_df.iloc[:20848800].drop('row_id',axis=1)\nhalf2 = test_df.iloc[20848800:].drop('row_id',axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results1 = []\nfor model in models:\n    if  len(results1)== 0:\n        results1 = (model.predict(half1, num_iteration=model.best_iteration)) / len(models)\n    else:\n        results1 += (model.predict(half1, num_iteration=model.best_iteration)) / len(models)\n    del model\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results2 = []\nfor model in models:\n    if  len(results2) == 0:\n        results2 = (model.predict(half2, num_iteration=model.best_iteration)) / len(models)\n    else:\n        results2 += (model.predict(half2, num_iteration=model.best_iteration)) / len(models)\n    del model\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del models\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = np.concatenate((results1, results2), axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del results1, results2\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['meter_reading'] = results * test_df['square_feet']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.drop(['building_id','primary_use','air_temperature','dew_temperature',\n              'wind_direction','wind_speed','hour','month','day','weekday','Weekend','square_feet'],axis = 1, inplace=True)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.loc[(test_df['site_id'] == 0) & (test_df['meter'] == 'electricity'), 'meter_reading'] = test_df[(test_df['site_id'] == 0) & (test_df['meter'] == 'electricity')]['meter_reading'].apply(lambda x: x / 0.2931 )\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output = pd.DataFrame({\"row_id\": test_df['row_id'], \"meter_reading\": test_df['meter_reading']})\ndel test_df\ngc.collect()\noutput.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"section-six\"></a>\n# 2. Conclusions"},{"metadata":{},"cell_type":"markdown","source":"Here is a summary table for all things I tried."},{"metadata":{},"cell_type":"markdown","source":"<a id=\"subsection-future\"></a>\n## Future Work"},{"metadata":{},"cell_type":"markdown","source":"* Removing outliers\n* Finding feature importance (changing selected features)\n* Changing parameters\n* Using cross validaiton \n* Trying different models for different meter types\n* Running model without divinding meter reading to area\n* Running model after taking log of meter readings\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}