{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Building (and meter) models with Half and Half.\nThe concept of this notebook is creating models for each building and the meter.  \nThe feature importance by LGBM model is dominated by buliding_id. Then, it would be nice to make a model specific to that building (or even more for each meter in that building) isn't it?  \n\nNote that there might be some mistakes in code or logic since this is my first kernel, as well as the first Kaggle competition.\n\nAlso based on notebook \"Half and Half\"(https://www.kaggle.com/rohanrao/ashrae-half-and-half)  \nThanks"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import gc\nimport os\nimport random\n\nimport lightgbm as lgb\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\n\nfrom matplotlib import pyplot as plt\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.preprocessing import LabelEncoder\n\npath_data = \"/kaggle/input/ashrae-energy-prediction/\"\npath_train = path_data + \"train.csv\"\npath_test = path_data + \"test.csv\"\npath_building = path_data + \"building_metadata.csv\"\npath_weather_train = path_data + \"weather_train.csv\"\npath_weather_test = path_data + \"weather_test.csv\"\npath_drops = path_data + \"../lier-list/rows_to_drop.csv\"\npath_model = path_data + \"../building-models/building_meter_models.pkl\"\n\nplt.style.use(\"seaborn\")\nsns.set(font_scale=1)\n\nmyfavouritenumber = 0\nseed = myfavouritenumber\nrandom.seed(seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading train data\nReading train data along with building and weather metadata.  \nTo remove outliers, using data published in the following notebooks. \n(https://www.kaggle.com/purist1024/ashrae-simple-data-cleanup-lb-1-08-no-leaks/output)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(path_train)\n\nbuilding = pd.read_csv(path_building)\nle = LabelEncoder()\nbuilding.primary_use = le.fit_transform(building.primary_use)\n\nweather_train = pd.read_csv(path_weather_train)\n\n# remove outlier\nbad_rows = pd.read_csv(path_drops)\ndf_train = df_train.drop(index=bad_rows[\"0\"])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"## Memory optimization\n\n# Original code from https://www.kaggle.com/gemartin/load-data-reduce-memory-usage by @gemartin\n# Modified to support timestamp type, categorical type\n# Modified to add option to use float16\n\nfrom pandas.api.types import is_datetime64_any_dtype as is_datetime\nfrom pandas.api.types import is_categorical_dtype\n\ndef reduce_mem_usage(df, use_float16=False):\n    \"\"\"\n    Iterate through all the columns of a dataframe and modify the data type to reduce memory usage.        \n    \"\"\"\n    \n    start_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage of dataframe is {:.2f} MB\".format(start_mem))\n    \n    for col in df.columns:\n        if is_datetime(df[col]) or is_categorical_dtype(df[col]):\n            continue\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == \"int\":\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if use_float16 and c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype(\"category\")\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage after optimization is: {:.2f} MB\".format(end_mem))\n    print(\"Decreased by {:.1f}%\".format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = reduce_mem_usage(df_train, use_float16=True)\nbuilding = reduce_mem_usage(building, use_float16=True)\nweather_train = reduce_mem_usage(weather_train, use_float16=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing data\nThere are two files with features that need to be merged with the data. One is building metadata that has information on the buildings and the other is weather data that has information on the weather.   \n\nAlthough it is not written for simplicity, it is better to perform feature engineering especially on weather data."},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_data(X, building_data, weather_data, test=False):\n    \"\"\"\n    Preparing final dataset with all features.\n    \"\"\"    \n    \n    X = X.merge(building_data, on=\"building_id\", how=\"left\")\n    X = X.merge(weather_data, on=[\"site_id\", \"timestamp\"], how=\"left\")\n    \n    X.timestamp = pd.to_datetime(X.timestamp, format=\"%Y-%m-%d %H:%M:%S\")\n#     X.square_feet = np.log1p(X.square_feet)\n#     X.year_built = 2020 - X.year_built\n\n    \n    if not test:\n        X.sort_values(\"timestamp\", inplace=True)\n        X.reset_index(drop=True, inplace=True)\n    \n    gc.collect()\n    \n    holidays = [\"2016-01-01\", \"2016-01-18\", \"2016-02-15\", \"2016-05-30\", \"2016-07-04\",\n                \"2016-09-05\", \"2016-10-10\", \"2016-11-11\", \"2016-11-24\", \"2016-12-26\",\n                \"2017-01-01\", \"2017-01-16\", \"2017-02-20\", \"2017-05-29\", \"2017-07-04\",\n                \"2017-09-04\", \"2017-10-09\", \"2017-11-10\", \"2017-11-23\", \"2017-12-25\",\n                \"2018-01-01\", \"2018-01-15\", \"2018-02-19\", \"2018-05-28\", \"2018-07-04\",\n                \"2018-09-03\", \"2018-10-08\", \"2018-11-12\", \"2018-11-22\", \"2018-12-25\",\n                \"2019-01-01\"]\n    \n    X[\"hour\"] = X.timestamp.dt.hour\n    X[\"weekday\"] = X.timestamp.dt.weekday\n    X[\"is_holiday\"] = (X.timestamp.dt.date.astype(\"str\").isin(holidays)).astype(int)\n    \n    drop_features = [\"timestamp\", \"sea_level_pressure\", \"wind_direction\", \"wind_speed\",\n                    \"site_id\", \"primary_use\", \"square_feet\", \"year_built\", \"floor_count\"]\n    X.drop(drop_features, axis=1, inplace=True)\n    \n#     if test:\n#         row_ids = X.row_id\n#         X.drop(\"row_id\", axis=1, inplace=True)\n#         return X, row_ids\n#     else:\n#         y = np.log1p(X.meter_reading)\n#         X.drop(\"meter_reading\", axis=1, inplace=True)\n#         return X, y\n    return X","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"X_train = prepare_data(df_train, building, weather_train)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"del df_train\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Two-fold LightGBM Model split half-and-half\nThe data is split into two based on time. Each half is used as the training data for a model.\n\n**Half 1:** The first 50% rows of train data   \n**Half 2:** The last 50% rows of train data"},{"metadata":{"trusted":true},"cell_type":"code","source":"def LGBM(X_train, y_train):\n    X_half_1 = X_train[:int(X_train.shape[0] / 2)]\n    X_half_2 = X_train[int(X_train.shape[0] / 2):]\n\n    y_half_1 = y_train[:int(X_train.shape[0] / 2)]\n    y_half_2 = y_train[int(X_train.shape[0] / 2):]\n\n    # categorical_features = [\"building_id\", \"site_id\", \"meter\", \"primary_use\", \"hour\", \"weekday\"]\n    categorical_features = [\"hour\", \"weekday\"]\n\n    ###\n    # LGBM Setting\n\n    d_half_1 = lgb.Dataset(X_half_1, label=y_half_1, categorical_feature=categorical_features, free_raw_data=False)\n    d_half_2 = lgb.Dataset(X_half_2, label=y_half_2, categorical_feature=categorical_features, free_raw_data=False)\n\n    watchlist_1 = [d_half_1, d_half_2]\n    watchlist_2 = [d_half_2, d_half_1]\n\n    params = {\n        \"objective\": \"regression\",\n        \"boosting\": \"gbdt\",\n        \"num_leaves\": 40,\n        \"learning_rate\": 0.05,\n        \"feature_fraction\": 0.85,\n        \"reg_lambda\": 2,\n        \"metric\": \"rmse\"\n    }\n#     print(\"Building model with first half and validating on second half:\")\n    model_half_1 = lgb.train(params, train_set=d_half_1, num_boost_round=1000, valid_sets=watchlist_1, verbose_eval=200, early_stopping_rounds=200)\n\n#     print(\"Building model with second half and validating on first half:\")\n    model_half_2 = lgb.train(params, train_set=d_half_2, num_boost_round=1000, valid_sets=watchlist_2, verbose_eval=200, early_stopping_rounds=200)\n    return model_half_1, model_half_2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Modeling\nA model is created by extracting the training data corresponding to each building and meter.   \nBy repeating this, we finally get 2380 models."},{"metadata":{"trusted":true},"cell_type":"code","source":"def modeling(X_train, stop=999999):\n    drop_list = [\"building_id\", \"meter\", \"meter_reading\"]\n    ids = np.sort(X_train[\"building_id\"].unique())\n    df_model = pd.DataFrame(columns=[\"building_id\", \"meter\", \"half_1\", \"half_2\"])\n\n    index = 0\n    for bid in ids:\n        if stop < bid:\n            break\n        meters = np.sort(X_train[X_train[\"building_id\"] == bid][\"meter\"].unique())\n        for meter in meters:\n#             print(\"---------\\nModeling building_id:{}, meter:{}\\n---------\\n\".format(bid, meter))\n\n            # get train data\n            X_each = X_train[(X_train[\"building_id\"] == bid) & (X_train[\"meter\"] == meter)]\n            y_each = np.log1p(X_each.meter_reading)\n            X_each.drop(drop_list, axis=1, inplace=True)\n            \n            # LGBM\n            model_half_1, model_half_2 = LGBM(X_each, y_each)\n            \n            # make DataFrame of models\n            df_model.loc[index] = [bid, meter, model_half_1, model_half_2]\n            index+=1\n    return df_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ShortCut\nHere, I'm using a previously saved model for shortening.  \nPlese run the code cell below when modeling."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normal\n# df_model = modeling(X_train)\n# df_model.to_pickle(\"building_meter_models.pkl\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Short Cut\ndf_model = pd.read_pickle(path_model)\ndf_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature Importance\nPlotting some of the feature importance from LGBM."},{"metadata":{"trusted":true},"cell_type":"code","source":"def feature_importance(columns, models, building_id, meter):\n    df_fimp = pd.DataFrame()\n    for model in models:\n        df_fimp_each = pd.DataFrame()\n        df_fimp_each[\"feature\"] = columns\n        df_fimp_each[\"importance\"] = model.feature_importance()\n        df_fimp = pd.concat([df_fimp, df_fimp_each], axis=0)\n\n    plt.figure(figsize=(14, 7))\n    sns.barplot(x=\"importance\", y=\"feature\", data=df_fimp.sort_values(by=\"importance\", ascending=False))\n    plt.title(\"LightGBM Feature Importance building_id:{}, meter:{}\".format(building_id, meter))\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"drop_list = [\"building_id\", \"meter\", \"meter_reading\"]\nmodel_names = [\"half_1\", \"half_2\"]\ncolmuns = X_train[(X_train[\"building_id\"] == 0) & (X_train[\"meter\"] == 0)].drop(drop_list, axis=1).columns.values\nfor bid in [31, 15, 75]:\n    meters = np.sort(X_train[X_train[\"building_id\"] == bid][\"meter\"].unique())\n    for meter in meters:\n        models = []\n        for m_name in model_names:\n            model = df_model[(df_model[\"building_id\"] == bid) & (df_model[\"meter\"] == meter)][m_name].values[0]\n            models.append(model)\n        feature_importance(colmuns, models, bid, meter)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"del X_train\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing test data\nPreparing test data with same features as train data."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"df_test = pd.read_csv(path_test)\nweather_test = pd.read_csv(path_weather_train)\n\ndf_test = reduce_mem_usage(df_test)\nweather_test = reduce_mem_usage(weather_test)\n\nX_test = prepare_data(df_test, building, weather_test, test=True)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"del df_test, building\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Main Predicting\nSimilar to training, extract test data corresponding to building_id and meter and make predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"def predicting(X_test, df_model, stop=9999999):\n    drop_list = [\"building_id\", \"meter\", \"row_id\"]\n\n    ids = np.sort(X_test[\"building_id\"].unique())\n    submission = pd.DataFrame(columns=[\"row_id\", \"meter_reading\"])\n    for bid in ids:\n        if stop < bid:\n            break\n        meters = np.sort(X_test[X_test[\"building_id\"] == bid][\"meter\"].unique())\n        for meter in meters:\n#             print(\"---------\\nPredecting building_id:{}, meter:{}\\n---------\\n\".format(bid, meter))\n            \n            # make test data and rows\n            X_each = X_test[(X_test[\"building_id\"] == bid) & (X_test[\"meter\"] == meter)]\n            row_ids = X_each.row_id\n            X_each.drop(drop_list, axis=1, inplace=True)\n            \n            # predicting\n            model_half_1 = df_model[(df_model[\"building_id\"] == bid) & (df_model[\"meter\"] == meter)][\"half_1\"].values[0]\n            model_half_2 = df_model[(df_model[\"building_id\"] == bid) & (df_model[\"meter\"] == meter)][\"half_2\"].values[0]\n            pred = np.expm1(model_half_1.predict(X_each, num_iteration=model_half_1.best_iteration)) / 2\n            pred += np.expm1(model_half_2.predict(X_each, num_iteration=model_half_2.best_iteration)) / 2\n            \n            # concating submission\n            sub_each = pd.DataFrame({\"row_id\": row_ids, \"meter_reading\": np.clip(pred, 0, a_max=None)})\n            submission = pd.concat([submission, sub_each])\n    return submission.sort_values(\"row_id\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Scoring test data\nAveraging predictions from the two half train data models."},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = predicting(X_test, df_model)\nsubmission","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission\nPreparing final file for submission."},{"metadata":{"trusted":true},"cell_type":"code","source":"# submission = pd.DataFrame({\"row_id\": row_ids, \"meter_reading\": np.clip(pred, 0, a_max=None)})\nsubmission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**P.S.** If you vote up this kernel, please don't forget to vote up the original Half and Half: https://www.kaggle.com/rohanrao/ashrae-half-and-half"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}