{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\n\nThis kernel helps to automate hyperparameter tuning for your lightgbm model. I'm using [Bayesian optimization](https://github.com/fmfn/BayesianOptimization) technique which gives you optimal set of parameters in less time spent than grid search or random search methods."},{"metadata":{},"cell_type":"markdown","source":"## Best Practices\n\nTrust me, these little tips can save your ton of efforts.\n\n* **A Good Start** - Don't try random combinations. Start with the best model publically shared. e.g [Half and Half](https://www.kaggle.com/rohanrao/ashrae-half-and-half)\n* **Be Organized** - Make a sheet and keep track of your all experiments so you can see progress and plan for next experiments. This script is already doing this for you.\n* **Don't Wait** - Such executions are time-consuming so commit your experiment and do other tasks :). I had a bad habit to look at screen while execution :)\n* **Multiple Commits** - Kaggle allows 10 CPU sessions simultaneously. It means you can do 10 experiments at a time. So plan multiple experiments on the sheet and commit them.\n"},{"metadata":{},"cell_type":"markdown","source":"## Import Libraries"},{"metadata":{"trusted":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport lightgbm as lgb\nfrom sklearn.preprocessing import LabelEncoder\nimport datetime\nimport gc\nfrom bayes_opt import BayesianOptimization\nimport warnings\nwarnings.filterwarnings('ignore')\n\nDATA_PATH = \"../input/ashrae-energy-prediction/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Data\n\nI'm experimenting with 1M training records (nrows=1000000) so you can remove it to use full dataset or change it.\n"},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df = pd.read_csv(DATA_PATH + 'train.csv',nrows=1000000)\ntrain_df = train_df [ train_df['building_id'] != 1099 ]\ntrain_df = train_df.query('not (building_id <= 104 & meter == 0 & timestamp <= \"2016-05-20\")')\nbuilding_df = pd.read_csv(DATA_PATH + 'building_metadata.csv')\nweather_df = pd.read_csv(DATA_PATH + 'weather_train.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Utility Functions"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Original code from https://www.kaggle.com/aitude/ashrae-missing-weather-data-handling by @aitude\n\ndef fill_weather_dataset(weather_df):\n    \n    # Find Missing Dates\n    time_format = \"%Y-%m-%d %H:%M:%S\"\n    start_date = datetime.datetime.strptime(weather_df['timestamp'].min(),time_format)\n    end_date = datetime.datetime.strptime(weather_df['timestamp'].max(),time_format)\n    total_hours = int(((end_date - start_date).total_seconds() + 3600) / 3600)\n    hours_list = [(end_date - datetime.timedelta(hours=x)).strftime(time_format) for x in range(total_hours)]\n\n    missing_hours = []\n    for site_id in range(16):\n        site_hours = np.array(weather_df[weather_df['site_id'] == site_id]['timestamp'])\n        new_rows = pd.DataFrame(np.setdiff1d(hours_list,site_hours),columns=['timestamp'])\n        new_rows['site_id'] = site_id\n        weather_df = pd.concat([weather_df,new_rows])\n\n        weather_df = weather_df.reset_index(drop=True)           \n\n    # Add new Features\n    weather_df[\"datetime\"] = pd.to_datetime(weather_df[\"timestamp\"])\n    weather_df[\"day\"] = weather_df[\"datetime\"].dt.day\n    weather_df[\"week\"] = weather_df[\"datetime\"].dt.week\n    weather_df[\"month\"] = weather_df[\"datetime\"].dt.month\n    \n    # Reset Index for Fast Update\n    weather_df = weather_df.set_index(['site_id','day','month'])\n\n    air_temperature_filler = pd.DataFrame(weather_df.groupby(['site_id','day','month'])['air_temperature'].mean(),columns=[\"air_temperature\"])\n    weather_df.update(air_temperature_filler,overwrite=False)\n\n    # Step 1\n    cloud_coverage_filler = weather_df.groupby(['site_id','day','month'])['cloud_coverage'].mean()\n    # Step 2\n    cloud_coverage_filler = pd.DataFrame(cloud_coverage_filler.fillna(method='ffill'),columns=[\"cloud_coverage\"])\n\n    weather_df.update(cloud_coverage_filler,overwrite=False)\n\n    due_temperature_filler = pd.DataFrame(weather_df.groupby(['site_id','day','month'])['dew_temperature'].mean(),columns=[\"dew_temperature\"])\n    weather_df.update(due_temperature_filler,overwrite=False)\n\n    # Step 1\n    sea_level_filler = weather_df.groupby(['site_id','day','month'])['sea_level_pressure'].mean()\n    # Step 2\n    sea_level_filler = pd.DataFrame(sea_level_filler.fillna(method='ffill'),columns=['sea_level_pressure'])\n\n    weather_df.update(sea_level_filler,overwrite=False)\n\n    wind_direction_filler =  pd.DataFrame(weather_df.groupby(['site_id','day','month'])['wind_direction'].mean(),columns=['wind_direction'])\n    weather_df.update(wind_direction_filler,overwrite=False)\n\n    wind_speed_filler =  pd.DataFrame(weather_df.groupby(['site_id','day','month'])['wind_speed'].mean(),columns=['wind_speed'])\n    weather_df.update(wind_speed_filler,overwrite=False)\n\n    # Step 1\n    precip_depth_filler = weather_df.groupby(['site_id','day','month'])['precip_depth_1_hr'].mean()\n    # Step 2\n    precip_depth_filler = pd.DataFrame(precip_depth_filler.fillna(method='ffill'),columns=['precip_depth_1_hr'])\n\n    weather_df.update(precip_depth_filler,overwrite=False)\n\n    weather_df = weather_df.reset_index()\n    weather_df = weather_df.drop(['datetime','day','week','month'],axis=1)\n        \n    return weather_df\n\n# Original code from https://www.kaggle.com/gemartin/load-data-reduce-memory-usage by @gemartin\n\nfrom pandas.api.types import is_datetime64_any_dtype as is_datetime\nfrom pandas.api.types import is_categorical_dtype\n\ndef reduce_mem_usage(df, use_float16=False):\n    \"\"\"\n    Iterate through all the columns of a dataframe and modify the data type to reduce memory usage.        \n    \"\"\"\n    \n    start_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage of dataframe is {:.2f} MB\".format(start_mem))\n    \n    for col in df.columns:\n        if is_datetime(df[col]) or is_categorical_dtype(df[col]):\n            continue\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == \"int\":\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if use_float16 and c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype(\"category\")\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage after optimization is: {:.2f} MB\".format(end_mem))\n    print(\"Decreased by {:.1f}%\".format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df\n\n\ndef features_engineering(df):\n    \n    # Sort by timestamp\n    df.sort_values(\"timestamp\")\n    df.reset_index(drop=True)\n    \n    # Add more features\n    df[\"timestamp\"] = pd.to_datetime(df[\"timestamp\"],format=\"%Y-%m-%d %H:%M:%S\")\n    df[\"hour\"] = df[\"timestamp\"].dt.hour\n    df[\"weekend\"] = df[\"timestamp\"].dt.weekday\n    holidays = [\"2016-01-01\", \"2016-01-18\", \"2016-02-15\", \"2016-05-30\", \"2016-07-04\",\n                    \"2016-09-05\", \"2016-10-10\", \"2016-11-11\", \"2016-11-24\", \"2016-12-26\",\n                    \"2017-01-02\", \"2017-01-16\", \"2017-02-20\", \"2017-05-29\", \"2017-07-04\",\n                    \"2017-09-04\", \"2017-10-09\", \"2017-11-10\", \"2017-11-23\", \"2017-12-25\",\n                    \"2018-01-01\", \"2018-01-15\", \"2018-02-19\", \"2018-05-28\", \"2018-07-04\",\n                    \"2018-09-03\", \"2018-10-08\", \"2018-11-12\", \"2018-11-22\", \"2018-12-25\",\n                    \"2019-01-01\"]\n    df[\"is_holiday\"] = (df.timestamp.isin(holidays)).astype(int)\n    df['square_feet'] =  np.log1p(df['square_feet'])\n    \n    # Remove Unused Columns\n    drop = [\"timestamp\",\"sea_level_pressure\", \"wind_direction\", \"wind_speed\",\"year_built\",\"floor_count\"]\n    df = df.drop(drop, axis=1)\n    gc.collect()\n    \n    # Encode Categorical Data\n    le = LabelEncoder()\n    df[\"primary_use\"] = le.fit_transform(df[\"primary_use\"])\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fill Weather Information\n\nI'm using [this kernel](https://www.kaggle.com/aitude/ashrae-missing-weather-data-handling) to handle missing weather information."},{"metadata":{"trusted":false},"cell_type":"code","source":"weather_df = fill_weather_dataset(weather_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Memory Reduction"},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df = reduce_mem_usage(train_df,use_float16=True)\nbuilding_df = reduce_mem_usage(building_df,use_float16=True)\nweather_df = reduce_mem_usage(weather_df,use_float16=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Merge Data\n\nWe need to add building and weather information into training dataset."},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df = train_df.merge(building_df, left_on='building_id',right_on='building_id',how='left')\ntrain_df = train_df.merge(weather_df,how='left',left_on=['site_id','timestamp'],right_on=['site_id','timestamp'])\ndel weather_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Features Engineering"},{"metadata":{"trusted":false},"cell_type":"code","source":"train_df = features_engineering(train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Features & Target Variables"},{"metadata":{"trusted":false},"cell_type":"code","source":"target = np.log1p(train_df[\"meter_reading\"])\nfeatures = train_df.drop('meter_reading', axis = 1)\ndel train_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##  Bayesian Optimization Functions"},{"metadata":{"trusted":false},"cell_type":"code","source":"def lgb_best_params(X, y,opt_params,init_points=2, optimization_round=20, n_folds=3, random_seed=0, cv_estimators=1000):\n    \n    # prepare dataset\n    categorical_features = [\"building_id\", \"site_id\", \"meter\", \"primary_use\", \"is_holiday\", \"weekend\"]\n    train_data = lgb.Dataset(data=X, label=y, categorical_feature = categorical_features, free_raw_data=False)\n    \n    def lgb_run(num_leaves, feature_fraction, bagging_fraction, max_depth, lambda_l1, lambda_l2, min_split_gain, min_child_weight,learning_rate):\n        params = {\"boosting\": \"gbdt\",'application':'regression','num_iterations':cv_estimators, 'early_stopping_round':int(cv_estimators/5), 'metric':'rmse'}\n        params[\"num_leaves\"] = int(round(num_leaves))\n        params['feature_fraction'] = max(min(feature_fraction, 1), 0)\n        params['bagging_fraction'] = max(min(bagging_fraction, 1), 0)\n        params['max_depth'] = int(round(max_depth))\n        params['lambda_l1'] = max(lambda_l1, 0)\n        params['lambda_l2'] = max(lambda_l2, 0)\n        params['min_split_gain'] = min_split_gain\n        params['min_child_weight'] = min_child_weight\n        params['learning_rate'] = learning_rate\n        cv_result = lgb.cv(params, train_data, nfold=n_folds, seed=random_seed, stratified=False, verbose_eval =cv_estimators, metrics=['rmse'])\n        return -min(cv_result['rmse-mean'])\n    \n    params_finder = BayesianOptimization(lgb_run, opt_params, random_state=2021)\n    # optimize\n    params_finder.maximize(init_points=init_points, n_iter=optimization_round)\n\n    # return best parameters\n    return params_finder.max","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configuration\n\nSmall range set is recommended. This [guide](https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html) will help you."},{"metadata":{"trusted":false},"cell_type":"code","source":"#  number of fold\nfold = 3\n\n# Hyperparameter range\nparams_range = {\n                'num_leaves': (1000, 1280),\n                'feature_fraction': (0.7, 0.9),\n                'bagging_fraction': (0.8, 1),\n                'max_depth': (10, 11),\n                'lambda_l1': (2, 5),\n                'lambda_l2': (2, 5),\n                'min_split_gain': (0.001, 0.1),\n                'min_child_weight': (5, 50),\n                'learning_rate' : (.05,.07)\n               }\n\n# You can experiments with different estimators in a single execution. I'm using small numbers for demonstration purpose so you must change to high numbers.\ncv_estimators = [50,100,200]\n\n#n_iter: How many steps of bayesian optimization you want to perform. The more steps the more likely to find a good maximum you are.\noptimization_round = 10 # Simply, 10 models with different parameter will be tested. And the best one will be returned.\n\n#init_points: How many steps of random exploration you want to perform. Random exploration can help by diversifying the exploration space.\ninit_points = 2\n\nrandom_seed = 2010","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Find Best Parameters"},{"metadata":{"trusted":false},"cell_type":"code","source":"best_params= []\nfor cv_estimator in cv_estimators:\n    opt_params = lgb_best_params(features, target,params_range, init_points=init_points, optimization_round=optimization_round, n_folds=fold, random_seed=random_seed, cv_estimators=cv_estimator)\n    opt_params['params']['iteration'] = cv_estimator\n    opt_params['params']['fold'] = fold\n    opt_params['params']['rmse'] = opt_params['target']\n    best_params.append(opt_params['params'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save CSV\n\nIt is good practise to keep track record of all experiments and best parameters."},{"metadata":{"trusted":false},"cell_type":"code","source":"df_params = pd.DataFrame(best_params).reset_index()\ndf_params = df_params[['iteration','fold','num_leaves','learning_rate','bagging_fraction',\n 'feature_fraction',\n 'lambda_l1',\n 'lambda_l2',\n 'max_depth',\n 'min_child_weight',\n 'min_split_gain',\n 'rmse']]\ndf_params.to_csv('best_params.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"df_params.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Give me your feedback and if you find my kernel is helpful, please UPVOTE**"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}