{"cells":[{"metadata":{},"cell_type":"markdown","source":"This kernel is a clean implementation of cross validation using different methods. Both will give you appox same results and you can adopt any one of these for testing your model performance.  "},{"metadata":{},"cell_type":"markdown","source":"# Implementation\n\n\n* Cross validation using cross_val_score method.\n* Cross validation using lgb.cv method.\n"},{"metadata":{},"cell_type":"markdown","source":"## Prepare Data\n\nI'm just loading data from csv and merging without any feature engineering. And then seperating features & target variables. To make this script easy to understand, I'm not applying any data preprocessing techniques here. "},{"metadata":{"trusted":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport gc\nimport warnings\nwarnings.filterwarnings('ignore')\n\nDATA_PATH = \"../input/ashrae-energy-prediction/\"\n\nfrom pandas.api.types import is_datetime64_any_dtype as is_datetime\nfrom pandas.api.types import is_categorical_dtype\n\ndef reduce_mem_usage(df, use_float16=False):\n    \"\"\"\n    Iterate through all the columns of a dataframe and modify the data type to reduce memory usage.        \n    \"\"\"\n    \n    start_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage of dataframe is {:.2f} MB\".format(start_mem))\n    \n    for col in df.columns:\n        if is_datetime(df[col]) or is_categorical_dtype(df[col]):\n            continue\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == \"int\":\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if use_float16 and c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype(\"category\")\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print(\"Memory usage after optimization is: {:.2f} MB\".format(end_mem))\n    print(\"Decreased by {:.1f}%\".format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df\n\ntrain_df = pd.read_csv(DATA_PATH + 'train.csv')\nbuilding_df = pd.read_csv(DATA_PATH + 'building_metadata.csv')\nweather_df = pd.read_csv(DATA_PATH + 'weather_train.csv')\n\ntrain_df = reduce_mem_usage(train_df,use_float16=True)\nbuilding_df = reduce_mem_usage(building_df,use_float16=True)\nweather_df = reduce_mem_usage(weather_df,use_float16=True)\n\ntrain_df = train_df.merge(building_df, left_on='building_id',right_on='building_id',how='left')\ntrain_df = train_df.merge(weather_df,how='left',left_on=['site_id','timestamp'],right_on=['site_id','timestamp'])\ndel building_df,weather_df\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Features & Target Variables"},{"metadata":{"trusted":false},"cell_type":"code","source":"target = train_df[\"meter_reading\"]\nfeatures = train_df.drop('meter_reading', axis = 1)\ndel train_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Understand RMSE & RMSLE\n\nThis is very confusing for novice programmers so first read this [article](https://medium.com/analytics-vidhya/root-mean-square-log-error-rmse-vs-rmlse-935c6cc1802a)."},{"metadata":{},"cell_type":"markdown","source":"# Python Formulation"},{"metadata":{"trusted":false},"cell_type":"code","source":"y_actual = np.array([2,4,6,8,10])\ny_pred = np.array([2,5,6,7,10])\n\n## Calculate RMSE - (R)sqrt->(M)mean>(Sqaure)power->(ERROR)loss\nrmse = np.sqrt( np.mean(  np.power( (y_pred-y_actual) ,2) ) )\nprint(\"RMSE Score is {:.2f}\".format(rmse))\n\n## Calculate RMSLE - (R)sqrt->(M)mean>(Sqaure)power->(L)log->(ERROR)loss\nrmsle = np.sqrt( np.mean(  np.power( (np.log1p(y_pred)-np.log1p(y_actual)) ,2) ) )\nprint(\"RMSLE Score is {:.2f}\".format(rmsle))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Conclusion** - So both are different and it matters in your evalutions. "},{"metadata":{},"cell_type":"markdown","source":"For this [competition](https://www.kaggle.com/c/ashrae-energy-prediction/overview/evaluation), evaluation metrics is **RMSLE** so keep focus on RMSLE."},{"metadata":{},"cell_type":"markdown","source":"## Convert Target Variable\n\nHere is a trick, If we transform the target variable with log1p then the evaluation metric **RMSLE** is the same as **rmse** as you can see in the formulation. "},{"metadata":{"trusted":false},"cell_type":"code","source":"target = np.log1p(target)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Method 1 - Using cross_val_score\n\ncross_val_score is widely used so below is the example to use this method with LGBMRegressor model. **neg_mean_squared_error** is available in the list of scoring parameters [here](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter) so use this and then calculate the square root."},{"metadata":{"trusted":false},"cell_type":"code","source":"from lightgbm import LGBMRegressor\nfrom sklearn.model_selection import cross_val_score\n\nlightgbm = LGBMRegressor( \n    task = 'train',\n    objective = \"regression\",\n    boosting = \"gbdt\", \n    num_leaves = 40,\n    learning_rate = 0.05,\n    feature_fraction = 1,\n    bagging_fraction = 1,\n    lambda_l1 = 5,\n    lambda_l2 = .1,\n    max_depth = 5,\n    min_child_weight = 1,\n    min_split_gain = 0.001,\n    num_boost_round=1,\n    verbose= 100)\n\nscores = cross_val_score(lightgbm, features, target, cv=3,scoring='neg_mean_squared_error')\n# first convert to positive and then sqrt.\nprint(\"Average cross-validation RMSLE score:{:.2f}\".format(np.sqrt(scores.mean()*-1)))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Method 2 - Using lgb.cv\n\nLightGBM has inbuilt method for [cross validation](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.cv.html) as well. **rmse** is available in the list of scoring parameters [here](https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric-parameters) so we're using this here."},{"metadata":{"trusted":false},"cell_type":"code","source":"import lightgbm as lgb\n\ntrain_data = lgb.Dataset(data=features, label=target, free_raw_data=False)\nparams = {}\nparams[\"task\"] = 'train'\nparams[\"objective\"] = 'regression'\nparams[\"boosting\"] = 'gbdt'\nparams[\"num_leaves\"] = 40\nparams['learning_rate'] = 0.05\nparams['feature_fraction'] = 1\nparams['bagging_fraction'] = 1\nparams['lambda_l1'] = 5\nparams['lambda_l2'] = .1\nparams['max_depth'] = 5\nparams['min_child_weight'] = 1\nparams['min_split_gain'] = 0.001\nparams['num_boost_round'] = 1\nparams['verbose'] = 100\n\ncv_result = lgb.cv(params, train_data, nfold=3,metrics='rmse',stratified=False)\nprint(\"Average cross-validation RMSLE score:{:.2f}\".format(np.min(cv_result['rmse-mean'])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'm using second method to find tunned hyper-parameters in [this](https://www.kaggle.com/aitude/ashrae-hyperparameter-tuning) kernel."},{"metadata":{},"cell_type":"markdown","source":"<font color=\"green\">**Give me your feedback and if you find my kernel is clean and helpful, please UPVOTE**</font>"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}