{"cells":[{"metadata":{"_uuid":"83c65ccf98c0f034d8fe3cab3acd238007473e27"},"cell_type":"markdown","source":"# 2.  Quick study: LGBM, XGB and Catboost\n\n&nbsp;\n\nHi, and welcome!  In this short kernel, we will: 1) run  baseline versions of `LightGBM`, `XGBoost` and `Catboost`  over the [Google Analytics Customer Revenue Prediction](https://www.kaggle.com/c/google-analytics-customer-revenue-prediction) Challenge dataset, 2)  present the offline `rmse`, the running time and the `public score` of each of them and 3) create a trivial linear ensemble obtaining a `1.6677` score in the public leaderboard.\n\n\n&nbsp;\n\n| Model        | Rounds | Train RMSE           | Validation RMSE | Train time | Public Score|\n| ------------- |------:|-----:|-----:| -----:| -----:|\n| `LightGBM`      | 5000| 1.505 | <span style='color:green'>1.60372 </span> | 7min 48s | <span style='color:green'>1.6717</span> |\n| `XGBoost`      | 2000| 1.568 | 1.64924 | <span style='color:red'>54min 54s </span> | 1.6946 |\n| `Catboost`      | 1000| 1.52184 | 1.61231  | <span style='color:green'>2min 24s</span> | 1.6722 |\n| `Ensemble`      | -- | --| -- | -- | <span style='color:green'>1.6677</span>|\nResult table from [Conclusions](#conclusions) section.\n\n\nThis kernel is strongly based on these previous work:\n* [LGBM (RF) starter [LB: 1.70]](https://www.kaggle.com/fabiendaniel/lgbm-rf-starter-lb-1-70) - Preprocessing is taken *as-is* from this awesome kernel by [FabienDaniel](https://www.kaggle.com/fabiendaniel/).\n* [LightGBM + XGBoost + Catboost](https://www.kaggle.com/samratp/lightgbm-xgboost-catboost) - LGBM, XGBoost and Catboost functions *taken and ligerely adapted* from this other awesome kernel by [Samrat P](https://www.kaggle.com/samratp).\n\n\n\n\nThe notebook has the following sections:\n\n1. [Preprocessing](#preprocessing)\n2. [Models](#models)\n  - 2.1. [LightGBM](#lightgbm)\n  - 2.2. [XGBoost](#xgboost)\n  - 2.3. [Catboost](#catboost)\n3. [Ensemble and submissions](#ensemble)\n4. [Conclusions](#conclusions)\n5. [References](#references)"},{"metadata":{"_uuid":"e016f4f6537fbb98a339afb78bd26126f21f60ab"},"cell_type":"markdown","source":"<a id='preprocessing'></a>\n## 1. Preprocessing\n\n&nbsp;\n\nThe preprocessing was taken as-is from [LGBM (RF) starter [LB: 1.70]](https://www.kaggle.com/fabiendaniel/lgbm-rf-starter-lb-1-70),  we just gathered it in one function. It consists of the following: drop columns with no information, label-encode the categorical columns with `pd.factorize()` and create few time-related columns. Refer to that already well known kernel for a detailed explanation of the preprocessing! \n\nHere, we will step over this section, hidding the code. The function `preprocess()` reads the original csvs (files pointed by variables `INPUT_TRAIN` and `INPUT_TEST`) and generates three processed ones: `TRAIN`, `TEST` and `Y`, where `Y` is separated from `TRAIN` for convenience. The resulting `TRAIN` and `TEST` files have 32 columns and `Y` has the logarithm of the `transactionRevenue` of the `TRAIN`."},{"metadata":{"trusted":true,"_uuid":"1b4d278ac4558b079ea09ecca7cd76ffed52dcb6"},"cell_type":"code","source":"INPUT_TRAIN = \"../input/train.csv\"\nINPUT_TEST = \"../input/test.csv\"\n\nTRAIN='train-processed.csv'\nTEST='test-processed.csv'\nY='y.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d97237aa7f659f238b5003d240e4264e01a1e63e","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import os\nimport gc\nimport json\nimport time\nfrom datetime import datetime\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Reference: https://www.kaggle.com/julian3833/1-quick-start-read-csv-and-flatten-json-fields\ndef load_df(csv_path=INPUT_TRAIN, nrows=None):\n    print(f\"Loading {csv_path}\")\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n\n\n# This function is just a packaged version of this kernel:\n# https://www.kaggle.com/fabiendaniel/lgbm-rf-starter-lb-1-70\ndef process_dfs(train_df, test_df):\n    print(\"Processing dfs...\")\n    print(\"Dropping repeated columns...\")\n    columns = [col for col in train_df.columns if train_df[col].nunique() > 1]\n    \n    train_df = train_df[columns]\n    test_df = test_df[columns]\n\n    trn_len = train_df.shape[0]\n    merged_df = pd.concat([train_df, test_df])\n\n    merged_df['diff_visitId_time'] = merged_df['visitId'] - merged_df['visitStartTime']\n    merged_df['diff_visitId_time'] = (merged_df['diff_visitId_time'] != 0).astype(int)\n    del merged_df['visitId']\n\n    del merged_df['sessionId']\n\n    print(\"Generating date columns...\")\n    format_str = '%Y%m%d' \n    merged_df['formated_date'] = merged_df['date'].apply(lambda x: datetime.strptime(str(x), format_str))\n    merged_df['WoY'] = merged_df['formated_date'].apply(lambda x: x.isocalendar()[1])\n    merged_df['month'] = merged_df['formated_date'].apply(lambda x:x.month)\n    merged_df['quarter_month'] = merged_df['formated_date'].apply(lambda x:x.day//8)\n    merged_df['weekday'] = merged_df['formated_date'].apply(lambda x:x.weekday())\n\n    del merged_df['date']\n    del merged_df['formated_date']\n\n    merged_df['formated_visitStartTime'] = merged_df['visitStartTime'].apply(\n        lambda x: time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(x)))\n    merged_df['formated_visitStartTime'] = pd.to_datetime(merged_df['formated_visitStartTime'])\n    merged_df['visit_hour'] = merged_df['formated_visitStartTime'].apply(lambda x: x.hour)\n\n    del merged_df['visitStartTime']\n    del merged_df['formated_visitStartTime']\n\n    print(\"Encoding columns with pd.factorize()\")\n    for col in merged_df.columns:\n        if col in ['fullVisitorId', 'month', 'quarter_month', 'weekday', 'visit_hour', 'WoY']: continue\n        if merged_df[col].dtypes == object or merged_df[col].dtypes == bool:\n            merged_df[col], indexer = pd.factorize(merged_df[col])\n\n    print(\"Splitting back...\")\n    train_df = merged_df[:trn_len]\n    test_df = merged_df[trn_len:]\n    return train_df, test_df\n\ndef preprocess():\n    train_df = load_df()\n    test_df = load_df(INPUT_TEST)\n\n    target = train_df['totals.transactionRevenue'].fillna(0).astype(float)\n    target = target.apply(lambda x: np.log1p(x))\n    del train_df['totals.transactionRevenue']\n\n    train_df, test_df = process_dfs(train_df, test_df)\n    train_df.to_csv(TRAIN, index=False)\n    test_df.to_csv(TEST, index=False)\n    target.to_csv(Y, index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"75ddedbd9c1053aae0b7718e65308eaa3ac8bd12"},"cell_type":"code","source":"%%time\npreprocess()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"002541874f4f3b14551e10349e14965b28dd39d7"},"cell_type":"markdown","source":"<a id='models'></a>\n## 2. Models\n\n&nbsp;\n\nBefore jumping into the models, we need some imports and auxiliary code. The most relevant things to note in this section are the imports of the three different libraries:\n\n```python\nimport lightgbm as lgb\nimport xgboost as xgb\nfrom catboost import CatBoostRegressor\n```\n\nBesides of that, we define the `rmse` evaluation metric based on sklearn's `mean_squared_error` and an auxiliary function for loading the preprocessed dataframes of the previous step,  `load_preprocessed_dfs()`\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import lightgbm as lgb\nimport xgboost as xgb\nfrom catboost import CatBoostRegressor\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\n\ndef rmse(y_true, y_pred):\n    return round(np.sqrt(mean_squared_error(y_true, y_pred)), 5)\n\ndef load_preprocessed_dfs(drop_full_visitor_id=True):\n    \"\"\"\n    Loads files `TRAIN`, `TEST` and `Y` generated by preprocess() into variables\n    \"\"\"\n    X_train = pd.read_csv(TRAIN, converters={'fullVisitorId': str})\n    X_test = pd.read_csv(TEST, converters={'fullVisitorId': str})\n    y_train = pd.read_csv(Y, names=['LogRevenue']).T.squeeze()\n    \n    # This is the only `object` column, we drop it for train and evaluation\n    if drop_full_visitor_id: \n        X_train = X_train.drop(['fullVisitorId'], axis=1)\n        X_test = X_test.drop(['fullVisitorId'], axis=1)\n    return X_train, y_train, X_test","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fe1b2ee398d196022a5ce9d6e589420f9572e7f"},"cell_type":"markdown","source":"In the cell below we load the dataframes `X`, `y` and `X_test` and generate a `train`, `validation` from the train data.  We drop `fullVisitorId`, so the dataframes have 31 columns each, while the `train` data has 768,000 rows, the `validation` 135,000 and the `test` set has 800,000."},{"metadata":{"trusted":true,"_uuid":"af7feb3929f4f6def4ff1c7cb7e108d8783f9717"},"cell_type":"code","source":"X, y, X_test = load_preprocessed_dfs()\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.15, random_state=1)\n\nprint(f\"Train shape: {X_train.shape}\")\nprint(f\"Validation shape: {X_val.shape}\")\nprint(f\"Test (submit) shape: {X_test.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3e668c886feff45f1bcd0f7f4e066060add9c2ee"},"cell_type":"markdown","source":"<a id='lightgbm'></a>\n### 2.1. LightGBM\n\n&nbsp;\n\nThe following code is an adaption of [LightGBM + XGBoost + Catboost](https://www.kaggle.com/samratp/lightgbm-xgboost-catboost) to this particular competition: it's pretty simple, in fact, so the best thing to do is to just look at `run_lgb()` below.  Here, some notes which may help to read the code:\n* lightgbm is installed with `pip install lightgbm` and typically imported as `lgb`.\n* `lgb` defines a `Dataset` object, which can be a tuple `(X, y)` or just a `X` (you'd typically have `train`, `validation` and `test` Datasets).\n* `create Dataset` $\\rightarrow$ `call train` is the most common workflow with `lgb`, but there is a [Scikit-learn API](https://lightgbm.readthedocs.io/en/latest/Python-API.html#scikit-learn-api) (which allows to do: `create model` $\\rightarrow$ `fit`  $\\rightarrow$ `predict`).\n* `lgb` has  **a lot** of [parameters](https://lightgbm.readthedocs.io/en/latest/Parameters.html).\n* `lgb` is *good out of the box* , *fast* and *it can handle categorical values*."},{"metadata":{"trusted":true,"_uuid":"528e023386d23343c50c32bfd47dbb0ca0c3b24c"},"cell_type":"code","source":"def run_lgb(X_train, y_train, X_val, y_val, X_test):\n    \n    params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\",\n        \"num_leaves\" : 40,\n        \"learning_rate\" : 0.005,\n        \"bagging_fraction\" : 0.6,\n        \"feature_fraction\" : 0.6,\n        \"bagging_frequency\" : 6,\n        \"bagging_seed\" : 42,\n        \"verbosity\" : -1,\n        \"seed\": 42\n    }\n    \n    lgb_train_data = lgb.Dataset(X_train, label=y_train)\n    lgb_val_data = lgb.Dataset(X_val, label=y_val)\n\n    model = lgb.train(params, lgb_train_data, \n                      num_boost_round=5000,\n                      valid_sets=[lgb_train_data, lgb_val_data],\n                      early_stopping_rounds=100,\n                      verbose_eval=500)\n\n    y_pred_train = model.predict(X_train, num_iteration=model.best_iteration)\n    y_pred_val = model.predict(X_val, num_iteration=model.best_iteration)\n    y_pred_submit = model.predict(X_test, num_iteration=model.best_iteration)\n\n    print(f\"LGBM: RMSE val: {rmse(y_val, y_pred_val)}  - RMSE train: {rmse(y_train, y_pred_train)}\")\n    return y_pred_submit, model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6ca05183788f2041fb482eefb74e9c68c57bcbc3"},"cell_type":"code","source":"%%time\n# Train LGBM and generate predictions\nlgb_preds, lgb_model = run_lgb(X_train, y_train, X_val, y_val, X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"90e95f42fc2c1fe0af62ff58b827280e6c495076"},"cell_type":"code","source":"print(\"LightGBM features importance...\")\ngain = lgb_model.feature_importance('gain')\nfeatureimp = pd.DataFrame({'feature': lgb_model.feature_name(), \n                   'split': lgb_model.feature_importance('split'), \n                   'gain': 100 * gain / gain.sum()}).sort_values('gain', ascending=False)\nprint(featureimp[:10])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"094c200651801ba5adcfd20bcf70f0e7ba2aaca0"},"cell_type":"markdown","source":"Further readings about `LightGBM`:\n* [What is LightGBM, How to implement it? How to fine tune the parameters?](https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc)\n* [Documentation](https://lightgbm.readthedocs.io/en/latest/).  In particular:\n   - [Python Quick Start](https://lightgbm.readthedocs.io/en/latest/Python-Intro.html)\n   - [Python API](https://lightgbm.readthedocs.io/en/latest/Python-API.html)\n   - [Parameters section](https://lightgbm.readthedocs.io/en/latest/Parameters.html)"},{"metadata":{"_uuid":"33421c8d90d08a3ad604776bac259834faa5a264"},"cell_type":"markdown","source":"<a id='xgboost'></a>\n### 2.2. XGBoost\n\n&nbsp;\n\n`XGBoost` stands for `Extreme Grandient Boosting`, which is a kind of `sklearn.ensemble`'s `GradientBoostingRegressor` on steroids.\n\n\n> XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable. \n\n\nIn concrete, the library is installed with `pip install xgboost`, its typically imported as `xgb` and,  once imported, it works similarly to `lgb`: we will create a custom dataset object (in this case, the `DMatrix`) and we will call the `train()` function of the module with some parameters and some `DMatrices`. "},{"metadata":{"trusted":true,"_uuid":"bf2dd9f3892b37350400d1ecafe1b27a21a7378d"},"cell_type":"code","source":"def run_xgb(X_train, y_train, X_val, y_val, X_test):\n    params = {'objective': 'reg:linear',\n              'eval_metric': 'rmse',\n              'eta': 0.001,\n              'max_depth': 10,\n              'subsample': 0.6,\n              'colsample_bytree': 0.6,\n              'alpha':0.001,\n              'random_state': 42,\n              'silent': True}\n\n    xgb_train_data = xgb.DMatrix(X_train, y_train)\n    xgb_val_data = xgb.DMatrix(X_val, y_val)\n    xgb_submit_data = xgb.DMatrix(X_test)\n\n    model = xgb.train(params, xgb_train_data, \n                      num_boost_round=2000, \n                      evals= [(xgb_train_data, 'train'), (xgb_val_data, 'valid')],\n                      early_stopping_rounds=100, \n                      verbose_eval=500\n                     )\n\n    y_pred_train = model.predict(xgb_train_data, ntree_limit=model.best_ntree_limit)\n    y_pred_val = model.predict(xgb_val_data, ntree_limit=model.best_ntree_limit)\n    y_pred_submit = model.predict(xgb_submit_data, ntree_limit=model.best_ntree_limit)\n\n    print(f\"XGB : RMSE val: {rmse(y_val, y_pred_val)}  - RMSE train: {rmse(y_train, y_pred_train)}\")\n    return y_pred_submit, model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2f3b8387d2911e9e4781ec33d5a07025516bab05"},"cell_type":"code","source":"%%time\nxgb_preds, xgb_model = run_xgb(X_train, y_train, X_val, y_val, X_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b8e8277e95f42a88f1e9f0516edc5c427acddff"},"cell_type":"markdown","source":"Further readings about `XGBoost`:\n* [Documentation](https://xgboost.readthedocs.io/en/latest/), in particular:\n - [Python Intro](https://xgboost.readthedocs.io/en/latest/python/python_intro.html)\n* [A Gentle Introduction to XGBoost for Applied Machine Learning](https://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/)"},{"metadata":{"_uuid":"af86dc8b38a181bc68004354e1d2a63519a2d64a"},"cell_type":"markdown","source":"<a id='catboost'></a>\n### 2.3. Catboost\n\n&nbsp;\n\n> `CatBoost` is a state-of-the-art open-source gradient boosting on decision trees library.\n\nCatBoost is a very good, fast and trendy boosting model, which handles categorical parameters (that's the `cat` in `catboost`!). It's installed with `pip install catboost` and proposes a scikit-learn like workflow with a Classifier or a Regressor and a `create` - `fit` -  `predict` cycle.\n\nNote that both `lgb` and `xgb` offer a \"Scikit-learn API\" option too: check out [here](https://lightgbm.readthedocs.io/en/latest/Python-API.html#scikit-learn-api) for `LightGBM` and [here](https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn) for `XGBoost`."},{"metadata":{"trusted":true,"_uuid":"1dfc73807e43fe48561096a23a104c32c0c65ff8"},"cell_type":"code","source":"def run_catboost(X_train, y_train, X_val, y_val, X_test):\n    model = CatBoostRegressor(iterations=1000,\n                             learning_rate=0.05,\n                             depth=10,\n                             eval_metric='RMSE',\n                             random_seed = 42,\n                             bagging_temperature = 0.2,\n                             od_type='Iter',\n                             metric_period = 50,\n                             od_wait=20)\n    model.fit(X_train, y_train,\n              eval_set=(X_val, y_val),\n              use_best_model=True,\n              verbose=True)\n    \n    y_pred_train = model.predict(X_train)\n    y_pred_val = model.predict(X_val)\n    y_pred_submit = model.predict(X_test)\n\n    print(f\"CatB: RMSE val: {rmse(y_val, y_pred_val)}  - RMSE train: {rmse(y_train, y_pred_train)}\")\n    return y_pred_submit, model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f97ac7203ae11aa6fbab191e98c90de926b076fb"},"cell_type":"code","source":"%%time\n# Train Catboost and generate predictions\ncat_preds, cat_model = run_catboost(X_train, y_train, X_val, y_val,  X_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af01b2e1c6988856d5859d9eaffbb90833b1a566"},"cell_type":"markdown","source":"Further readings on `Catboost`:\n* [Documentation](https://tech.yandex.com/catboost/doc/dg/concepts/about-docpage/), in particular:\n  - [Python Quickstart](https://tech.yandex.com/catboost/doc/dg/concepts/python-quickstart-docpage/)\n* [CatBoost: A machine learning library to handle categorical (CAT) data automatically](https://www.analyticsvidhya.com/blog/2017/08/catboost-automated-categorical-data/)"},{"metadata":{"_uuid":"58cc48a60dc4e819bfd4d8e6e68749fb9cb03c18"},"cell_type":"markdown","source":"<a id='ensemble'></a>\n## 3. Ensemble and submissions"},{"metadata":{"_uuid":"b7a2bea6c5384b2b644d365bb82e148fe1694e28"},"cell_type":"markdown","source":"In the previous section we trained baseline versions of `LightGBM`, `XGBoost` and `Catboost`.  \n\nIn this section we will create a trivial linear ensemble using hardcoded coefficients (70/30/0). We will create the submissions for the 3 baseline models and for the new ensemble as well."},{"metadata":{"trusted":true,"_uuid":"99c2de9381285001aabb7724a208a3f4cca984d6"},"cell_type":"code","source":"# Note: this is currently being reconstructed!\nensemble_preds_70_30_00 = 0.7 * lgb_preds + 0.3 * cat_preds + 0.0 * xgb_preds \nensemble_preds_70_25_05 = 0.7 * lgb_preds + 0.25 * cat_preds + 0.05 * xgb_preds ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c1420fd5405c2e1bc530d71c26187e10cb1bfa7"},"cell_type":"code","source":"def submit(predictions, filename='submit.csv'):\n    \"\"\"\n    Takes a (804684,) 1d-array of predictions and generates a submission file named filename\n    \"\"\"\n    _, _, X_submit = load_preprocessed_dfs(drop_full_visitor_id=False)\n    submission = X_submit[['fullVisitorId']].copy()\n    \n    submission.loc[:, 'PredictedLogRevenue'] = predictions\n    grouped_test = submission[['fullVisitorId', 'PredictedLogRevenue']].groupby('fullVisitorId').sum().reset_index()\n    grouped_test.to_csv(filename,index=False)\n\nsubmit(lgb_preds, \"submit-lgb.csv\")\n# Note: I disabled XGB to make the notebook run faster\nsubmit(xgb_preds, \"submit-xgb.csv\")\nsubmit(cat_preds, \"submit-cat.csv\")\nsubmit(ensemble_preds_70_30_00, \"submit-ensemble-70_30_00.csv\")\nsubmit(ensemble_preds_70_25_05, \"submit-ensemble-70_25_05.csv\")\n\nensemble_preds_70_30_00_pos = np.where(ensemble_preds_70_30_00 < 0, 0, ensemble_preds_70_30_00)\nsubmit(ensemble_preds_70_30_00_pos, \"submit-ensemble-70_30_00-positive.csv\")\n\nensemble_preds_70_25_05_pos = np.where(ensemble_preds_70_25_05 < 0, 0, ensemble_preds_70_25_05)\nsubmit(ensemble_preds_70_25_05_pos, \"submit-ensemble-70_25_05-positive.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1956a165c34458b69e9d07be1e5320be0d0bdbe"},"cell_type":"markdown","source":"<a id='conclusions'></a>\n## 4. Conclusions\n\n<!-- Before going to the conclusions, some disclaimers about the scope of this notebooks:\n* The `RMSE` we are calculating is `per visit`, while the online `RMSE` is `per visitor`. \n* The `lgb` submit contains negative results. This can be trivially fixed after the prediction or, I think, setting some parameters. \n* We didn't hold-out a `test` split of the dataset, so we don't have a test score. This was done in order to simplify the code and a more advanced version of it should consider that split.\n* `LightGBM` or `Catboost` can handle categorical values, but we are not doing it here.\n-->\n\nWith a baseline parameter configuration, the following results were obtained:\n\n| Model        | Rounds | Train RMSE           | Validation RMSE | Train time | Submit Score|\n| ------------- |------:|-----:|-----:| -----:| -----:|\n| `LightGBM`      | 5000| 1.505 | <span style='color:green'>1.60372 </span> | 7min 48s | 1.6717 |\n| `XGBoost`      | 2000| 1.568 | 1.64924 | <span style='color:red'>54min 54s </span> | 1.6946|\n| `Catboost`      | 1000| 1.52184 | 1.61231  | <span style='color:green'>2min 24s</span> | 1.6722|\n| `Ensemble`      | -- | --| -- | -- | 1.6677|\n\n&nbsp;\n\n`LightGBM` achieved the best results in train, validation and public score, while `Catboost` obtained the best train-time and an extremely competitive score as well.  `XGBoost`, on the other hand, took much more time (from minutes to hours), and didn't perform well in score either.\n\nOf course, we are not trying to prove any general point, this notebook is just a very narrow and educational comparison experiment of very baseline usages of these three trendy libraries!\n\nWell, that's it, those are some baseline implementations of the 3 boosting kings (?) applied to the GA challenge. \n\nComments/ideas/improvements are welcomed!"},{"metadata":{"_uuid":"86c5f75d899e69790b7fcf867a3daeb81cf1f8be"},"cell_type":"markdown","source":"<a id='references'></a>\n## 5. References\n\n#### Kernels\n* [LGBM (RF) starter [LB: 1.70]](https://www.kaggle.com/fabiendaniel/lgbm-rf-starter-lb-1-70)\n* [LightGBM + XGBoost + Catboost](https://www.kaggle.com/samratp/lightgbm-xgboost-catboost) \n\n#### Articles\n* [What is LightGBM, How to implement it? How to fine tune the parameters?](https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc)\n* [A Gentle Introduction to XGBoost for Applied Machine Learning](https://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/)\n* [CatBoost: A machine learning library to handle categorical (CAT) data automatically](https://www.analyticsvidhya.com/blog/2017/08/catboost-automated-categorical-data/)\n* [CatBoost vs. Light GBM vs. XGBoost](https://towardsdatascience.com/catboost-vs-light-gbm-vs-xgboost-5f93620723db?gi=4e06b8e37886)\n* [Machine Learning Challenge Winning Solutions](https://github.com/Microsoft/LightGBM/blob/master/examples/README.md#machine-learning-challenge-winning-solutions) - a list of challenges won by some version of LightGBM.\n\n#### Documentation\n*  [LightGBM Documentation: Python Quick Start](https://lightgbm.readthedocs.io/en/latest/Python-Intro.html)\n*  [LightGBM Documentation: Python API](https://lightgbm.readthedocs.io/en/latest/Python-API.html)\n* [LightGBM Documentation: Parameters section](https://lightgbm.readthedocs.io/en/latest/Parameters.html)\n* [XGBoost Documentation: Python Intro](https://xgboost.readthedocs.io/en/latest/python/python_intro.html)\n* [Catboost Documentation: Python Quickstart](https://tech.yandex.com/catboost/doc/dg/concepts/python-quickstart-docpage/)\n* [LightGBM Documentation: Scikit-learn API](https://lightgbm.readthedocs.io/en/latest/Python-API.html#scikit-learn-api)\n* [XGBoost Documentation: Scikit-learn API](https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn)\n"},{"metadata":{"trusted":true,"_uuid":"935f56f0df824c80df5c85760aa868a1832e8d53","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Delete the files created by catboost. \n# TODO: check if `allow_writing_files` params works or find another better way to do this\n!rm -rf ./catboost_info","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}