I wanted to check if I can come up with better default configurations and/or training procedures for the different boosted tree models (XGBoost, LightGBM, HistGradientBoosting (scikit-learn), and Catboost). Ideally, I would be tuning the hyperparameters individually for each dataset and task. However, since there were about 49 tasks, I decided to opt a more cost-effective solution.
\nMy best solution in this direction was an ensemble of 5 lightgbm models. The code can be found here.
\nn_estimators to 1000 and learning_rate to 0.05, but kept the other hyperparameters the same.I used a similar procedure for the other boosting models as well, with the xgboost ensemble the closest in performance.
\n| Model type | \nPrivate Score | \nPublic Score | \n
|---|---|---|
| LightGBM | \n0.0180 | \n0.0233 | \n
| XGBoost | \n0.0178 | \n0.0184 | \n
| HistGradientBoosting | \n0.0144 | \n0.0182 | \n
| Catboost | \n0.0013 | \n0.0020 | \n
I wanted to check if I can come up with better default configurations and/or training procedures for the different boosted tree models (XGBoost, LightGBM, HistGradientBoosting (scikit-learn), and Catboost). Ideally, I would be tuning the hyperparameters individually for each dataset and task. However, since there were about 49 tasks, I decided to opt a more cost-effective solution.
\nMy best solution in this direction was an ensemble of 5 lightgbm models. The code can be found here.
\nn_estimators to 1000 and learning_rate to 0.05, but kept the other hyperparameters the same.I used a similar procedure for the other boosting models as well, with the xgboost ensemble the closest in performance.
\n| Model type | \nPrivate Score | \nPublic Score | \n
|---|---|---|
| LightGBM | \n0.0180 | \n0.0233 | \n
| XGBoost | \n0.0178 | \n0.0184 | \n
| HistGradientBoosting | \n0.0144 | \n0.0182 | \n
| Catboost | \n0.0013 | \n0.0020 | \n
Thanks for the feedback. lgbm seems to benefit more from ensembling ? Maybe catboost is performing some ensembling internally. May be it is the way to go. Since lgbm calibrate 20 times faster than catboost, calibrating 5 of them seems to achieve better performance and would still be 4 time fasters.
", "rawMarkdown": "Thanks for the feedback. lgbm seems to benefit more from ensembling ? Maybe catboost is performing some ensembling internally. May be it is the way to go. Since lgbm calibrate 20 times faster than catboost, calibrating 5 of them seems to achieve better performance and would still be 4 time fasters.", "votes": null }, { "id": "2799020", "postDate": "05/07/2024 15:00:40", "content": "I am not so sure if we can conclude that LightGBM benefits more from ensembling than catboost because I haven't tuned the hyperparameters for either model. Also, the values for n_estimators and learning_rate I chose here are quite different from the default values for LightGBM (100 and 0.1 respectively), while Catboost uses the same or similar values (1000 and ~0.03 respectively).
However, as you mention, ensembling is more practical with LightGBM because it tends to have significantly faster training times than catboost in many scenarios.
", "rawMarkdown": "I am not so sure if we can conclude that LightGBM benefits more from ensembling than catboost because I haven't tuned the hyperparameters for either model. Also, the values for `n_estimators` and `learning_rate` I chose here are quite different from the default values for LightGBM (100 and 0.1 respectively), while Catboost uses the same or similar values (1000 and ~0.03 respectively).\n\nHowever, as you mention, ensembling is more practical with LightGBM because it tends to have significantly faster training times than catboost in many scenarios.", "votes": null }, { "id": "2803904", "postDate": "05/09/2024 17:39:03", "content": "Based on this competition I get the feeling that curse of dimensionality is much bigger during hyperparameter than I thought.
\nI have been pretty much at the end of the field even though I tuned for every dataset. The score got better the less parameters I tuned for (I capped at 200 trials per dataset). I am still doing late submissions to see what else can stay on default params.
@thomasmeiner I tried hyperparameter tuning for LightGBM through Optuna separately for each dataset. For each optuna run, I fixed the budget to a maximum of 50 trials and a maximum of 7.5 minutes runtime. Here's the link to the notebook: https://www.kaggle.com/code/syerramilli/benchtab-v1-lightgbm-optuna.
\nThe private and public scores with hyperparameter tuning are 0.0238 and 0.0250 respectively, which are a small improvement over those of my best submitted solution (0.0180 and 0.0233 respectively). Note that this is with a very small time budget and relatively small number of trials. The performance could further improve with a larger budget.
", "rawMarkdown": "thomasmeiner I tried hyperparameter tuning for LightGBM through Optuna separately for each dataset. For each optuna run, I fixed the budget to a maximum of 50 trials and a maximum of 7.5 minutes runtime. Here's the link to the notebook: https://www.kaggle.com/code/syerramilli/benchtab-v1-lightgbm-optuna. \n\nThe private and public scores with hyperparameter tuning are 0.0238 and 0.0250 respectively, which are a small improvement over those of my best submitted solution (0.0180 and 0.0233 respectively). Note that this is with a very small time budget and relatively small number of trials. The performance could further improve with a larger budget.", "votes": null } ], "comments": [ { "id": 2798707, "author_name": "lucasmorin", "author_url": "", "post_date": "05/07/2024 11:27:52", "content": "Thanks for the feedback. lgbm seems to benefit more from ensembling ? Maybe catboost is performing some ensembling internally. May be it is the way to go. Since lgbm calibrate 20 times faster than catboost, calibrating 5 of them seems to achieve better performance and would still be 4 time fasters.
", "votes": null, "replies": [ { "id": 2799020, "author_name": "syerramilli", "author_url": "", "post_date": "05/07/2024 15:00:40", "content": "I am not so sure if we can conclude that LightGBM benefits more from ensembling than catboost because I haven't tuned the hyperparameters for either model. Also, the values for n_estimators and learning_rate I chose here are quite different from the default values for LightGBM (100 and 0.1 respectively), while Catboost uses the same or similar values (1000 and ~0.03 respectively).
However, as you mention, ensembling is more practical with LightGBM because it tends to have significantly faster training times than catboost in many scenarios.
", "votes": null, "replies": [] } ] }, { "id": 2803904, "author_name": "thomasmeiner", "author_url": "", "post_date": "05/09/2024 17:39:03", "content": "Based on this competition I get the feeling that curse of dimensionality is much bigger during hyperparameter than I thought.
\nI have been pretty much at the end of the field even though I tuned for every dataset. The score got better the less parameters I tuned for (I capped at 200 trials per dataset). I am still doing late submissions to see what else can stay on default params.
@thomasmeiner I tried hyperparameter tuning for LightGBM through Optuna separately for each dataset. For each optuna run, I fixed the budget to a maximum of 50 trials and a maximum of 7.5 minutes runtime. Here's the link to the notebook: https://www.kaggle.com/code/syerramilli/benchtab-v1-lightgbm-optuna.
\nThe private and public scores with hyperparameter tuning are 0.0238 and 0.0250 respectively, which are a small improvement over those of my best submitted solution (0.0180 and 0.0233 respectively). Note that this is with a very small time budget and relatively small number of trials. The performance could further improve with a larger budget.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2797813": "I wanted to check if I can come up with better default configurations and/or training procedures for the different boosted tree models (XGBoost, LightGBM, HistGradientBoosting (scikit-learn), and Catboost). Ideally, I would be tuning the hyperparameters individually for each dataset and task. However, since there were about 49 tasks, I decided to opt a more cost-effective solution.\n\nMy best solution in this direction was an ensemble of 5 lightgbm models. The code can be found [here](https://www.kaggle.com/code/syerramilli/benchtab-v1-lightgbm-early-stopping). \n- Each model implemented early stopping on a validation set, partitioned from the training set. I adopted a 90-10 training-validation split.\n- To improve diversity in model predictions, I used a different training-validation split for each model in the ensemble.\n- I set the `n_estimators` to 1000 and `learning_rate` to 0.05, but kept the other hyperparameters the same.\n\nI used a similar procedure for the other boosting models as well, with the xgboost ensemble the closest in performance.\n\n| Model type | Private Score | Public Score |\n|----------------------|:-------------:|:------------:|\n| LightGBM | 0.0180 | 0.0233 |\n| XGBoost | 0.0178 | 0.0184 |\n| HistGradientBoosting | 0.0144 | 0.0182 |\n| Catboost | 0.0013 | 0.0020 |", "2798707": "Thanks for the feedback. lgbm seems to benefit more from ensembling ? Maybe catboost is performing some ensembling internally. May be it is the way to go. Since lgbm calibrate 20 times faster than catboost, calibrating 5 of them seems to achieve better performance and would still be 4 time fasters.", "2799020": "I am not so sure if we can conclude that LightGBM benefits more from ensembling than catboost because I haven't tuned the hyperparameters for either model. Also, the values for `n_estimators` and `learning_rate` I chose here are quite different from the default values for LightGBM (100 and 0.1 respectively), while Catboost uses the same or similar values (1000 and ~0.03 respectively).\n\nHowever, as you mention, ensembling is more practical with LightGBM because it tends to have significantly faster training times than catboost in many scenarios.", "2803904": "Based on this competition I get the feeling that curse of dimensionality is much bigger during hyperparameter than I thought.\nI have been pretty much at the end of the field even though I tuned for every dataset. The score got better the less parameters I tuned for (I capped at 200 trials per dataset). I am still doing late submissions to see what else can stay on default params.", "2806137": "thomasmeiner I tried hyperparameter tuning for LightGBM through Optuna separately for each dataset. For each optuna run, I fixed the budget to a maximum of 50 trials and a maximum of 7.5 minutes runtime. Here's the link to the notebook: https://www.kaggle.com/code/syerramilli/benchtab-v1-lightgbm-optuna. \n\nThe private and public scores with hyperparameter tuning are 0.0238 and 0.0250 respectively, which are a small improvement over those of my best submitted solution (0.0180 and 0.0233 respectively). Note that this is with a very small time budget and relatively small number of trials. The performance could further improve with a larger budget." }, "source": "meta" }