I have realised that almost all the public notebooks do not perform cross validation (CV) of any kind. Based on my experience, writing a correct CV function itself is not as simple as other comps with basic metrics like MAE or RMSE
\nTherefore, I am releasing a starter notebook with a working version of CV. In this example, it uses a 5-fold StratifiedGroupKFold with a single XGB model, based on the architecture of most high scoring GBT public notebooks. It can be modified to suit Polars or NN use cases.
\nThe notebook is available here. Please upvote if you liked it or find it helpful! If you have any questions, feel free to write them in the comments
", "messages": [ { "id": "2502838", "postDate": "10/28/2023 14:10:20", "content": "I have realised that almost all the public notebooks do not perform cross validation (CV) of any kind. Based on my experience, writing a correct CV function itself is not as simple as other comps with basic metrics like MAE or RMSE
\nTherefore, I am releasing a starter notebook with a working version of CV. In this example, it uses a 5-fold StratifiedGroupKFold with a single XGB model, based on the architecture of most high scoring GBT public notebooks. It can be modified to suit Polars or NN use cases.
\nThe notebook is available here. Please upvote if you liked it or find it helpful! If you have any questions, feel free to write them in the comments
", "rawMarkdown": "I have realised that almost all the public notebooks do not perform cross validation (CV) of any kind. Based on my experience, writing a correct CV function itself is not as simple as other comps with basic metrics like MAE or RMSE\n\nTherefore, I am releasing a starter notebook with a working version of CV. **In this example, it uses a 5-fold StratifiedGroupKFold with a single XGB model, based on the architecture of most high scoring GBT public notebooks**. It can be modified to suit Polars or NN use cases.\n\nThe notebook is available [here](https://www.kaggle.com/code/yeoyunsianggeremie/how-to-perform-cv-gradient-boosting). Please upvote if you liked it or find it helpful! If you have any questions, feel free to write them in the comments", "votes": null }, { "id": "2502985", "postDate": "10/28/2023 16:07:35", "content": "Thanks, this is important for me to verify the local cv.
", "rawMarkdown": "Thanks, this is important for me to verify the local cv.", "votes": null }, { "id": "2503524", "postDate": "10/29/2023 07:09:45", "content": "@yeoyunsianggeremie : thanks for sharing it; loved to see StratifiedGroupKFold used in an instrumental way (this one is the optimal one for the datetime-aware feature sets).
\nJust FYI - there is another CV example (for lightgbm though) published recently , per https://www.kaggle.com/code/andreytikhomirov/feature-selection-cv
\nGood luck in the contest!
", "rawMarkdown": "yeoyunsianggeremie : thanks for sharing it; loved to see StratifiedGroupKFold used in an instrumental way (this one is the optimal one for the datetime-aware feature sets).\n\nJust FYI - there is another CV example (for lightgbm though) published recently , per https://www.kaggle.com/code/andreytikhomirov/feature-selection-cv \n\nGood luck in the contest!", "votes": null }, { "id": "2503551", "postDate": "10/29/2023 07:45:01", "content": "I was wondering if you could help me understand the benefits of using StratifiedGroupKFold over just GroupKFold in this context?
\nI'd love to know if there's any specific stratification criteria we're aiming for. Thanks!
", "rawMarkdown": "I was wondering if you could help me understand the benefits of using StratifiedGroupKFold over just GroupKFold in this context? \n\nI'd love to know if there's any specific stratification criteria we're aiming for. Thanks!", "votes": null }, { "id": "2503561", "postDate": "10/29/2023 07:52:43", "content": "@illidan7 : as mentioned in Scikit-Learn Documentation,
\n\nThe difference between GroupKFold and StratifiedGroupKFold is that the former attempts to create balanced folds such that the number of distinct groups is approximately the same in each fold, whereas StratifiedGroupKFold attempts to create folds which preserve the percentage of samples for each class as much as possible given the constraint of non-overlapping groups between splits.\n\n
You can delve more into the matter by reviewing the nice blog posts below
\nI hope it is helpful.
", "rawMarkdown": "illidan7 : as mentioned in [Scikit-Learn Documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html),\n\nThe difference between GroupKFold and StratifiedGroupKFold is that the former attempts to create balanced folds such that the number of distinct groups is approximately the same in each fold, whereas StratifiedGroupKFold attempts to create folds which preserve the percentage of samples for each class as much as possible given the constraint of non-overlapping groups between splits.\n\n\nYou can delve more into the matter by reviewing the nice blog posts below\n- https://towardsdatascience.com/stratified-k-fold-cross-validation-on-grouped-datasets-b3bca8f0f53e\n- https://medium.com/the-techlife/5-different-ways-to-cross-validate-your-data-376a79b7f205\n- etc. (acutally, there is plenty of such posts available in Internet)\n\nI hope it is helpful.", "votes": null }, { "id": "2503584", "postDate": "10/29/2023 08:18:47", "content": "
The other CV example uses the accuracy metric which is very different from the competition metric
", "rawMarkdown": "The other CV example uses the accuracy metric which is very different from the competition metric", "votes": null }, { "id": "2504542", "postDate": "10/30/2023 01:21:06", "content": "Thanks for sharing, I took your work and made this notebook to optimize the hyper-parameters of the models
", "rawMarkdown": "Thanks for sharing, I took your work and made this [notebook](https://www.kaggle.com/code/renatoreggiani/zzzs-how-to-optimize-models) to optimize the hyper-parameters of the models", "votes": null }, { "id": "2505087", "postDate": "10/30/2023 11:05:45", "content": "Thank you.
", "rawMarkdown": "Thank you.", "votes": null } ], "comments": [ { "id": 2502985, "author_name": "gentlezdh", "author_url": "", "post_date": "10/28/2023 16:07:35", "content": "Thanks, this is important for me to verify the local cv.
", "votes": null, "replies": [] }, { "id": 2503524, "author_name": "gvyshnya", "author_url": "", "post_date": "10/29/2023 07:09:45", "content": "@yeoyunsianggeremie : thanks for sharing it; loved to see StratifiedGroupKFold used in an instrumental way (this one is the optimal one for the datetime-aware feature sets).
\nJust FYI - there is another CV example (for lightgbm though) published recently , per https://www.kaggle.com/code/andreytikhomirov/feature-selection-cv
\nGood luck in the contest!
", "votes": null, "replies": [ { "id": 2503584, "author_name": "yeoyunsianggeremie", "author_url": "", "post_date": "10/29/2023 08:18:47", "content": "The other CV example uses the accuracy metric which is very different from the competition metric
", "votes": null, "replies": [] } ] }, { "id": 2503551, "author_name": "illidan7", "author_url": "", "post_date": "10/29/2023 07:45:01", "content": "I was wondering if you could help me understand the benefits of using StratifiedGroupKFold over just GroupKFold in this context?
\nI'd love to know if there's any specific stratification criteria we're aiming for. Thanks!
", "votes": null, "replies": [ { "id": 2503561, "author_name": "gvyshnya", "author_url": "", "post_date": "10/29/2023 07:52:43", "content": "@illidan7 : as mentioned in Scikit-Learn Documentation,
\n\nThe difference between GroupKFold and StratifiedGroupKFold is that the former attempts to create balanced folds such that the number of distinct groups is approximately the same in each fold, whereas StratifiedGroupKFold attempts to create folds which preserve the percentage of samples for each class as much as possible given the constraint of non-overlapping groups between splits.\n\n
You can delve more into the matter by reviewing the nice blog posts below
\nI hope it is helpful.
", "votes": null, "replies": [] } ] }, { "id": 2504542, "author_name": "renatoreggiani", "author_url": "", "post_date": "10/30/2023 01:21:06", "content": "Thanks for sharing, I took your work and made this notebook to optimize the hyper-parameters of the models
", "votes": null, "replies": [] }, { "id": 2505087, "author_name": "dongyk", "author_url": "", "post_date": "10/30/2023 11:05:45", "content": "Thank you.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2502838": "I have realised that almost all the public notebooks do not perform cross validation (CV) of any kind. Based on my experience, writing a correct CV function itself is not as simple as other comps with basic metrics like MAE or RMSE\n\nTherefore, I am releasing a starter notebook with a working version of CV. **In this example, it uses a 5-fold StratifiedGroupKFold with a single XGB model, based on the architecture of most high scoring GBT public notebooks**. It can be modified to suit Polars or NN use cases.\n\nThe notebook is available [here](https://www.kaggle.com/code/yeoyunsianggeremie/how-to-perform-cv-gradient-boosting). Please upvote if you liked it or find it helpful! If you have any questions, feel free to write them in the comments", "2502985": "Thanks, this is important for me to verify the local cv.", "2503524": "yeoyunsianggeremie : thanks for sharing it; loved to see StratifiedGroupKFold used in an instrumental way (this one is the optimal one for the datetime-aware feature sets).\n\nJust FYI - there is another CV example (for lightgbm though) published recently , per https://www.kaggle.com/code/andreytikhomirov/feature-selection-cv \n\nGood luck in the contest!", "2503551": "I was wondering if you could help me understand the benefits of using StratifiedGroupKFold over just GroupKFold in this context? \n\nI'd love to know if there's any specific stratification criteria we're aiming for. Thanks!", "2503561": "illidan7 : as mentioned in [Scikit-Learn Documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html),\n\nThe difference between GroupKFold and StratifiedGroupKFold is that the former attempts to create balanced folds such that the number of distinct groups is approximately the same in each fold, whereas StratifiedGroupKFold attempts to create folds which preserve the percentage of samples for each class as much as possible given the constraint of non-overlapping groups between splits.\n\n\nYou can delve more into the matter by reviewing the nice blog posts below\n- https://towardsdatascience.com/stratified-k-fold-cross-validation-on-grouped-datasets-b3bca8f0f53e\n- https://medium.com/the-techlife/5-different-ways-to-cross-validate-your-data-376a79b7f205\n- etc. (acutally, there is plenty of such posts available in Internet)\n\nI hope it is helpful.", "2503584": "The other CV example uses the accuracy metric which is very different from the competition metric", "2504542": "Thanks for sharing, I took your work and made this [notebook](https://www.kaggle.com/code/renatoreggiani/zzzs-how-to-optimize-models) to optimize the hyper-parameters of the models", "2505087": "Thank you." }, "source": "meta" }