![\"\"](\"https://i.ibb.co/1T1Ms9y/trust-your-cv.png\")
'
\nAfter observing that the CV score and LB have a higher correlation with the new Purged TS CV scheme. I assume that we can start tuning hyperparameters and trust the CV. So I performed some experiments to find the optimal hyperparameters for my NN models and GBM models. However, CV score did increase, some LB increased a lot. Here are the experiments:
1st Place of Jane Street + Purged Time Series CV + Keras Tuner
Time Series Transformer + Purged Time Series CV + Keras Tuner
However, I still think that the optimization can overfit the training set. But how do we prevent that? after all, the market is changing all the time.
\nFor now, here are some more tips for better hyperparameters optimization:
\nBayesian optimization or grid search are good choices for hyperparameters optimization.
Start optimizing just a few parameters like num_leaves, feature_fraction and bagging_fraction with const learning_rate and use early stopping. When this is over, play with other parameters, over time you get some intuition about what is a good value for some of the parameters.
Search over only 1 fold and start with a small grip search. When it is done, you should have a good idea for good parameter sets. This is the time to move over the search to be using all all folds (again with earlystopping). Just make sure to use the ranges found on the prior grid-search.
At the top end of the leaderboard most hyperparameters optimization needs to be manual at that point. A hyper-parameter optimizer can give you an initial idea of the search space while you leave it running.
One last tip about stacking:
\nHow not to be an Idiot like me:
\nBefore we are done, let's talk about a common mistake. I overfitted a bit in the past when trying to get a stacking system to run during a kaggle comptition. so if you dont want to make the same mistake is myself, read along:
\nA common mistake is using early stopping per fold. While it is a good way to get a baseline models, you still overfit a bit each fold. This kind of OOF predictions doesn't work well if you plan to perform a stacking later [This single mistake cost me a full month]
\nThe best way to avoid overfiting is performing earlystop on all folds per round. Meaning, you find a \"good number\" of itterations you should to perform [For NN: epochs, for GBMs: n_estimators].
\nAfter you got the \"ideal\" number of rounds that perform well on the test set, you use the same number of rounds for ALL folds. This makes it possible to use stacking later on those OOF predictions.
\n[Note: This also works well when training a model to use all the dataset]
References:
\n'
\nAfter observing that the CV score and LB have a higher correlation with the new Purged TS CV scheme. I assume that we can start tuning hyperparameters and trust the CV. So I performed some experiments to find the optimal hyperparameters for my NN models and GBM models. However, CV score did increase, some LB increased a lot. Here are the experiments:
1st Place of Jane Street + Purged Time Series CV + Keras Tuner
Time Series Transformer + Purged Time Series CV + Keras Tuner
However, I still think that the optimization can overfit the training set. But how do we prevent that? after all, the market is changing all the time.
\nFor now, here are some more tips for better hyperparameters optimization:
\nBayesian optimization or grid search are good choices for hyperparameters optimization.
Start optimizing just a few parameters like num_leaves, feature_fraction and bagging_fraction with const learning_rate and use early stopping. When this is over, play with other parameters, over time you get some intuition about what is a good value for some of the parameters.
Search over only 1 fold and start with a small grip search. When it is done, you should have a good idea for good parameter sets. This is the time to move over the search to be using all all folds (again with earlystopping). Just make sure to use the ranges found on the prior grid-search.
At the top end of the leaderboard most hyperparameters optimization needs to be manual at that point. A hyper-parameter optimizer can give you an initial idea of the search space while you leave it running.
One last tip about stacking:
\nHow not to be an Idiot like me:
\nBefore we are done, let's talk about a common mistake. I overfitted a bit in the past when trying to get a stacking system to run during a kaggle comptition. so if you dont want to make the same mistake is myself, read along:
\nA common mistake is using early stopping per fold. While it is a good way to get a baseline models, you still overfit a bit each fold. This kind of OOF predictions doesn't work well if you plan to perform a stacking later [This single mistake cost me a full month]
\nThe best way to avoid overfiting is performing earlystop on all folds per round. Meaning, you find a \"good number\" of itterations you should to perform [For NN: epochs, for GBMs: n_estimators].
\nAfter you got the \"ideal\" number of rounds that perform well on the test set, you use the same number of rounds for ALL folds. This makes it possible to use stacking later on those OOF predictions.
\n[Note: This also works well when training a model to use all the dataset]
References:
\n", "rawMarkdown": "'\nAfter observing that the CV score and LB have a higher correlation with the new Purged TS CV scheme. I assume that we can start tuning hyperparameters and trust the CV. So I performed some experiments to find the optimal hyperparameters for my NN models and GBM models. However, CV score did increase, some LB increased a lot. Here are the experiments:\n\n\n- [LightGBM + Purged Time Series CV + Optuna](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna)\n\n- [Catboost GPU + Purged Time Series CV + Optuna](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna)\n\n- [XGBoost GPU + Purged Time Series CV + Optuna](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna)\n\n- [Keras NN + Purged Time Series CV + Keras Tuner](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner)\n\n- [1st Place of Jane Street + Purged Time Series CV + Keras Tuner](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner)\n\n- [Time Series Transformer + Purged Time Series CV + Keras Tuner](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline)\n\n\n\n____\n\nHowever, I still think that the optimization can overfit the training set. But how do we prevent that? after all, the market is changing all the time. \n\n____\n\nFor now, here are some more tips for better hyperparameters optimization:\n\n- Bayesian optimization or grid search are good choices for hyperparameters optimization. \n\n- Start optimizing just a few parameters like num_leaves, feature_fraction and bagging_fraction with const learning_rate and use early stopping. When this is over, play with other parameters, over time you get some intuition about what is a good value for some of the parameters.\n\n- Search over only 1 fold and start with a small grip search. When it is done, you should have a good idea for good parameter sets. This is the time to move over the search to be using all all folds (again with earlystopping). Just make sure to use the ranges found on the prior grid-search. \n\n- At the top end of the leaderboard most hyperparameters optimization needs to be manual at that point. A hyper-parameter optimizer can give you an initial idea of the search space while you leave it running.\n\n____\n\nOne last tip about stacking: \n\n**How not to be an Idiot like me:**\n\nBefore we are done, let's talk about a common mistake. I overfitted a bit in the past when trying to get a stacking system to run during a kaggle comptition. so if you dont want to make the same mistake is myself, read along:\n\nA common mistake is using early stopping per fold. While it is a good way to get a baseline models, you still overfit a bit each fold. This kind of OOF predictions doesn't work well if you plan to perform a stacking later [This single mistake cost me a full month]\n\nThe best way to avoid overfiting is performing earlystop **on all folds per round**. Meaning, you find a \"good number\" of itterations you should to perform [For NN: epochs, for GBMs: n_estimators].\nAfter you got the \"ideal\" number of rounds that perform well on the test set, you use the same number of rounds for ALL folds. This makes it possible to use stacking later on those OOF predictions. \n[Note: This also works well when training a model to use all the dataset]\n\n____\nReferences:\n- https://www.kaggle.com/c/lish-moa/discussion/195928\n- https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/89320", "votes": 16 } ], "comments": [], "raw_markdown_by_id": { "1582026": "'\nAfter observing that the CV score and LB have a higher correlation with the new Purged TS CV scheme. I assume that we can start tuning hyperparameters and trust the CV. So I performed some experiments to find the optimal hyperparameters for my NN models and GBM models. However, CV score did increase, some LB increased a lot. Here are the experiments:\n\n\n- [LightGBM + Purged Time Series CV + Optuna](https://www.kaggle.com/yamqwe/purged-time-series-cv-lightgbm-optuna)\n\n- [Catboost GPU + Purged Time Series CV + Optuna](https://www.kaggle.com/yamqwe/purged-time-series-cv-catboost-gpu-optuna)\n\n- [XGBoost GPU + Purged Time Series CV + Optuna](https://www.kaggle.com/yamqwe/purged-time-series-cv-xgboost-gpu-optuna)\n\n- [Keras NN + Purged Time Series CV + Keras Tuner](https://www.kaggle.com/yamqwe/bottleneck-encoder-mlp-keras-tuner)\n\n- [1st Place of Jane Street + Purged Time Series CV + Keras Tuner](https://www.kaggle.com/yamqwe/1st-place-of-jane-street-keras-tuner)\n\n- [Time Series Transformer + Purged Time Series CV + Keras Tuner](https://www.kaggle.com/yamqwe/sh-tcoins-transformer-baseline)\n\n\n\n____\n\nHowever, I still think that the optimization can overfit the training set. But how do we prevent that? after all, the market is changing all the time. \n\n____\n\nFor now, here are some more tips for better hyperparameters optimization:\n\n- Bayesian optimization or grid search are good choices for hyperparameters optimization. \n\n- Start optimizing just a few parameters like num_leaves, feature_fraction and bagging_fraction with const learning_rate and use early stopping. When this is over, play with other parameters, over time you get some intuition about what is a good value for some of the parameters.\n\n- Search over only 1 fold and start with a small grip search. When it is done, you should have a good idea for good parameter sets. This is the time to move over the search to be using all all folds (again with earlystopping). Just make sure to use the ranges found on the prior grid-search. \n\n- At the top end of the leaderboard most hyperparameters optimization needs to be manual at that point. A hyper-parameter optimizer can give you an initial idea of the search space while you leave it running.\n\n____\n\nOne last tip about stacking: \n\n**How not to be an Idiot like me:**\n\nBefore we are done, let's talk about a common mistake. I overfitted a bit in the past when trying to get a stacking system to run during a kaggle comptition. so if you dont want to make the same mistake is myself, read along:\n\nA common mistake is using early stopping per fold. While it is a good way to get a baseline models, you still overfit a bit each fold. This kind of OOF predictions doesn't work well if you plan to perform a stacking later [This single mistake cost me a full month]\n\nThe best way to avoid overfiting is performing earlystop **on all folds per round**. Meaning, you find a \"good number\" of itterations you should to perform [For NN: epochs, for GBMs: n_estimators].\nAfter you got the \"ideal\" number of rounds that perform well on the test set, you use the same number of rounds for ALL folds. This makes it possible to use stacking later on those OOF predictions. \n[Note: This also works well when training a model to use all the dataset]\n\n____\nReferences:\n- https://www.kaggle.com/c/lish-moa/discussion/195928\n- https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/89320" } }