Now I'm no expert but I always thought that a Cross Validation Strategy was something you ran locally to test new features or processioning strategies independently of the LB. People in this competition seem to be doing this well, but then when it comes to submitting, they use the exact same models they trained during cross validation to do the submitting.
\n\nThere's nothing wrong with this of course, and if it gets you higher on the LB then more power to you, but I think there's a big benefit to having a submission strategy separate to your CV strategy.
\n\nFor instance, a super common CV strategy is to train a model m on 80% of the data and then score m based on the remaining 20%. You then use this score to try out different things. For instance you try adding different feature sets to your dataset and see which feature set trains the m that scores the highest.
\n\nNow that you've used this CV strategy to find a good feature set though, you wouldn't use m to make your submission. We know that models perform better if you feed them more data, so every time you want to submit you'd throw out m and train a new model M on 100% of the data and the best feature set you found though CV. You then submit M's predictions.
\n\nPoint is: you don't need to use your CV strategy to make submissions, and in fact, having a separate submission strategy gives you some nice freedoms. For instance often my CV strategy involves a very high LR and low number of iterations, or only uses a small fraction of the data to make it faster. If I made submissions with the same models though, I'd do really badly.
", "messages": [ { "id": "666608", "postDate": "11/06/2019 09:29:09", "content": "Now I'm no expert but I always thought that a Cross Validation Strategy was something you ran locally to test new features or processioning strategies independently of the LB. People in this competition seem to be doing this well, but then when it comes to submitting, they use the exact same models they trained during cross validation to do the submitting.
\n\nThere's nothing wrong with this of course, and if it gets you higher on the LB then more power to you, but I think there's a big benefit to having a submission strategy separate to your CV strategy.
\n\nFor instance, a super common CV strategy is to train a model m on 80% of the data and then score m based on the remaining 20%. You then use this score to try out different things. For instance you try adding different feature sets to your dataset and see which feature set trains the m that scores the highest.
\n\nNow that you've used this CV strategy to find a good feature set though, you wouldn't use m to make your submission. We know that models perform better if you feed them more data, so every time you want to submit you'd throw out m and train a new model M on 100% of the data and the best feature set you found though CV. You then submit M's predictions.
\n\nPoint is: you don't need to use your CV strategy to make submissions, and in fact, having a separate submission strategy gives you some nice freedoms. For instance often my CV strategy involves a very high LR and low number of iterations, or only uses a small fraction of the data to make it faster. If I made submissions with the same models though, I'd do really badly.
", "rawMarkdown": "Now I'm no expert but I always thought that a *Cross Validation Strategy* was something you ran locally to test new features or processioning strategies independently of the LB. People in this competition seem to be doing this well, but then when it comes to submitting, they use the *exact same models* they trained during cross validation to do the submitting. \n\nThere's nothing wrong with this of course, and if it gets you higher on the LB then more power to you, but I think there's a big benefit to having a submission strategy separate to your CV strategy. \n\nFor instance, a super common CV strategy is to train a model **m** on 80% of the data and then score **m** based on the remaining 20%. You then use this score to try out different things. For instance you try adding different feature sets to your dataset and see which feature set trains the **m** that scores the highest.\n\nNow that you've used this CV strategy to find a good feature set though, you wouldn't use **m** to make your *submission*. We know that models perform better if you feed them more data, so every time you want to submit you'd throw out **m** and train a new model **M** on 100% of the data and the best feature set you found though CV. You then submit **M**'s predictions. \n\nPoint is: you don't need to use your CV strategy to make submissions, and in fact, having a separate submission strategy gives you some nice freedoms. For instance often my CV strategy involves a very high LR and low number of iterations, or only uses a small fraction of the data to make it faster. If I made submissions with the same models though, I'd do really badly.", "votes": null }, { "id": "666626", "postDate": "11/06/2019 09:43:20", "content": "> train a new model M on 100% of the data
\n\nIf you do this, how can you assure that the new model is not overfitting the training data?
", "rawMarkdown": "> train a new model M on 100% of the data\n\nIf you do this, how can you assure that the new model is not overfitting the training data?", "votes": null }, { "id": "666672", "postDate": "11/06/2019 11:08:05", "content": "Right, well you cant.
\n\nBut the assumption is that if you trained a model on 80% of the training data (or whatever CV you used) and that didn't overfit then doing exactly the same thing again just with the whole training dataset also won't overfit. In fact (and in my experience), generally the more data you train your model on the less likely you are to overfit.
\n\nAlso, CV can never tell you if you are over-fitting your training set because CV only scores you based on your training set. If you get a good CV score, it means you aren't over-fitting some subset of the training set. If the training set is super different from the overall distribution of the data, then you can have an excellent CV score and great CV strategy but still horribly overfit the training set.
\n\nIn kaggle the only way you can know if you are over-fitting the training set is by making submissions. In real life you have to look for new observations.
", "rawMarkdown": "Right, well you cant. \n\nBut the assumption is that if you trained a model on 80% of the training data (or whatever CV you used) and *that* didn't overfit then doing **exactly the same thing again** just with the *whole* training dataset also won't overfit. In fact (and in my experience), generally the more data you train your model on the less likely you are to overfit. \n\nAlso, CV can never tell you if you are over-fitting your training set because CV only scores you *based on your training set*. If you get a good CV score, it means you aren't over-fitting some **subset** of the training set. If the training set is super different from the overall distribution of the data, then you can have an excellent CV score and great CV strategy but still horribly overfit the training set. \n\nIn kaggle the only way you can know if you are over-fitting the training set is by making submissions. In real life you have to look for new observations.", "votes": null }, { "id": "667081", "postDate": "11/06/2019 19:47:26", "content": "shouldn't k-fold take care of this issue?
", "rawMarkdown": "shouldn't k-fold take care of this issue?", "votes": null }, { "id": "667142", "postDate": "11/06/2019 21:29:46", "content": "well here my knowledge starts to run dry, but generally I'm pretty sure a single model trained on 100% of the data will behave differently to the average of 3 models, each trained on a different 66% of the data, and different could easily mean better.
\n\nI think someone made a kernel demonstrating something similar...
", "rawMarkdown": "well here my knowledge starts to run dry, but generally I'm pretty sure a single model trained on 100% of the data will behave differently to the average of 3 models, each trained on a different 66% of the data, and *different* could easily mean *better*.\n\nI think someone made a kernel demonstrating something similar...", "votes": null }, { "id": "667163", "postDate": "11/06/2019 22:17:16", "content": "Consider algorithms such as random forests and GBMs which use partial data (subsample) at each iteration. Subsampling is similar to your case of \"the average of 3 models, each trained on a different 66% of the data\". Subsampling makes different models, but I would consider that between subsampling and not subsampling, both would have taken into account the full set of data.
", "rawMarkdown": "Consider algorithms such as random forests and GBMs which use partial data (subsample) at each iteration. Subsampling is similar to your case of \"the average of 3 models, each trained on a different 66% of the data\". Subsampling makes different models, but I would consider that between subsampling and not subsampling, both would have taken into account the full set of data.", "votes": null }, { "id": "671016", "postDate": "11/12/2019 06:37:39", "content": "Exactly. This is a common strategy as mentioned by @cdeotte here (in the context of IEEE Fraud Detection) :
\n\n```\nMany people don't realize that your validation (model tuning) and inference (predicting test) procedures can be different.
\n\nFor example, you can split train dataset by time into early 75% and late 25%. Tune your features and hyperparameters with this validation scheme. Also use early stopping to determine a good number of iterations. Then predict test dataset with 4-Kfold using the same features, hyperparameters, and number of iterations with early stopping disabled.
\n\nThis works well because your features, hyperparameters, and early stopping were determined with 0.75*len(train) rows of data. And when you predict with 4-KFold, each model will also use 0.75*len(train) rows of data. Also your features, hyperparameters, and early stopping were determined with a scenario that somewhat mimics the relationship between train and test.\n```
", "rawMarkdown": "Exactly. This is a common strategy as mentioned by @cdeotte [here ](https://www.kaggle.com/c/ieee-fraud-detection/discussion/108571#625965)(in the context of IEEE Fraud Detection) : \n\n```\nMany people don't realize that your validation (model tuning) and inference (predicting test) procedures can be different.\n\nFor example, you can split train dataset by time into early 75% and late 25%. Tune your features and hyperparameters with this validation scheme. Also use early stopping to determine a good number of iterations. Then predict test dataset with 4-Kfold using the same features, hyperparameters, and number of iterations with early stopping disabled.\n\nThis works well because your features, hyperparameters, and early stopping were determined with 0.75*len(train) rows of data. And when you predict with 4-KFold, each model will also use 0.75*len(train) rows of data. Also your features, hyperparameters, and early stopping were determined with a scenario that somewhat mimics the relationship between train and test.\n```", "votes": null }, { "id": "671543", "postDate": "11/12/2019 21:18:25", "content": "Did you find a way to insure the overall distribution between train and val is similar ? This is very hard with only 1y of data.
", "rawMarkdown": "Did you find a way to insure the overall distribution between train and val is similar ? This is very hard with only 1y of data.", "votes": null } ], "comments": [ { "id": 666626, "author_name": "elysii", "author_url": "", "post_date": "11/06/2019 09:43:20", "content": "> train a new model M on 100% of the data
\n\nIf you do this, how can you assure that the new model is not overfitting the training data?
", "votes": null, "replies": [ { "id": 666672, "author_name": "lewington", "author_url": "", "post_date": "11/06/2019 11:08:05", "content": "Right, well you cant.
\n\nBut the assumption is that if you trained a model on 80% of the training data (or whatever CV you used) and that didn't overfit then doing exactly the same thing again just with the whole training dataset also won't overfit. In fact (and in my experience), generally the more data you train your model on the less likely you are to overfit.
\n\nAlso, CV can never tell you if you are over-fitting your training set because CV only scores you based on your training set. If you get a good CV score, it means you aren't over-fitting some subset of the training set. If the training set is super different from the overall distribution of the data, then you can have an excellent CV score and great CV strategy but still horribly overfit the training set.
\n\nIn kaggle the only way you can know if you are over-fitting the training set is by making submissions. In real life you have to look for new observations.
", "votes": null, "replies": [] }, { "id": 671543, "author_name": "areveillon", "author_url": "", "post_date": "11/12/2019 21:18:25", "content": "Did you find a way to insure the overall distribution between train and val is similar ? This is very hard with only 1y of data.
", "votes": null, "replies": [] } ] }, { "id": 667081, "author_name": "poedator", "author_url": "", "post_date": "11/06/2019 19:47:26", "content": "shouldn't k-fold take care of this issue?
", "votes": null, "replies": [ { "id": 667142, "author_name": "lewington", "author_url": "", "post_date": "11/06/2019 21:29:46", "content": "well here my knowledge starts to run dry, but generally I'm pretty sure a single model trained on 100% of the data will behave differently to the average of 3 models, each trained on a different 66% of the data, and different could easily mean better.
\n\nI think someone made a kernel demonstrating something similar...
", "votes": null, "replies": [] }, { "id": 667163, "author_name": "gideonteo", "author_url": "", "post_date": "11/06/2019 22:17:16", "content": "Consider algorithms such as random forests and GBMs which use partial data (subsample) at each iteration. Subsampling is similar to your case of \"the average of 3 models, each trained on a different 66% of the data\". Subsampling makes different models, but I would consider that between subsampling and not subsampling, both would have taken into account the full set of data.
", "votes": null, "replies": [] } ] }, { "id": 671016, "author_name": "arnabbiswas1", "author_url": "", "post_date": "11/12/2019 06:37:39", "content": "Exactly. This is a common strategy as mentioned by @cdeotte here (in the context of IEEE Fraud Detection) :
\n\n```\nMany people don't realize that your validation (model tuning) and inference (predicting test) procedures can be different.
\n\nFor example, you can split train dataset by time into early 75% and late 25%. Tune your features and hyperparameters with this validation scheme. Also use early stopping to determine a good number of iterations. Then predict test dataset with 4-Kfold using the same features, hyperparameters, and number of iterations with early stopping disabled.
\n\nThis works well because your features, hyperparameters, and early stopping were determined with 0.75*len(train) rows of data. And when you predict with 4-KFold, each model will also use 0.75*len(train) rows of data. Also your features, hyperparameters, and early stopping were determined with a scenario that somewhat mimics the relationship between train and test.\n```
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "666608": "Now I'm no expert but I always thought that a *Cross Validation Strategy* was something you ran locally to test new features or processioning strategies independently of the LB. People in this competition seem to be doing this well, but then when it comes to submitting, they use the *exact same models* they trained during cross validation to do the submitting. \n\nThere's nothing wrong with this of course, and if it gets you higher on the LB then more power to you, but I think there's a big benefit to having a submission strategy separate to your CV strategy. \n\nFor instance, a super common CV strategy is to train a model **m** on 80% of the data and then score **m** based on the remaining 20%. You then use this score to try out different things. For instance you try adding different feature sets to your dataset and see which feature set trains the **m** that scores the highest.\n\nNow that you've used this CV strategy to find a good feature set though, you wouldn't use **m** to make your *submission*. We know that models perform better if you feed them more data, so every time you want to submit you'd throw out **m** and train a new model **M** on 100% of the data and the best feature set you found though CV. You then submit **M**'s predictions. \n\nPoint is: you don't need to use your CV strategy to make submissions, and in fact, having a separate submission strategy gives you some nice freedoms. For instance often my CV strategy involves a very high LR and low number of iterations, or only uses a small fraction of the data to make it faster. If I made submissions with the same models though, I'd do really badly.", "666626": "> train a new model M on 100% of the data\n\nIf you do this, how can you assure that the new model is not overfitting the training data?", "666672": "Right, well you cant. \n\nBut the assumption is that if you trained a model on 80% of the training data (or whatever CV you used) and *that* didn't overfit then doing **exactly the same thing again** just with the *whole* training dataset also won't overfit. In fact (and in my experience), generally the more data you train your model on the less likely you are to overfit. \n\nAlso, CV can never tell you if you are over-fitting your training set because CV only scores you *based on your training set*. If you get a good CV score, it means you aren't over-fitting some **subset** of the training set. If the training set is super different from the overall distribution of the data, then you can have an excellent CV score and great CV strategy but still horribly overfit the training set. \n\nIn kaggle the only way you can know if you are over-fitting the training set is by making submissions. In real life you have to look for new observations.", "667081": "shouldn't k-fold take care of this issue?", "667142": "well here my knowledge starts to run dry, but generally I'm pretty sure a single model trained on 100% of the data will behave differently to the average of 3 models, each trained on a different 66% of the data, and *different* could easily mean *better*.\n\nI think someone made a kernel demonstrating something similar...", "667163": "Consider algorithms such as random forests and GBMs which use partial data (subsample) at each iteration. Subsampling is similar to your case of \"the average of 3 models, each trained on a different 66% of the data\". Subsampling makes different models, but I would consider that between subsampling and not subsampling, both would have taken into account the full set of data.", "671016": "Exactly. This is a common strategy as mentioned by @cdeotte [here ](https://www.kaggle.com/c/ieee-fraud-detection/discussion/108571#625965)(in the context of IEEE Fraud Detection) : \n\n```\nMany people don't realize that your validation (model tuning) and inference (predicting test) procedures can be different.\n\nFor example, you can split train dataset by time into early 75% and late 25%. Tune your features and hyperparameters with this validation scheme. Also use early stopping to determine a good number of iterations. Then predict test dataset with 4-Kfold using the same features, hyperparameters, and number of iterations with early stopping disabled.\n\nThis works well because your features, hyperparameters, and early stopping were determined with 0.75*len(train) rows of data. And when you predict with 4-KFold, each model will also use 0.75*len(train) rows of data. Also your features, hyperparameters, and early stopping were determined with a scenario that somewhat mimics the relationship between train and test.\n```", "671543": "Did you find a way to insure the overall distribution between train and val is similar ? This is very hard with only 1y of data." }, "source": "meta" }