I realize that there was leakeage training with 5 StratifiedKFold (builiding_id), can't use this schema for validation. I train with the first 8 month and predict the last 4 month and then for the test use all the training data with previous early stopping but the score was not good. Any idea on how to make a good cross validations strategy in this competition?\nCheers!
", "messages": [ { "id": "656708", "postDate": "10/24/2019 14:51:50", "content": "I realize that there was leakeage training with 5 StratifiedKFold (builiding_id), can't use this schema for validation. I train with the first 8 month and predict the last 4 month and then for the test use all the training data with previous early stopping but the score was not good. Any idea on how to make a good cross validations strategy in this competition?\nCheers!
", "rawMarkdown": "I realize that there was leakeage training with 5 StratifiedKFold (builiding_id), can't use this schema for validation. I train with the first 8 month and predict the last 4 month and then for the test use all the training data with previous early stopping but the score was not good. Any idea on how to make a good cross validations strategy in this competition?\nCheers!", "votes": null }, { "id": "656937", "postDate": "10/24/2019 19:19:35", "content": "Hi @ragnar123 , I tried TimeSeriesSplit CV but I hadn't reliable results. I think that is mainly due to the fact that building ids ratio was not the same...
\n\nAs you said, StratifiedKFold easily gets temporal leakage. I think we should implement something in between, probably we could lose some data to get a stratified validation divided by time. Did someone try something like that?
", "rawMarkdown": "Hi @ragnar123 , I tried TimeSeriesSplit CV but I hadn't reliable results. I think that is mainly due to the fact that building ids ratio was not the same...\n\nAs you said, StratifiedKFold easily gets temporal leakage. I think we should implement something in between, probably we could lose some data to get a stratified validation divided by time. Did someone try something like that?", "votes": null }, { "id": "657013", "postDate": "10/24/2019 20:56:28", "content": "Maybee try a groupkfold with 4 folds by quarter..... i will try other ideas and if i have a good result i will share it here
\n\nRecently i made some feature selection and feature engineering with the following schema:
\n\nTrain with the first 8 month and then evaluate with the last 4 month
\n\nThen i pick those features and for predicting the test i use 5 StratifiedKFold (builiding_id) but i manually put 1000 boosting rounds to avoid offerfitting.
\n\nI want to check if the avaluation strategy align with the test, in other words if my eval with the 8/4 improve, also my test improve training with 5 StratifiedKFold with the features i selected in the 8/4 schema.
", "rawMarkdown": "Maybee try a groupkfold with 4 folds by quarter..... i will try other ideas and if i have a good result i will share it here\n\nRecently i made some feature selection and feature engineering with the following schema:\n\nTrain with the first 8 month and then evaluate with the last 4 month\n\nThen i pick those features and for predicting the test i use 5 StratifiedKFold (builiding_id) but i manually put 1000 boosting rounds to avoid offerfitting. \n\nI want to check if the avaluation strategy align with the test, in other words if my eval with the 8/4 improve, also my test improve training with 5 StratifiedKFold with the features i selected in the 8/4 schema.", "votes": null }, { "id": "657021", "postDate": "10/24/2019 21:07:43", "content": "The fact is using just a time-based split doesn't guarantee you having all building ids in both training and validation sets. Then, I think GroupKFold on quarter doesn't prevent time leakage too cause. Good link to visualize CV methods.
\n\nI'm not sure about predicting using a different method from training. Isn't that like forcing different train and test distributions?
", "rawMarkdown": "The fact is using just a time-based split doesn't guarantee you having all building ids in both training and validation sets. Then, I think GroupKFold on quarter doesn't prevent time leakage too cause. [Good link to visualize CV methods](https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html#sphx-glr-auto-examples-model-selection-plot-cv-indices-py).\n\nI'm not sure about predicting using a different method from training. Isn't that like forcing different train and test distributions?", "votes": null }, { "id": "657730", "postDate": "10/25/2019 11:51:47", "content": "I tried my strategy but it failed. In my validation schema month was a very important feature but for predicting the test it overfitts big time. Maybee the the validation schema that we are looking for is just using the leader board because the test and train have very different distributions. In other words no local validation strategy to do feature engineering and feature selection, just use the lb.
", "rawMarkdown": "I tried my strategy but it failed. In my validation schema month was a very important feature but for predicting the test it overfitts big time. Maybee the the validation schema that we are looking for is just using the leader board because the test and train have very different distributions. In other words no local validation strategy to do feature engineering and feature selection, just use the lb.", "votes": null }, { "id": "661435", "postDate": "10/30/2019 09:41:55", "content": "To avoid data leakage in timeseries you should use TimeSeriesSplit. The CV score will be worser than kfold, but will be closer to the test set/production. Just make sure to have enough data.\nhttps://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\nhttps://towardsdatascience.com/time-series-nested-cross-validation-76adba623eb9\nhttps://conlanscientific.com/posts/category/blog/post/avoiding-data-leakage-machine-learning/
", "rawMarkdown": "To avoid data leakage in timeseries you should use TimeSeriesSplit. The CV score will be worser than kfold, but will be closer to the test set/production. Just make sure to have enough data.\nhttps://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\nhttps://towardsdatascience.com/time-series-nested-cross-validation-76adba623eb9\nhttps://conlanscientific.com/posts/category/blog/post/avoiding-data-leakage-machine-learning/", "votes": null }, { "id": "665291", "postDate": "11/04/2019 21:18:21", "content": "It is necessary to first order the data in time before use TimeSeriesSplit. It was noted that they may not be consistent in time in this competition.
Hi @ragnar123 , I tried TimeSeriesSplit CV but I hadn't reliable results. I think that is mainly due to the fact that building ids ratio was not the same...
\n\nAs you said, StratifiedKFold easily gets temporal leakage. I think we should implement something in between, probably we could lose some data to get a stratified validation divided by time. Did someone try something like that?
", "votes": null, "replies": [ { "id": 657730, "author_name": "ragnar123", "author_url": "", "post_date": "10/25/2019 11:51:47", "content": "I tried my strategy but it failed. In my validation schema month was a very important feature but for predicting the test it overfitts big time. Maybee the the validation schema that we are looking for is just using the leader board because the test and train have very different distributions. In other words no local validation strategy to do feature engineering and feature selection, just use the lb.
", "votes": null, "replies": [] } ] }, { "id": 657013, "author_name": "ragnar123", "author_url": "", "post_date": "10/24/2019 20:56:28", "content": "Maybee try a groupkfold with 4 folds by quarter..... i will try other ideas and if i have a good result i will share it here
\n\nRecently i made some feature selection and feature engineering with the following schema:
\n\nTrain with the first 8 month and then evaluate with the last 4 month
\n\nThen i pick those features and for predicting the test i use 5 StratifiedKFold (builiding_id) but i manually put 1000 boosting rounds to avoid offerfitting.
\n\nI want to check if the avaluation strategy align with the test, in other words if my eval with the 8/4 improve, also my test improve training with 5 StratifiedKFold with the features i selected in the 8/4 schema.
", "votes": null, "replies": [ { "id": 657021, "author_name": "raimonds1993", "author_url": "", "post_date": "10/24/2019 21:07:43", "content": "The fact is using just a time-based split doesn't guarantee you having all building ids in both training and validation sets. Then, I think GroupKFold on quarter doesn't prevent time leakage too cause. Good link to visualize CV methods.
\n\nI'm not sure about predicting using a different method from training. Isn't that like forcing different train and test distributions?
", "votes": null, "replies": [] } ] }, { "id": 661435, "author_name": "hmavrodiev", "author_url": "", "post_date": "10/30/2019 09:41:55", "content": "To avoid data leakage in timeseries you should use TimeSeriesSplit. The CV score will be worser than kfold, but will be closer to the test set/production. Just make sure to have enough data.\nhttps://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\nhttps://towardsdatascience.com/time-series-nested-cross-validation-76adba623eb9\nhttps://conlanscientific.com/posts/category/blog/post/avoiding-data-leakage-machine-learning/
", "votes": null, "replies": [] }, { "id": 665291, "author_name": "vbmokin", "author_url": "", "post_date": "11/04/2019 21:18:21", "content": "It is necessary to first order the data in time before use TimeSeriesSplit. It was noted that they may not be consistent in time in this competition.