For me, I tried unshuffle 5-fold CV and holdout (train on the first 4 months, skip 4 months, validate on the last 4 months), both of 2 schemes seems not very stable though.
", "messages": [ { "id": "653846", "postDate": "10/21/2019 03:29:11", "content": "For me, I tried unshuffle 5-fold CV and holdout (train on the first 4 months, skip 4 months, validate on the last 4 months), both of 2 schemes seems not very stable though.
", "rawMarkdown": "For me, I tried unshuffle 5-fold CV and holdout (train on the first 4 months, skip 4 months, validate on the last 4 months), both of 2 schemes seems not very stable though.", "votes": null }, { "id": "654267", "postDate": "10/21/2019 16:26:50", "content": "Split into halves by time\n\ntrain = train.sort_index(by='timestamp').reset_index(drop=True)\ntrain, holdout = train.iloc[:int(len(train) * 0.5)], train.iloc[int(len(train) * 0.5):]\n\nStratify by building_id\n\nX_train, X_test, y_train, y_test = train_test_split(train, y, test_size=0.25, random_state=47, stratify=train['building_id'])\n\nFit with CV control over X_test
Then predict on holdout and get the score.\n\npreds = clf.predict(holdout)\nnp.sqrt(mean_squared_error(preds, y_holdout))\n
Thanks for your reply! I will definitely try your method
", "rawMarkdown": "Thanks for your reply! I will definitely try your method", "votes": null }, { "id": "655130", "postDate": "10/22/2019 17:50:19", "content": "@nroman How consistent are your holdout predictions with the LB score?
", "rawMarkdown": "nroman How consistent are your holdout predictions with the LB score?", "votes": null }, { "id": "655324", "postDate": "10/22/2019 23:11:17", "content": "I might be missing something obvious here but to me it doesn't make sense to split by time.
\n\nYou will be unable to use yearly features in your model which can have a large impact such as:
\n\n*seasons\n*holidays\n*school vacation\n*key business dates ( Q1/Q2 etc) , 4/15,
\n\nThese and other yearly effects are more likely to have a bigger impact (variance) on meter reading in the short term of this dataset ( 3 years) then a small 3 year trend increase in meter reading. For this reason, I don't think you should be splitting on the basis of time.
\n\nthink the task here is not forecasting & prediction, but estimation of a hidden state or latent variable.
", "rawMarkdown": "I might be missing something obvious here but to me it doesn't make sense to split by time. \n\nYou will be unable to use yearly features in your model which can have a large impact such as:\n\n*seasons\n*holidays\n*school vacation\n*key business dates ( Q1/Q2 etc) , 4/15,\n\nThese and other yearly effects are more likely to have a bigger impact (variance) on meter reading in the short term of this dataset ( 3 years) then a small 3 year trend increase in meter reading. For this reason, I don't think you should be splitting on the basis of time.\n\n\n think the task here is not forecasting & prediction, but estimation of a hidden state or latent variable.", "votes": null }, { "id": "655530", "postDate": "10/23/2019 06:12:45", "content": "Pretty consistent. Even though there is a difference between LB and CV in around 0.01 but improvement in CV always leads to improvement on LB.
", "rawMarkdown": "Pretty consistent. Even though there is a difference between LB and CV in around 0.01 but improvement in CV always leads to improvement on LB.", "votes": null }, { "id": "658715", "postDate": "10/26/2019 12:23:17", "content": "@nroman Thanks for your sharing. I have a large gap between the cv rmse loss and my holdout loss after trying your method. I'm wondering whether it's normal or not?
", "rawMarkdown": "nroman Thanks for your sharing. I have a large gap between the cv rmse loss and my holdout loss after trying your method. I'm wondering whether it's normal or not?", "votes": null }, { "id": "658757", "postDate": "10/26/2019 13:39:49", "content": "Overfitting is an issue in this competition. Mostly because of the regression type of the task and the way gradient boosting models perform them.\nTry lowering number of boosting rounds (iterations) and see how that would affect your holdout set.
", "rawMarkdown": "Overfitting is an issue in this competition. Mostly because of the regression type of the task and the way gradient boosting models perform them.\nTry lowering number of boosting rounds (iterations) and see how that would affect your holdout set.", "votes": null }, { "id": "658931", "postDate": "10/26/2019 19:13:26", "content": "\"Overfitting is an issue in this competition\"...
\n\nas allways here, you are very welcome !
", "rawMarkdown": "\"Overfitting is an issue in this competition\"...\n\nas allways here, you are very welcome !", "votes": null }, { "id": "659682", "postDate": "10/28/2019 04:39:22", "content": "@nroman I am assuming that you are using this scheme to generate your local CV score and feature engineering. But for submission you must be building a model using entire 2016 data. Are you using any CV at that stage?
", "rawMarkdown": "nroman I am assuming that you are using this scheme to generate your local CV score and feature engineering. But for submission you must be building a model using entire 2016 data. Are you using any CV at that stage?", "votes": null }, { "id": "659828", "postDate": "10/28/2019 10:04:42", "content": "Pretty sure making time split will ruin your model. But it could be used to ensure your features generalize well.
", "rawMarkdown": "Pretty sure making time split will ruin your model. But it could be used to ensure your features generalize well.", "votes": null }, { "id": "659889", "postDate": "10/28/2019 11:52:41", "content": "@nroman \n> improvement in CV always leads to improvement on LB.
\n\nwhat kind of changes do you make to see the corr between cv and lb? parameter tuning, feature engineering, or feature selection?
", "rawMarkdown": "nroman \n> improvement in CV always leads to improvement on LB.\n\nwhat kind of changes do you make to see the corr between cv and lb? parameter tuning, feature engineering, or feature selection?", "votes": null }, { "id": "659976", "postDate": "10/28/2019 14:15:19", "content": "Feature engineering and number of iterations.
", "rawMarkdown": "Feature engineering and number of iterations.", "votes": null }, { "id": "660052", "postDate": "10/28/2019 15:51:32", "content": "Until few days back, I used to think there should be only one validation scheme selected for a particular competition/use case and that should simulate the test set (Public/Private LB). But, based on the discussions which happened during IEEE Fraud Detection competition, I understood, that may not strictly be the case.
\n\nAs suggested by Chris (here), there could be two validation schemes:
\n\nMoreover, even for #1, @cdeotte & @kyakovlev used multiple validation schemes (here). Even, I guess @senkin13 has suggested the same here:
\n\nSince it's a timeseries problem,so you would be better to split train and validation set by timestamp , sometimes kfold is also can be used for timeseries problem but it's better to check first,you can split train and validation set by timestamp, then train a kfold-model using train set to watch the validation set performance.
Enlightened with that knowledge, here is my thought for this competition:
\n\nHere the data is temporal in nature. Hence, for FE and hyperparameter tuning, we should have one or more time based holdout scheme(s). It should noted that the features built should be able to capture very long term trend. Considering, Train data - 2016, Public Test Data - Jan 2017 to June 2018, Private Test Data - July 2018 to Dec 2018, the gap between Train Data and Private LB is almost 1.5 years. So, the feature should be capable of predicting 1.5 years ahead. I don't think it's at all possible to simulate such a validation mechanism (train.csv is just for a year). Also, the models built at this stage will not be trained on the entire train data, hence can't be used for prediction/submission. @kyakovlev has his suggestion here.
Once the FE and hypeparemeter selection is done, for the prediction, may be Stratified K-Fold based CV mechanism should be used (using the entire training data). As @nroman has suggested that could possibly be StratifiedKFold using building_id.
Should we use feedback from Public LB or not? Since the size of the public LB > Training Data > Private LB (as per the LB probing that has happened till this point of time), I think it should be.
Now, I have never won any medal in any competition. So, take my words with lots of salt. Also, if you think otherwise, please let me know.
", "rawMarkdown": "Until few days back, I used to think there should be only one validation scheme selected for a particular competition/use case and that should simulate the test set (Public/Private LB). But, based on the discussions which happened during IEEE Fraud Detection competition, I understood, that may not strictly be the case.\n\nAs suggested by Chris ([here](https://www.kaggle.com/c/ieee-fraud-detection/discussion/108571#625045)), there could be two validation schemes:\n\n1. First one to understand the effectiveness of the created features and hyperparameter tuning. \n2. The second one is for prediction on the test data (submission).\n\nMoreover, even for #1, @cdeotte & @kyakovlev used multiple validation schemes ([here](https://www.kaggle.com/c/ieee-fraud-detection/discussion/111308#latest-657565)). Even, I guess @senkin13 has suggested the same [here](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113249#latest-658145):\n\n`Since it's a timeseries problem,so you would be better to split train and validation set by timestamp , sometimes kfold is also can be used for timeseries problem but it's better to check first,you can split train and validation set by timestamp, then train a kfold-model using train set to watch the validation set performance.`\n\nEnlightened with that knowledge, here is my thought for this competition:\n\n1. Here the data is temporal in nature. Hence, for FE and hyperparameter tuning, we should have one or more time based holdout scheme(s). It should noted that the features built should be able to capture very long term trend. Considering, Train data - 2016, Public Test Data - Jan 2017 to June 2018, Private Test Data - July 2018 to Dec 2018, the gap between Train Data and Private LB is almost 1.5 years. So, the feature should be capable of predicting 1.5 years ahead. I don't think it's at all possible to simulate such a validation mechanism (train.csv is just for a year). Also, the models built at this stage will not be trained on the entire train data, hence can't be used for prediction/submission. @kyakovlev has his suggestion [here](https://www.kaggle.com/kyakovlev/ashrae-cv-options#658800). \n\n2. Once the FE and hypeparemeter selection is done, for the prediction, may be Stratified K-Fold based CV mechanism should be used (using the entire training data). As @nroman has suggested that could possibly be StratifiedKFold using building_id.\n\n3. Should we use feedback from Public LB or not? Since the size of the public LB > Training Data > Private LB (as per the LB probing that has happened till this point of time), I think it should be.\n\n**Now, I have never won any medal in any competition. So, take my words with lots of salt.** Also, if you think otherwise, please let me know.", "votes": null }, { "id": "660575", "postDate": "10/29/2019 10:57:48", "content": "Why would splitting into folds based on time ruin your model?
", "rawMarkdown": "Why would splitting into folds based on time ruin your model?", "votes": null }, { "id": "662554", "postDate": "10/31/2019 17:22:10", "content": "Thanks for sharing. When generating the submission file, do you retrain the model using the whole training set or just X_train? If the whole training set, what is the number of boosting rounds?
", "rawMarkdown": "Thanks for sharing. When generating the submission file, do you retrain the model using the whole training set or just X_train? If the whole training set, what is the number of boosting rounds?", "votes": null }, { "id": "662970", "postDate": "11/01/2019 09:23:56", "content": "You will lose a lot of seasonal information
", "rawMarkdown": "You will lose a lot of seasonal information", "votes": null }, { "id": "665311", "postDate": "11/04/2019 21:35:24", "content": "I'm using a variant of nroman's strategy below where I also validate backwards using the second half (nroman_cv_double). It seems pretty stable locally, but not super well correlated to LB currently. Nothing I have tried is well correlated with LB currently.
```\nfrom lepmlutils import pdutils as pdu
\n\ndef nroman_cv_double(train, lgb_params, bad_cols):\n trn_e, trn_l = pdu.split_at_proportion(train, 0.5)\n score1, val1 = nroman_cv(trn_e, trn_l, lgb_params, bad_cols)\n print(f\"forwards score: {score1}, val: {val1}\")
\n\nscore2, val2 = nroman_cv(trn_l, trn_e, lgb_params, bad_cols)\nfinal = (score1 + score2) / 2\n\nprint(f\"backwards score: {score2}, val: {val2}\")\nprint(f\"final score: {final}, val: {(val1 + val2) / 2}\")\nreturn final \ndef nroman_cv(train, test, lgb_params, bad_cols):\n Xtrn, Xval, ytrn, yval = train_test_split(train, train[\"meter_reading\"], test_size=0.25, random_state=47, stratify=train['building_id'])\n preds, val_score = predict(Xtrn, ytrn, Xval, yval, test, lgb_params, bad_cols)\n return mean_squared_error(np.log1p(preds), np.log1p(test[\"meter_reading\"])), val_score
\n\ndef predict(Xtrn, ytrn, Xval, yval, test, prms, bad_cols):\n to_drop = bad_cols + [\"meter_reading\"]\n reg = lgb.LGBMRegressor(**prms)\n reg.fit(\n Xtrn.drop(to_drop, axis=1), \n np.log1p(ytrn), \n eval_metric=\"rmse\",\n eval_set=(Xval.drop(to_drop, axis=1), np.log1p(yval)),\n verbose = 50,\n )
\n\nval_score = best_scores(reg, \"rmse\")\n\npreds = np.expm1(reg.predict(test.drop(bad_cols, axis=1)))\nreturn preds, val_score\n```
\n\nI suggest not actually copy-pasting the code, since it relies on lepmlutils a library I wrote, but maybe it will give someone some fun ideas.
Mind explaining what you mean by 'Fit with CV control over X_test'?
", "rawMarkdown": "Mind explaining what you mean by 'Fit with CV control over X_test'?", "votes": null }, { "id": "674341", "postDate": "11/16/2019 09:53:58", "content": "Thanks for ur sharing!\nAre u still using this CV scheme?
", "rawMarkdown": "Thanks for ur sharing!\nAre u still using this CV scheme?", "votes": null } ], "comments": [ { "id": 654267, "author_name": "nroman", "author_url": "", "post_date": "10/21/2019 16:26:50", "content": "Split into halves by time\n\ntrain = train.sort_index(by='timestamp').reset_index(drop=True)\ntrain, holdout = train.iloc[:int(len(train) * 0.5)], train.iloc[int(len(train) * 0.5):]\n\nStratify by building_id\n\nX_train, X_test, y_train, y_test = train_test_split(train, y, test_size=0.25, random_state=47, stratify=train['building_id'])\n\nFit with CV control over X_test
Then predict on holdout and get the score.\n\npreds = clf.predict(holdout)\nnp.sqrt(mean_squared_error(preds, y_holdout))\n
Thanks for your reply! I will definitely try your method
", "votes": null, "replies": [] }, { "id": 655130, "author_name": "graf10a", "author_url": "", "post_date": "10/22/2019 17:50:19", "content": "@nroman How consistent are your holdout predictions with the LB score?
", "votes": null, "replies": [] }, { "id": 655530, "author_name": "nroman", "author_url": "", "post_date": "10/23/2019 06:12:45", "content": "Pretty consistent. Even though there is a difference between LB and CV in around 0.01 but improvement in CV always leads to improvement on LB.
", "votes": null, "replies": [] }, { "id": 658715, "author_name": "syoya1997", "author_url": "", "post_date": "10/26/2019 12:23:17", "content": "@nroman Thanks for your sharing. I have a large gap between the cv rmse loss and my holdout loss after trying your method. I'm wondering whether it's normal or not?
", "votes": null, "replies": [] }, { "id": 658757, "author_name": "nroman", "author_url": "", "post_date": "10/26/2019 13:39:49", "content": "Overfitting is an issue in this competition. Mostly because of the regression type of the task and the way gradient boosting models perform them.\nTry lowering number of boosting rounds (iterations) and see how that would affect your holdout set.
", "votes": null, "replies": [] }, { "id": 658931, "author_name": "pfichou", "author_url": "", "post_date": "10/26/2019 19:13:26", "content": "\"Overfitting is an issue in this competition\"...
\n\nas allways here, you are very welcome !
", "votes": null, "replies": [] }, { "id": 659682, "author_name": "arnabbiswas1", "author_url": "", "post_date": "10/28/2019 04:39:22", "content": "@nroman I am assuming that you are using this scheme to generate your local CV score and feature engineering. But for submission you must be building a model using entire 2016 data. Are you using any CV at that stage?
", "votes": null, "replies": [] }, { "id": 659889, "author_name": "marquis08", "author_url": "", "post_date": "10/28/2019 11:52:41", "content": "@nroman \n> improvement in CV always leads to improvement on LB.
\n\nwhat kind of changes do you make to see the corr between cv and lb? parameter tuning, feature engineering, or feature selection?
", "votes": null, "replies": [] }, { "id": 659976, "author_name": "nroman", "author_url": "", "post_date": "10/28/2019 14:15:19", "content": "Feature engineering and number of iterations.
", "votes": null, "replies": [] }, { "id": 662554, "author_name": "superds", "author_url": "", "post_date": "10/31/2019 17:22:10", "content": "Thanks for sharing. When generating the submission file, do you retrain the model using the whole training set or just X_train? If the whole training set, what is the number of boosting rounds?
", "votes": null, "replies": [] }, { "id": 674267, "author_name": "lawrencetoh", "author_url": "", "post_date": "11/16/2019 06:04:39", "content": "Mind explaining what you mean by 'Fit with CV control over X_test'?
", "votes": null, "replies": [] }, { "id": 674341, "author_name": "jerrywu0605", "author_url": "", "post_date": "11/16/2019 09:53:58", "content": "Thanks for ur sharing!\nAre u still using this CV scheme?
", "votes": null, "replies": [] } ] }, { "id": 655324, "author_name": "ptolemy", "author_url": "", "post_date": "10/22/2019 23:11:17", "content": "I might be missing something obvious here but to me it doesn't make sense to split by time.
\n\nYou will be unable to use yearly features in your model which can have a large impact such as:
\n\n*seasons\n*holidays\n*school vacation\n*key business dates ( Q1/Q2 etc) , 4/15,
\n\nThese and other yearly effects are more likely to have a bigger impact (variance) on meter reading in the short term of this dataset ( 3 years) then a small 3 year trend increase in meter reading. For this reason, I don't think you should be splitting on the basis of time.
\n\nthink the task here is not forecasting & prediction, but estimation of a hidden state or latent variable.
", "votes": null, "replies": [] }, { "id": 659828, "author_name": "wispzero", "author_url": "", "post_date": "10/28/2019 10:04:42", "content": "Pretty sure making time split will ruin your model. But it could be used to ensure your features generalize well.
", "votes": null, "replies": [ { "id": 660575, "author_name": "lewington", "author_url": "", "post_date": "10/29/2019 10:57:48", "content": "Why would splitting into folds based on time ruin your model?
", "votes": null, "replies": [] }, { "id": 662970, "author_name": "wispzero", "author_url": "", "post_date": "11/01/2019 09:23:56", "content": "You will lose a lot of seasonal information
", "votes": null, "replies": [] } ] }, { "id": 660052, "author_name": "arnabbiswas1", "author_url": "", "post_date": "10/28/2019 15:51:32", "content": "Until few days back, I used to think there should be only one validation scheme selected for a particular competition/use case and that should simulate the test set (Public/Private LB). But, based on the discussions which happened during IEEE Fraud Detection competition, I understood, that may not strictly be the case.
\n\nAs suggested by Chris (here), there could be two validation schemes:
\n\nMoreover, even for #1, @cdeotte & @kyakovlev used multiple validation schemes (here). Even, I guess @senkin13 has suggested the same here:
\n\nSince it's a timeseries problem,so you would be better to split train and validation set by timestamp , sometimes kfold is also can be used for timeseries problem but it's better to check first,you can split train and validation set by timestamp, then train a kfold-model using train set to watch the validation set performance.
Enlightened with that knowledge, here is my thought for this competition:
\n\nHere the data is temporal in nature. Hence, for FE and hyperparameter tuning, we should have one or more time based holdout scheme(s). It should noted that the features built should be able to capture very long term trend. Considering, Train data - 2016, Public Test Data - Jan 2017 to June 2018, Private Test Data - July 2018 to Dec 2018, the gap between Train Data and Private LB is almost 1.5 years. So, the feature should be capable of predicting 1.5 years ahead. I don't think it's at all possible to simulate such a validation mechanism (train.csv is just for a year). Also, the models built at this stage will not be trained on the entire train data, hence can't be used for prediction/submission. @kyakovlev has his suggestion here.
Once the FE and hypeparemeter selection is done, for the prediction, may be Stratified K-Fold based CV mechanism should be used (using the entire training data). As @nroman has suggested that could possibly be StratifiedKFold using building_id.
Should we use feedback from Public LB or not? Since the size of the public LB > Training Data > Private LB (as per the LB probing that has happened till this point of time), I think it should be.
Now, I have never won any medal in any competition. So, take my words with lots of salt. Also, if you think otherwise, please let me know.
", "votes": null, "replies": [] }, { "id": 665311, "author_name": "lewington", "author_url": "", "post_date": "11/04/2019 21:35:24", "content": "I'm using a variant of nroman's strategy below where I also validate backwards using the second half (nroman_cv_double). It seems pretty stable locally, but not super well correlated to LB currently. Nothing I have tried is well correlated with LB currently.
```\nfrom lepmlutils import pdutils as pdu
\n\ndef nroman_cv_double(train, lgb_params, bad_cols):\n trn_e, trn_l = pdu.split_at_proportion(train, 0.5)\n score1, val1 = nroman_cv(trn_e, trn_l, lgb_params, bad_cols)\n print(f\"forwards score: {score1}, val: {val1}\")
\n\nscore2, val2 = nroman_cv(trn_l, trn_e, lgb_params, bad_cols)\nfinal = (score1 + score2) / 2\n\nprint(f\"backwards score: {score2}, val: {val2}\")\nprint(f\"final score: {final}, val: {(val1 + val2) / 2}\")\nreturn final \ndef nroman_cv(train, test, lgb_params, bad_cols):\n Xtrn, Xval, ytrn, yval = train_test_split(train, train[\"meter_reading\"], test_size=0.25, random_state=47, stratify=train['building_id'])\n preds, val_score = predict(Xtrn, ytrn, Xval, yval, test, lgb_params, bad_cols)\n return mean_squared_error(np.log1p(preds), np.log1p(test[\"meter_reading\"])), val_score
\n\ndef predict(Xtrn, ytrn, Xval, yval, test, prms, bad_cols):\n to_drop = bad_cols + [\"meter_reading\"]\n reg = lgb.LGBMRegressor(**prms)\n reg.fit(\n Xtrn.drop(to_drop, axis=1), \n np.log1p(ytrn), \n eval_metric=\"rmse\",\n eval_set=(Xval.drop(to_drop, axis=1), np.log1p(yval)),\n verbose = 50,\n )
\n\nval_score = best_scores(reg, \"rmse\")\n\npreds = np.expm1(reg.predict(test.drop(bad_cols, axis=1)))\nreturn preds, val_score\n```
\n\nI suggest not actually copy-pasting the code, since it relies on lepmlutils a library I wrote, but maybe it will give someone some fun ideas.