It is said that creating a validation dataset that is similar to test dataset helps us keep a check on local cv and public lb. How should a validation dataset be made here? On the basis of time/customers/some random sampling/anything else..
", "messages": [ { "id": 389126, "postDate": "2018-09-18T07:47:08.310Z", "content": "It is said that creating a validation dataset that is similar to test dataset helps us keep a check on local cv and public lb. How should a validation dataset be made here? On the basis of time/customers/some random sampling/anything else..
", "rawMarkdown": "It is said that creating a validation dataset that is similar to test dataset helps us keep a check on local cv and public lb. How should a validation dataset be made here? On the basis of time/customers/some random sampling/anything else..", "votes": 3 }, { "id": 389708, "postDate": "2018-09-19T05:25:49.143Z", "content": "Thank you guys for your suggestions. Regarding the time period, I am going to create a validation data set with the length of time period same as test dataset. I think this will help in emulating the test dataset and the similar results in local cv and public lb can be more trustworthy
", "rawMarkdown": "Thank you guys for your suggestions. Regarding the time period, I am going to create a validation data set with the length of time period same as test dataset. I think this will help in emulating the test dataset and the similar results in local cv and public lb can be more trustworthy" }, { "id": 389308, "postDate": "2018-09-18T14:17:34.287Z", "content": "Currently to create my validation set i'm using the basic random sampling, but as Jacob mentioned seems that using time would be a better choice, also it is a better practice for production models, as it mimics reality, but maybe you can try both and see what results get more close to the public result.
", "rawMarkdown": "Currently to create my validation set i'm using the basic random sampling, but as Jacob mentioned seems that using time would be a better choice, also it is a better practice for production models, as it mimics reality, but maybe you can try both and see what results get more close to the public result.", "replies": [ { "id": 389443, "postDate": "2018-09-18T18:14:45.203Z", "rawMarkdown": "", "isDeleted": true }, { "id": 389513, "postDate": "2018-09-18T21:02:10.373Z", "content": "The public leaderboard results are calculated on a portion of the testing data and the final evaluation is calculated on the ones left out of the leaderboard. You don't want to overfit the public leaderboard - that's true. However, both the final evaluation and public leaderboard use the testing data, which is separated in time. If time plays a role the predictions, making a CV set that emulates predicting a different time period could be important.
", "rawMarkdown": "The public leaderboard results are calculated on a portion of the testing data and the final evaluation is calculated on the ones left out of the leaderboard. You don't want to overfit the public leaderboard - that's true. However, both the final evaluation and public leaderboard use the testing data, which is separated in time. If time plays a role the predictions, making a CV set that emulates predicting a different time period could be important.", "votes": 2 } ] }, { "id": 389167, "postDate": "2018-09-18T09:31:42.597Z", "content": "The training and testing data are separated by time, so it's probably worthwhile to mimic this in your validation set.
", "rawMarkdown": "The training and testing data are separated by time, so it's probably worthwhile to mimic this in your validation set." } ], "comments": [ { "id": 389708, "author_name": "DebayanDaw", "author_url": "", "post_date": "2018-09-19T05:25:49.143000", "content": "Thank you guys for your suggestions. Regarding the time period, I am going to create a validation data set with the length of time period same as test dataset. I think this will help in emulating the test dataset and the similar results in local cv and public lb can be more trustworthy
", "votes": 0, "replies": [] }, { "id": 389308, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2018-09-18T14:17:34.287000", "content": "Currently to create my validation set i'm using the basic random sampling, but as Jacob mentioned seems that using time would be a better choice, also it is a better practice for production models, as it mimics reality, but maybe you can try both and see what results get more close to the public result.
", "votes": 0, "replies": [ { "id": 389443, "author_name": "", "author_url": "", "post_date": "2018-09-18T18:14:45.203000", "content": "", "votes": 0, "replies": [] }, { "id": 389513, "author_name": "Jacob van Gogh", "author_url": "", "post_date": "2018-09-18T21:02:10.373000", "content": "The public leaderboard results are calculated on a portion of the testing data and the final evaluation is calculated on the ones left out of the leaderboard. You don't want to overfit the public leaderboard - that's true. However, both the final evaluation and public leaderboard use the testing data, which is separated in time. If time plays a role the predictions, making a CV set that emulates predicting a different time period could be important.
", "votes": 2, "replies": [] } ] }, { "id": 389167, "author_name": "Jacob van Gogh", "author_url": "", "post_date": "2018-09-18T09:31:42.597000", "content": "The training and testing data are separated by time, so it's probably worthwhile to mimic this in your validation set.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "389126": "It is said that creating a validation dataset that is similar to test dataset helps us keep a check on local cv and public lb. How should a validation dataset be made here? On the basis of time/customers/some random sampling/anything else..", "389708": "Thank you guys for your suggestions. Regarding the time period, I am going to create a validation data set with the length of time period same as test dataset. I think this will help in emulating the test dataset and the similar results in local cv and public lb can be more trustworthy", "389308": "Currently to create my validation set i'm using the basic random sampling, but as Jacob mentioned seems that using time would be a better choice, also it is a better practice for production models, as it mimics reality, but maybe you can try both and see what results get more close to the public result.", "389167": "The training and testing data are separated by time, so it's probably worthwhile to mimic this in your validation set." } }