Hi, I understand we are supposed to predict the future unseen data. But at this stage, I want to first try can I predict the revenue of the test set between May 01, 2018 to Oct 15, 2018. I preprocessed the test set exactly the same as training. If I further split my training set to train, and test (ignore the original test set), then my cv score is close to score on the test set (the new test set I extracted from the original training). However, if I report the CV socre on the original training set, and check the test score on the original test set. This two score is wildly off (test error is ten times bigger than cv). I'm wondering if there is something special in the test set (it shouldn't be...) or my validation strategy is incorrect?
", "messages": [ { "id": "424133", "postDate": "11/19/2018 16:10:41", "content": "Hi, I understand we are supposed to predict the future unseen data. But at this stage, I want to first try can I predict the revenue of the test set between May 01, 2018 to Oct 15, 2018. I preprocessed the test set exactly the same as training. If I further split my training set to train, and test (ignore the original test set), then my cv score is close to score on the test set (the new test set I extracted from the original training). However, if I report the CV socre on the original training set, and check the test score on the original test set. This two score is wildly off (test error is ten times bigger than cv). I'm wondering if there is something special in the test set (it shouldn't be...) or my validation strategy is incorrect?
", "rawMarkdown": "Hi, I understand we are supposed to predict the future unseen data. But at this stage, I want to first try can I predict the revenue of the test set between May 01, 2018 to Oct 15, 2018. I preprocessed the test set exactly the same as training. If I further split my training set to train, and test (ignore the original test set), then my cv score is close to score on the test set (the new test set I extracted from the original training). However, if I report the CV socre on the original training set, and check the test score on the original test set. This two score is wildly off (test error is ten times bigger than cv). I'm wondering if there is something special in the test set (it shouldn't be...) or my validation strategy is incorrect?", "votes": null }, { "id": "424140", "postDate": "11/19/2018 16:22:40", "content": "I'm not sure what you mean by \"cv score on validation\". CV involves taking a large dataset (one set, usually 'train', which can technically be a superset but I've never heard of that before) and running train_test multiple times on that one set. It's OOF results will be a more robust predictor of out-of-sample prediction accuracy than will a traditional train_test split. As a side-note, if I were you, I wouldn't discard the provided test set, that's meant to be your validation data.
", "rawMarkdown": "I'm not sure what you mean by \"cv score on validation\". CV involves taking a large dataset (one set, usually 'train', which can technically be a superset but I've never heard of that before) and running train\\_test multiple times on that one set. It's OOF results will be a more robust predictor of out-of-sample prediction accuracy than will a traditional train\\_test split. As a side-note, if I were you, I wouldn't discard the provided test set, that's meant to be your validation data.", "votes": null }, { "id": "424147", "postDate": "11/19/2018 16:30:06", "content": "Sorry for the confusion. I've edited my question. Yes, I will definitely not discard the provided test set. Just curious why my test score on the test set I split is so different from the score on the provided.
", "rawMarkdown": "Sorry for the confusion. I've edited my question. Yes, I will definitely not discard the provided test set. Just curious why my test score on the test set I split is so different from the score on the provided.", "votes": null } ], "comments": [ { "id": 424140, "author_name": "alecthekulak", "author_url": "", "post_date": "11/19/2018 16:22:40", "content": "I'm not sure what you mean by \"cv score on validation\". CV involves taking a large dataset (one set, usually 'train', which can technically be a superset but I've never heard of that before) and running train_test multiple times on that one set. It's OOF results will be a more robust predictor of out-of-sample prediction accuracy than will a traditional train_test split. As a side-note, if I were you, I wouldn't discard the provided test set, that's meant to be your validation data.
", "votes": null, "replies": [ { "id": 424147, "author_name": "guoziwei", "author_url": "", "post_date": "11/19/2018 16:30:06", "content": "Sorry for the confusion. I've edited my question. Yes, I will definitely not discard the provided test set. Just curious why my test score on the test set I split is so different from the score on the provided.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "424133": "Hi, I understand we are supposed to predict the future unseen data. But at this stage, I want to first try can I predict the revenue of the test set between May 01, 2018 to Oct 15, 2018. I preprocessed the test set exactly the same as training. If I further split my training set to train, and test (ignore the original test set), then my cv score is close to score on the test set (the new test set I extracted from the original training). However, if I report the CV socre on the original training set, and check the test score on the original test set. This two score is wildly off (test error is ten times bigger than cv). I'm wondering if there is something special in the test set (it shouldn't be...) or my validation strategy is incorrect?", "424140": "I'm not sure what you mean by \"cv score on validation\". CV involves taking a large dataset (one set, usually 'train', which can technically be a superset but I've never heard of that before) and running train\\_test multiple times on that one set. It's OOF results will be a more robust predictor of out-of-sample prediction accuracy than will a traditional train\\_test split. As a side-note, if I were you, I wouldn't discard the provided test set, that's meant to be your validation data.", "424147": "Sorry for the confusion. I've edited my question. Yes, I will definitely not discard the provided test set. Just curious why my test score on the test set I split is so different from the score on the provided." }, "source": "meta" }