{ "id": 122008, "title": "Great validation score but poor test score", "url": "/competitions/ashrae-energy-prediction/discussion/122008", "author_name": "ram sastry", "post_date": "2019-12-17T02:48:49.601000", "votes": 3, "comment_count": 6, "views": 0, "content": "

Friends,

\n\n

I am having this frustrating issue and not sure what I am doing wrong. I using LightGBM with 5-fold CV, and averaging the predictions over these 5 models. I get great validation scores, see for e.g.

\n\n

[50] training's rmse: 0.636076 valid_1's rmse: 0.64541\n[100] training's rmse: 0.590285 valid_1's rmse: 0.605914\n[150] training's rmse: 0.562677 valid_1's rmse: 0.584563\n[200] training's rmse: 0.54351 valid_1's rmse: 0.570618\n[250] training's rmse: 0.528848 valid_1's rmse: 0.560723\n[300] training's rmse: 0.516948 valid_1's rmse: 0.553563\n[350] training's rmse: 0.506634 valid_1's rmse: 0.547509

\n\n

But when do predictions on the test set, score is woefully poor, on the order of 2.3+. Any thoughts on what I could be doing wrong?

\n\n

Cheers\nRam

", "messages": [ { "id": 696780, "postDate": "2019-12-17T03:37:12.020Z", "content": "

you are either shuffling or using a feature that is overfitting. My first guess is you are shuffling.

", "rawMarkdown": "you are either shuffling or using a feature that is overfitting. My first guess is you are shuffling.", "votes": 3 }, { "id": 696766, "postDate": "2019-12-17T02:48:49.600Z", "content": "

Friends,

\n\n

I am having this frustrating issue and not sure what I am doing wrong. I using LightGBM with 5-fold CV, and averaging the predictions over these 5 models. I get great validation scores, see for e.g.

\n\n

[50] training's rmse: 0.636076 valid_1's rmse: 0.64541\n[100] training's rmse: 0.590285 valid_1's rmse: 0.605914\n[150] training's rmse: 0.562677 valid_1's rmse: 0.584563\n[200] training's rmse: 0.54351 valid_1's rmse: 0.570618\n[250] training's rmse: 0.528848 valid_1's rmse: 0.560723\n[300] training's rmse: 0.516948 valid_1's rmse: 0.553563\n[350] training's rmse: 0.506634 valid_1's rmse: 0.547509

\n\n

But when do predictions on the test set, score is woefully poor, on the order of 2.3+. Any thoughts on what I could be doing wrong?

\n\n

Cheers\nRam

", "rawMarkdown": "Friends,\n\nI am having this frustrating issue and not sure what I am doing wrong. I using LightGBM with 5-fold CV, and averaging the predictions over these 5 models. I get great validation scores, see for e.g.\n\n`[50]\ttraining's rmse: 0.636076\tvalid_1's rmse: 0.64541\n[100]\ttraining's rmse: 0.590285\tvalid_1's rmse: 0.605914\n[150]\ttraining's rmse: 0.562677\tvalid_1's rmse: 0.584563\n[200]\ttraining's rmse: 0.54351\tvalid_1's rmse: 0.570618\n[250]\ttraining's rmse: 0.528848\tvalid_1's rmse: 0.560723\n[300]\ttraining's rmse: 0.516948\tvalid_1's rmse: 0.553563\n[350]\ttraining's rmse: 0.506634\tvalid_1's rmse: 0.547509`\n\nBut when do predictions on the test set, score is woefully poor, on the order of 2.3+. Any thoughts on what I could be doing wrong?\n\nCheers\nRam", "votes": 3 }, { "id": 697278, "postDate": "2019-12-17T17:12:01.883Z", "content": "

As I recall I got a 2.11 and a 3.32 when I had my very best rmse values, slightly better than your numbers.

\n\n

I was deleting all the zero and NAN rows from the training set along with the initial months of the site_0 values - my created model was a good match to the training but my elimination of those rows made the training different from the test.

", "rawMarkdown": "As I recall I got a 2.11 and a 3.32 when I had my very best rmse values, slightly better than your numbers. \n\nI was deleting all the zero and NAN rows from the training set along with the initial months of the site_0 values - my created model was a good match to the training but my elimination of those rows made the training different from the test. ", "votes": 1, "replies": [ { "id": 697619, "postDate": "2019-12-18T07:09:56.570Z", "content": "

Data cleaning/outlier removal is particularly tricky for just this reason. It's quite hard to judge whether you are training a better model or just eliminating challenging but legitimate entries from your validation set. I usually try to err on the side of caution and adapt my CV to drop potential outliers from training but not from eval. Either that or hold my nose and resort to leaderboard probing.

", "rawMarkdown": "Data cleaning/outlier removal is particularly tricky for just this reason. It's quite hard to judge whether you are training a better model or just eliminating challenging but legitimate entries from your validation set. I usually try to err on the side of caution and adapt my CV to drop potential outliers from training but *not* from eval. Either that or hold my nose and resort to leaderboard probing." } ] }, { "id": 696918, "postDate": "2019-12-17T08:22:03.680Z", "content": "

I'd agree that you need to be careful about how you do your CV split. Since this competition is time-sequence, it's pretty much vital that you do some sort of time-based split -- and finding just the right split can be pretty hard.

\n\n

In addition, your brief snippet leads me to suspect that you are evaluating based on early-stopping scores. I try to avoid doing this because the early-stopping process is pretty much designed to overfit on its validation data, in the interest of not overfitting on the training data. Sadly, I suspect that the only safe way to evaluate early stopping is to put CVs inside your CVs or to do a three-way split instead of a two-way split.

", "rawMarkdown": "I'd agree that you need to be careful about how you do your CV split. Since this competition is time-sequence, it's pretty much vital that you do some sort of time-based split -- and finding just the right split can be pretty hard.\n\nIn addition, your brief snippet leads me to suspect that you are evaluating based on early-stopping scores. I try to avoid doing this because the early-stopping process is pretty much designed to overfit on its validation data, in the interest of *not* overfitting on the training data. Sadly, I suspect that the only safe way to evaluate early stopping is to put CVs inside your CVs or to do a three-way split instead of a two-way split.", "votes": 2 }, { "id": 697204, "postDate": "2019-12-17T16:02:28.077Z", "content": "

Tim/Rob, thx for your prompt replies. Indeed I set shuffling to True, and maybe that was the cause. As Rob was saying, this is time series data and its vital to do some sort of time-based split. I will set Shuffling = False, much easier :-).

", "rawMarkdown": "Tim/Rob, thx for your prompt replies. Indeed I set shuffling to True, and maybe that was the cause. As Rob was saying, this is time series data and its vital to do some sort of time-based split. I will set Shuffling = False, much easier :-). " }, { "id": 697369, "postDate": "2019-12-17T19:48:25.267Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 696780, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-12-17T03:37:12.020000", "content": "

you are either shuffling or using a feature that is overfitting. My first guess is you are shuffling.

", "votes": 3, "replies": [] }, { "id": 697278, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-12-17T17:12:01.883000", "content": "

As I recall I got a 2.11 and a 3.32 when I had my very best rmse values, slightly better than your numbers.

\n\n

I was deleting all the zero and NAN rows from the training set along with the initial months of the site_0 values - my created model was a good match to the training but my elimination of those rows made the training different from the test.

", "votes": 1, "replies": [ { "id": 697619, "author_name": "Robert Stockton", "author_url": "", "post_date": "2019-12-18T07:09:56.570000", "content": "

Data cleaning/outlier removal is particularly tricky for just this reason. It's quite hard to judge whether you are training a better model or just eliminating challenging but legitimate entries from your validation set. I usually try to err on the side of caution and adapt my CV to drop potential outliers from training but not from eval. Either that or hold my nose and resort to leaderboard probing.

", "votes": 0, "replies": [] } ] }, { "id": 696918, "author_name": "Robert Stockton", "author_url": "", "post_date": "2019-12-17T08:22:03.680000", "content": "

I'd agree that you need to be careful about how you do your CV split. Since this competition is time-sequence, it's pretty much vital that you do some sort of time-based split -- and finding just the right split can be pretty hard.

\n\n

In addition, your brief snippet leads me to suspect that you are evaluating based on early-stopping scores. I try to avoid doing this because the early-stopping process is pretty much designed to overfit on its validation data, in the interest of not overfitting on the training data. Sadly, I suspect that the only safe way to evaluate early stopping is to put CVs inside your CVs or to do a three-way split instead of a two-way split.

", "votes": 2, "replies": [] }, { "id": 697204, "author_name": "ram sastry", "author_url": "", "post_date": "2019-12-17T16:02:28.077000", "content": "

Tim/Rob, thx for your prompt replies. Indeed I set shuffling to True, and maybe that was the cause. As Rob was saying, this is time series data and its vital to do some sort of time-based split. I will set Shuffling = False, much easier :-).

", "votes": 0, "replies": [] }, { "id": 697369, "author_name": "", "author_url": "", "post_date": "2019-12-17T19:48:25.267000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "696780": "you are either shuffling or using a feature that is overfitting. My first guess is you are shuffling.", "696766": "Friends,\n\nI am having this frustrating issue and not sure what I am doing wrong. I using LightGBM with 5-fold CV, and averaging the predictions over these 5 models. I get great validation scores, see for e.g.\n\n`[50]\ttraining's rmse: 0.636076\tvalid_1's rmse: 0.64541\n[100]\ttraining's rmse: 0.590285\tvalid_1's rmse: 0.605914\n[150]\ttraining's rmse: 0.562677\tvalid_1's rmse: 0.584563\n[200]\ttraining's rmse: 0.54351\tvalid_1's rmse: 0.570618\n[250]\ttraining's rmse: 0.528848\tvalid_1's rmse: 0.560723\n[300]\ttraining's rmse: 0.516948\tvalid_1's rmse: 0.553563\n[350]\ttraining's rmse: 0.506634\tvalid_1's rmse: 0.547509`\n\nBut when do predictions on the test set, score is woefully poor, on the order of 2.3+. Any thoughts on what I could be doing wrong?\n\nCheers\nRam", "697278": "As I recall I got a 2.11 and a 3.32 when I had my very best rmse values, slightly better than your numbers. \n\nI was deleting all the zero and NAN rows from the training set along with the initial months of the site_0 values - my created model was a good match to the training but my elimination of those rows made the training different from the test. ", "696918": "I'd agree that you need to be careful about how you do your CV split. Since this competition is time-sequence, it's pretty much vital that you do some sort of time-based split -- and finding just the right split can be pretty hard.\n\nIn addition, your brief snippet leads me to suspect that you are evaluating based on early-stopping scores. I try to avoid doing this because the early-stopping process is pretty much designed to overfit on its validation data, in the interest of *not* overfitting on the training data. Sadly, I suspect that the only safe way to evaluate early stopping is to put CVs inside your CVs or to do a three-way split instead of a two-way split.", "697204": "Tim/Rob, thx for your prompt replies. Indeed I set shuffling to True, and maybe that was the cause. As Rob was saying, this is time series data and its vital to do some sort of time-based split. I will set Shuffling = False, much easier :-). ", "697369": "" } }