Hi!
\n\nI am lost here, I have two similar models with very different scores. Model2 is a bit more aggressively hyper-parameter optimized and a small amount of new features, than Model1.
\n\nModel 1 CV: -1.53\nModel 1 LB about: -1.72
\n\nModel 2 CV: -1.72\nModel 2 LB: -1.49
\n\nYou could think that Model2 is over-fitting but training data is more than 4.600.000 records and leader board data is just around 730.000. It means for 1JHN only about 6500 records!
\n\nBoth my models are LightGBM with Kfold of 5 folds.
\n\nSo what happens here? Model2 performs better and LB dataset is too small, or Model2 is over-fitted?
", "messages": [ { "id": "608441", "postDate": "08/26/2019 18:54:34", "content": "Hi!
\n\nI am lost here, I have two similar models with very different scores. Model2 is a bit more aggressively hyper-parameter optimized and a small amount of new features, than Model1.
\n\nModel 1 CV: -1.53\nModel 1 LB about: -1.72
\n\nModel 2 CV: -1.72\nModel 2 LB: -1.49
\n\nYou could think that Model2 is over-fitting but training data is more than 4.600.000 records and leader board data is just around 730.000. It means for 1JHN only about 6500 records!
\n\nBoth my models are LightGBM with Kfold of 5 folds.
\n\nSo what happens here? Model2 performs better and LB dataset is too small, or Model2 is over-fitted?
", "rawMarkdown": "Hi!\n\nI am lost here, I have two similar models with very different scores. Model2 is a bit more aggressively hyper-parameter optimized and a small amount of new features, than Model1.\n\nModel 1 CV: -1.53\nModel 1 LB about: -1.72\n\nModel 2 CV: -1.72\nModel 2 LB: -1.49\n\nYou could think that Model2 is over-fitting but training data is more than 4.600.000 records and leader board data is just around 730.000. It means for 1JHN only about 6500 records!\n\nBoth my models are LightGBM with Kfold of 5 folds.\n\nSo what happens here? Model2 performs better and LB dataset is too small, or Model2 is over-fitted?", "votes": null }, { "id": "608473", "postDate": "08/26/2019 19:34:42", "content": "First of all, remember that you can use 2 submission as final ones.\nFirst case seems to be common situation for this competion because if you use 5 folds and only one split seed every prediction in your CV is a result of 1 model while the predictions in the submission is averaged result of 5 models. In this competioion averaging let boost significantly the performance of your model (and at my experience getting median works better than mean in this particular competion).\nI'm not sure about your Model 2 but I am inclined to believe that it's kind of mistake in preprocessing. It could be over-fitting, especially if you use early-stopping, the using of the one validation set for both early-stopping and evaluation can lead to over-fitting. But the difference is too big, so I'd check the pre-processing operations like Scalers and so on.
", "rawMarkdown": "First of all, remember that you can use 2 submission as final ones.\nFirst case seems to be common situation for this competion because if you use 5 folds and only one split seed every prediction in your CV is a result of 1 model while the predictions in the submission is averaged result of 5 models. In this competioion averaging let boost significantly the performance of your model (and at my experience getting median works better than mean in this particular competion).\nI'm not sure about your Model 2 but I am inclined to believe that it's kind of mistake in preprocessing. It could be over-fitting, especially if you use early-stopping, the using of the one validation set for both early-stopping and evaluation can lead to over-fitting. But the difference is too big, so I'd check the pre-processing operations like Scalers and so on.", "votes": null }, { "id": "608508", "postDate": "08/26/2019 20:54:36", "content": "\n\n\nYou could think that Model2 is over-fitting
\n
Do you have any doubt about this?
\n\n\n\n\nModel2 is a bit more aggressively hyper-parameter optimized
\n
I often say that HPO is the best way I know to overfit. Thanks for providing a blatant example.
", "rawMarkdown": "> You could think that Model2 is over-fitting\n\nDo you have any doubt about this?\n\n> Model2 is a bit more aggressively hyper-parameter optimized \n\nI often say that HPO is the best way I know to overfit. Thanks for providing a blatant example.", "votes": null }, { "id": "608541", "postDate": "08/26/2019 22:48:46", "content": "Previous answer may sound sarcastic when it is not the intent. One rule I try to obey is to only change one thing at a time. This way, one can be sure about the root of an issue if there is an issue. I therefore suggest you create model 3 and model 4 that are model1 for features and model 2 for parameters, or vice versa. This way you'll know if HPO is the culprit (as I assumed it was, maybe erroneously), or if your features are the issue.
", "rawMarkdown": "Previous answer may sound sarcastic when it is not the intent. One rule I try to obey is to only change one thing at a time. This way, one can be sure about the root of an issue if there is an issue. I therefore suggest you create model 3 and model 4 that are model1 for features and model 2 for parameters, or vice versa. This way you'll know if HPO is the culprit (as I assumed it was, maybe erroneously), or if your features are the issue.", "votes": null }, { "id": "608769", "postDate": "08/27/2019 06:46:46", "content": "@cpmpml long answer short you would trust the LB?
", "rawMarkdown": "cpmpml long answer short you would trust the LB?", "votes": null }, { "id": "608804", "postDate": "08/27/2019 07:18:00", "content": "I trust my cv, and it is very well correlated with my LB ;) I mean the cv lb gap is constant at least to 2nd decimal.
", "rawMarkdown": "I trust my cv, and it is very well correlated with my LB ;) I mean the cv lb gap is constant at least to 2nd decimal.", "votes": null }, { "id": "609305", "postDate": "08/27/2019 15:16:16", "content": "Thanks @cpmpml for your answers! Before my CV and LB score was also correlated. I am still investigating, with two new models which are only different in parameters.
", "rawMarkdown": "Thanks @cpmpml for your answers! Before my CV and LB score was also correlated. I am still investigating, with two new models which are only different in parameters.", "votes": null }, { "id": "609310", "postDate": "08/27/2019 15:19:25", "content": "Hi @silantev! You have a really good point on why Kfold can be better on test prediction than on train! Also good luck for the competition! You are a really cool guy to help someone how is so close behind you :D
", "rawMarkdown": "Hi @silantev! You have a really good point on why Kfold can be better on test prediction than on train! Also good luck for the competition! You are a really cool guy to help someone how is so close behind you :D", "votes": null }, { "id": "609457", "postDate": "08/27/2019 18:32:47", "content": "> the cv lb gap is constant at least to 2nd decimal
\n\nYour CV or LB is better score? CV is better, right? I am asking because I made some more submissions and it looks like my CV is constantly around 0.2 better than LB.
", "rawMarkdown": "> the cv lb gap is constant at least to 2nd decimal\n\n\nYour CV or LB is better score? CV is better, right? I am asking because I made some more submissions and it looks like my CV is constantly around 0.2 better than LB.", "votes": null }, { "id": "609491", "postDate": "08/27/2019 19:26:58", "content": "I met the same case as your second one. \nOne of my results has -1.788 cv score but only -1.3 lb score.
\n\nThen I check it manually and found there are several very wried predictions in my submission file, e.g. np.amax(test_prediction)=1092976.625, which influence my lb score a lot.
\n\nI just did median stacking with other predictions to fix it.
", "rawMarkdown": "I met the same case as your second one. \nOne of my results has -1.788 cv score but only -1.3 lb score.\n\nThen I check it manually and found there are several very wried predictions in my submission file, e.g. np.amax(test_prediction)=1092976.625, which influence my lb score a lot.\n\nI just did median stacking with other predictions to fix it.", "votes": null }, { "id": "610612", "postDate": "08/28/2019 23:02:09", "content": "TLDR; It was overfit.
\n\n@cpmpml @silantev @kirkchen1993 Thanks guys for you replays, it meant a lot to me. I was thinking very hard what happened, because all the way through the competition my CV and LB score was correlated, LB was always better than CV. Than suddenly three days before finish LB started to be worse than CV.
\n\nWhat happened is really easy, honestly it's a rookie mistake. I thought K-fold split is enough, I always treated my CV score as a good measure of model performance. This is not the case. Before K-fold split I should have separated some data from train.csv to validate how well my model is performing. Or if not, I should have been submitted much more data to kaggle servers, to have my models validated for me. But none of these have been done, I just treated my CV score as a good measure.
\n\nI just realized all this 2 hours before competition finish :) But it bothered me a lot, and I am happy that is clear to me now 👍
", "rawMarkdown": "TLDR; It was overfit.\n\n@cpmpml @silantev @kirkchen1993 Thanks guys for you replays, it meant a lot to me. I was thinking very hard what happened, because all the way through the competition my CV and LB score was correlated, LB was always better than CV. Than suddenly three days before finish LB started to be worse than CV.\n\nWhat happened is really easy, honestly it's a rookie mistake. I thought K-fold split is enough, I always treated my CV score as a good measure of model performance. This is not the case. Before K-fold split I should have separated some data from train.csv to validate how well my model is performing. Or if not, I should have been submitted much more data to kaggle servers, to have my models validated for me. But none of these have been done, I just treated my CV score as a good measure. \n\nI just realized all this 2 hours before competition finish :) But it bothered me a lot, and I am happy that is clear to me now 👍", "votes": null } ], "comments": [ { "id": 608473, "author_name": "silantev", "author_url": "", "post_date": "08/26/2019 19:34:42", "content": "First of all, remember that you can use 2 submission as final ones.\nFirst case seems to be common situation for this competion because if you use 5 folds and only one split seed every prediction in your CV is a result of 1 model while the predictions in the submission is averaged result of 5 models. In this competioion averaging let boost significantly the performance of your model (and at my experience getting median works better than mean in this particular competion).\nI'm not sure about your Model 2 but I am inclined to believe that it's kind of mistake in preprocessing. It could be over-fitting, especially if you use early-stopping, the using of the one validation set for both early-stopping and evaluation can lead to over-fitting. But the difference is too big, so I'd check the pre-processing operations like Scalers and so on.
", "votes": null, "replies": [ { "id": 609310, "author_name": "edgarpe", "author_url": "", "post_date": "08/27/2019 15:19:25", "content": "Hi @silantev! You have a really good point on why Kfold can be better on test prediction than on train! Also good luck for the competition! You are a really cool guy to help someone how is so close behind you :D
", "votes": null, "replies": [] } ] }, { "id": 608508, "author_name": "cpmpml", "author_url": "", "post_date": "08/26/2019 20:54:36", "content": "\n\n\nYou could think that Model2 is over-fitting
\n
Do you have any doubt about this?
\n\n\n\n\nModel2 is a bit more aggressively hyper-parameter optimized
\n
I often say that HPO is the best way I know to overfit. Thanks for providing a blatant example.
", "votes": null, "replies": [ { "id": 608541, "author_name": "cpmpml", "author_url": "", "post_date": "08/26/2019 22:48:46", "content": "Previous answer may sound sarcastic when it is not the intent. One rule I try to obey is to only change one thing at a time. This way, one can be sure about the root of an issue if there is an issue. I therefore suggest you create model 3 and model 4 that are model1 for features and model 2 for parameters, or vice versa. This way you'll know if HPO is the culprit (as I assumed it was, maybe erroneously), or if your features are the issue.
", "votes": null, "replies": [] }, { "id": 608769, "author_name": "optimo", "author_url": "", "post_date": "08/27/2019 06:46:46", "content": "@cpmpml long answer short you would trust the LB?
", "votes": null, "replies": [] }, { "id": 608804, "author_name": "cpmpml", "author_url": "", "post_date": "08/27/2019 07:18:00", "content": "I trust my cv, and it is very well correlated with my LB ;) I mean the cv lb gap is constant at least to 2nd decimal.
", "votes": null, "replies": [] }, { "id": 609305, "author_name": "edgarpe", "author_url": "", "post_date": "08/27/2019 15:16:16", "content": "Thanks @cpmpml for your answers! Before my CV and LB score was also correlated. I am still investigating, with two new models which are only different in parameters.
", "votes": null, "replies": [] }, { "id": 609457, "author_name": "edgarpe", "author_url": "", "post_date": "08/27/2019 18:32:47", "content": "> the cv lb gap is constant at least to 2nd decimal
\n\nYour CV or LB is better score? CV is better, right? I am asking because I made some more submissions and it looks like my CV is constantly around 0.2 better than LB.
", "votes": null, "replies": [] } ] }, { "id": 609491, "author_name": "kirkchen1993", "author_url": "", "post_date": "08/27/2019 19:26:58", "content": "I met the same case as your second one. \nOne of my results has -1.788 cv score but only -1.3 lb score.
\n\nThen I check it manually and found there are several very wried predictions in my submission file, e.g. np.amax(test_prediction)=1092976.625, which influence my lb score a lot.
\n\nI just did median stacking with other predictions to fix it.
", "votes": null, "replies": [] }, { "id": 610612, "author_name": "edgarpe", "author_url": "", "post_date": "08/28/2019 23:02:09", "content": "TLDR; It was overfit.
\n\n@cpmpml @silantev @kirkchen1993 Thanks guys for you replays, it meant a lot to me. I was thinking very hard what happened, because all the way through the competition my CV and LB score was correlated, LB was always better than CV. Than suddenly three days before finish LB started to be worse than CV.
\n\nWhat happened is really easy, honestly it's a rookie mistake. I thought K-fold split is enough, I always treated my CV score as a good measure of model performance. This is not the case. Before K-fold split I should have separated some data from train.csv to validate how well my model is performing. Or if not, I should have been submitted much more data to kaggle servers, to have my models validated for me. But none of these have been done, I just treated my CV score as a good measure.
\n\nI just realized all this 2 hours before competition finish :) But it bothered me a lot, and I am happy that is clear to me now 👍
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "608441": "Hi!\n\nI am lost here, I have two similar models with very different scores. Model2 is a bit more aggressively hyper-parameter optimized and a small amount of new features, than Model1.\n\nModel 1 CV: -1.53\nModel 1 LB about: -1.72\n\nModel 2 CV: -1.72\nModel 2 LB: -1.49\n\nYou could think that Model2 is over-fitting but training data is more than 4.600.000 records and leader board data is just around 730.000. It means for 1JHN only about 6500 records!\n\nBoth my models are LightGBM with Kfold of 5 folds.\n\nSo what happens here? Model2 performs better and LB dataset is too small, or Model2 is over-fitted?", "608473": "First of all, remember that you can use 2 submission as final ones.\nFirst case seems to be common situation for this competion because if you use 5 folds and only one split seed every prediction in your CV is a result of 1 model while the predictions in the submission is averaged result of 5 models. In this competioion averaging let boost significantly the performance of your model (and at my experience getting median works better than mean in this particular competion).\nI'm not sure about your Model 2 but I am inclined to believe that it's kind of mistake in preprocessing. It could be over-fitting, especially if you use early-stopping, the using of the one validation set for both early-stopping and evaluation can lead to over-fitting. But the difference is too big, so I'd check the pre-processing operations like Scalers and so on.", "608508": "> You could think that Model2 is over-fitting\n\nDo you have any doubt about this?\n\n> Model2 is a bit more aggressively hyper-parameter optimized \n\nI often say that HPO is the best way I know to overfit. Thanks for providing a blatant example.", "608541": "Previous answer may sound sarcastic when it is not the intent. One rule I try to obey is to only change one thing at a time. This way, one can be sure about the root of an issue if there is an issue. I therefore suggest you create model 3 and model 4 that are model1 for features and model 2 for parameters, or vice versa. This way you'll know if HPO is the culprit (as I assumed it was, maybe erroneously), or if your features are the issue.", "608769": "cpmpml long answer short you would trust the LB?", "608804": "I trust my cv, and it is very well correlated with my LB ;) I mean the cv lb gap is constant at least to 2nd decimal.", "609305": "Thanks @cpmpml for your answers! Before my CV and LB score was also correlated. I am still investigating, with two new models which are only different in parameters.", "609310": "Hi @silantev! You have a really good point on why Kfold can be better on test prediction than on train! Also good luck for the competition! You are a really cool guy to help someone how is so close behind you :D", "609457": "> the cv lb gap is constant at least to 2nd decimal\n\n\nYour CV or LB is better score? CV is better, right? I am asking because I made some more submissions and it looks like my CV is constantly around 0.2 better than LB.", "609491": "I met the same case as your second one. \nOne of my results has -1.788 cv score but only -1.3 lb score.\n\nThen I check it manually and found there are several very wried predictions in my submission file, e.g. np.amax(test_prediction)=1092976.625, which influence my lb score a lot.\n\nI just did median stacking with other predictions to fix it.", "610612": "TLDR; It was overfit.\n\n@cpmpml @silantev @kirkchen1993 Thanks guys for you replays, it meant a lot to me. I was thinking very hard what happened, because all the way through the competition my CV and LB score was correlated, LB was always better than CV. Than suddenly three days before finish LB started to be worse than CV.\n\nWhat happened is really easy, honestly it's a rookie mistake. I thought K-fold split is enough, I always treated my CV score as a good measure of model performance. This is not the case. Before K-fold split I should have separated some data from train.csv to validate how well my model is performing. Or if not, I should have been submitted much more data to kaggle servers, to have my models validated for me. But none of these have been done, I just treated my CV score as a good measure. \n\nI just realized all this 2 hours before competition finish :) But it bothered me a lot, and I am happy that is clear to me now 👍" }, "source": "meta" }