My CV is high around 1.57, receive a LB of about of 1.43
\n\nThen I improve CV to 1.55, LB go up to 1.44
\n\nThis is counter intuitive. I'm not doing anything sketchy either, I'm just doing large feature engineering to some public kernels.
\n\nAlso, is anyone else's CV a lot higher than the LB?
", "messages": [ { "id": "398461", "postDate": "10/04/2018 06:23:30", "content": "My CV is high around 1.57, receive a LB of about of 1.43
\n\nThen I improve CV to 1.55, LB go up to 1.44
\n\nThis is counter intuitive. I'm not doing anything sketchy either, I'm just doing large feature engineering to some public kernels.
\n\nAlso, is anyone else's CV a lot higher than the LB?
", "rawMarkdown": "My CV is high around 1.57, receive a LB of about of 1.43\n\nThen I improve CV to 1.55, LB go up to 1.44\n\nThis is counter intuitive. I'm not doing anything sketchy either, I'm just doing large feature engineering to some public kernels.\n\nAlso, is anyone else's CV a lot higher than the LB?", "votes": null }, { "id": "398474", "postDate": "10/04/2018 06:55:04", "content": "I am facing the same issues. Till now I can't figure out what's happening.
\n\nUser level model:\nAt start my model has cv score of 1.59 and lb was 1.44.
\n\nWhen cv was 1.5632 I got a lb of 1.4388.
\n\nThen decreased it to 1.5593 and lb comes as 1.4403.
\n\nFor user level I am using stratified k fold ( stratification using target > 0 and target < 0).
\n\nDo you have any other validation strategy for user level models ?
", "rawMarkdown": "I am facing the same issues. Till now I can't figure out what's happening. \n\nUser level model:\nAt start my model has cv score of 1.59 and lb was 1.44. \n\nWhen cv was 1.5632 I got a lb of 1.4388. \n\nThen decreased it to 1.5593 and lb comes as 1.4403.\n\nFor user level I am using stratified k fold ( stratification using target > 0 and target < 0).\n\nDo you have any other validation strategy for user level models ?", "votes": null }, { "id": "398733", "postDate": "10/04/2018 14:08:11", "content": "\n\nI have faced the same issue. I think CV is getting better, LB is getting worse !\nI think the issue is, the features we are creating, are differentiating train & test. So, the distribution of those features in train & test are not same.
\n\nMay be that's why they are beneficial in train data but worse in test data. \nBut if difference is not too high, we can trust CV.
", "rawMarkdown": "returnofsputnik,\n\nI have faced the same issue. I think CV is getting better, LB is getting worse !\nI think the issue is, the features we are creating, are differentiating train & test. So, the distribution of those features in train & test are not same. \n\nMay be that's why they are beneficial in train data but worse in test data. \nBut if difference is not too high, we can trust CV.", "votes": null }, { "id": "398734", "postDate": "10/04/2018 14:08:19", "content": "Hi Corey, the CV higher than LB seems to be the case for everybody, at least all public kernel that I saw.
\n\nFor the LB / CV moving in different directly a of people including me are facing this issue. But also some people say that it's not the case for them, and they managed to have some improvement in CV that translate in LB.
\n\nA simple example is the \"month\" feature, this seems to clearly overfit the training set and doesn't improve the LB. I have a lot of example like that where the feature improve my 5fold CV but not LB.
", "rawMarkdown": "Hi Corey, the CV higher than LB seems to be the case for everybody, at least all public kernel that I saw.\n\nFor the LB / CV moving in different directly a of people including me are facing this issue. But also some people say that it's not the case for them, and they managed to have some improvement in CV that translate in LB.\n\nA simple example is the \"month\" feature, this seems to clearly overfit the training set and doesn't improve the LB. I have a lot of example like that where the feature improve my 5fold CV but not LB.", "votes": null }, { "id": "398736", "postDate": "10/04/2018 14:09:26", "content": "@suchith0312 give a try to GroupKFold. As @ogrellier says, it avoid leakage.
", "rawMarkdown": "suchith0312 give a try to GroupKFold. As @ogrellier says, it avoid leakage.", "votes": null }, { "id": "398740", "postDate": "10/04/2018 14:14:19", "content": "May I ask was your 'large feature engineering' made mostly by totals.hits and totals.pageviews features or others?
", "rawMarkdown": "May I ask was your 'large feature engineering' made mostly by totals.hits and totals.pageviews features or others?", "votes": null }, { "id": "398744", "postDate": "10/04/2018 14:23:30", "content": "I use groupKfold but occur leakage
", "rawMarkdown": "I use groupKfold but occur leakage", "votes": null }, { "id": "398746", "postDate": "10/04/2018 14:27:16", "content": "I was talking about user level models. For user-level models, I think we don't need to use groupkfold.
", "rawMarkdown": "I was talking about user level models. For user-level models, I think we don't need to use groupkfold.", "votes": null }, { "id": "398748", "postDate": "10/04/2018 14:27:59", "content": "Yeah, My features include other stuff, but focuses on totals.pageviews and totals.hits. Is this creating a problem ?
", "rawMarkdown": "Yeah, My features include other stuff, but focuses on totals.pageviews and totals.hits. Is this creating a problem ?", "votes": null }, { "id": "398752", "postDate": "10/04/2018 14:36:33", "content": "Hi Prashant, how do you do your features selection in this case ? do you validate all your CV improvements with a submission ?
", "rawMarkdown": "Hi Prashant, how do you do your features selection in this case ? do you validate all your CV improvements with a submission ?", "votes": null }, { "id": "398790", "postDate": "10/04/2018 15:32:52", "content": "I have the same sort of issues with a huge CV decrease and an LB increase.
\n\nI then decided to split train in 2 folds (6 months each to sort of reproduce train/test) and the CV decrease in the first fold is reflected in the 2nd fold.
\n\nI'm not doing anything special and absolutely not using stuff in the 2nd fold (i.e. data leakage). Not sure what's going on here...
", "rawMarkdown": "I have the same sort of issues with a huge CV decrease and an LB increase.\n\nI then decided to split train in 2 folds (6 months each to sort of reproduce train/test) and the CV decrease in the first fold is reflected in the 2nd fold.\n\nI'm not doing anything special and absolutely not using stuff in the 2nd fold (i.e. data leakage). Not sure what's going on here...", "votes": null }, { "id": "398792", "postDate": "10/04/2018 15:33:28", "content": "I don't know but I'm very suspicious about them since this.
", "rawMarkdown": "I don't know but I'm very suspicious about them since [this][1].\n\n\n [1]: http://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66774#397417", "votes": null }, { "id": "398816", "postDate": "10/04/2018 16:13:46", "content": "same here. i gave up on building any sorts of cv. i think the problem is we have 1 year data to predict next 6 months. it is very hard to set up a reliable time split cv. i tried to understand how LB is structured back then, phil told me kaggle has spent lots of time set up a realiable LB. i hope if LB is stratified by fullsivistorID and by their revenue, then LB will be my cv..
", "rawMarkdown": "same here. i gave up on building any sorts of cv. i think the problem is we have 1 year data to predict next 6 months. it is very hard to set up a reliable time split cv. i tried to understand how LB is structured back then, phil told me kaggle has spent lots of time set up a realiable LB. i hope if LB is stratified by fullsivistorID and by their revenue, then LB will be my cv..", "votes": null }, { "id": "398819", "postDate": "10/04/2018 16:16:23", "content": "@fatihozturk large feat eng. has been stuff not mentioned in any kernels or discussions as far as Im aware. It is stuff Ive independently thought of, and I have not done any feature engineering with totals.hits or totals.pageviews
", "rawMarkdown": "fatihozturk large feat eng. has been stuff not mentioned in any kernels or discussions as far as Im aware. It is stuff Ive independently thought of, and I have not done any feature engineering with totals.hits or totals.pageviews", "votes": null }, { "id": "398820", "postDate": "10/04/2018 16:18:48", "content": "@suchith0312 i am using just kfold as well, not stratified because i dont know how to stratify it, unless you stratify it by 0/1 purchased or not. And i agree, there is no leakage in a user-level model
", "rawMarkdown": "suchith0312 i am using just kfold as well, not stratified because i dont know how to stratify it, unless you stratify it by 0/1 purchased or not. And i agree, there is no leakage in a user-level model", "votes": null }, { "id": "398842", "postDate": "10/04/2018 16:46:08", "content": "\n\nYep, \nIf I add some features, and CV/LB score got worse, remove that feature.
\n\nIf CV/LB goes up, keep it.
\n\nIf CV get better, LB worse a bit, keep it.
\n\nIf CV get worse, LB better, there may be some time related information included, so, normalize that feature and keep it.
", "rawMarkdown": "areveillon, \n\nYep, \nIf I add some features, and CV/LB score got worse, remove that feature.\n\nIf CV/LB goes up, keep it.\n\nIf CV get better, LB worse a bit, keep it.\n\nIf CV get worse, LB better, there may be some time related information included, so, normalize that feature and keep it.", "votes": null }, { "id": "398849", "postDate": "10/04/2018 17:08:27", "content": "I agree with Shubin. CV going up and LB going down may just mean we overfit train less and generalization gets better. I've seen people using month but this would just help overfitting training data without any evidence of generalization capabilities.
\n\nAlso the amount of zeros probably increases the number of duplicate samples (on particular features) with different target values, which implies the model will try to average everything and a simple outlier predicted a bit worse may increase LB or CV score.
", "rawMarkdown": "I agree with Shubin. CV going up and LB going down may just mean we overfit train less and generalization gets better. I've seen people using month but this would just help overfitting training data without any evidence of generalization capabilities.\n\nAlso the amount of zeros probably increases the number of duplicate samples (on particular features) with different target values, which implies the model will try to average everything and a simple outlier predicted a bit worse may increase LB or CV score.", "votes": null }, { "id": "398918", "postDate": "10/04/2018 19:55:22", "content": "The only reason I could see to explain those differences would be some kind of periodical events that create different user behaviors between train and test. For instance :
\n\nTime data (month week of year etc) seems to clearly overfit. Some traffic source as well, I found one that had very high rate of revenue, but removing it from the data actually improved my LB and decreased my CV.
", "rawMarkdown": "The only reason I could see to explain those differences would be some kind of periodical events that create different user behaviors between train and test. For instance :\n\n - the release of a new product\n - a special promotional coupon from a specific source, and for a specific period of time\n - a new advertisement that was ran only for a short period of time\n - some marketing email campaign\n\nTime data (month week of year etc) seems to clearly overfit. Some traffic source as well, I found one that had very high rate of revenue, but removing it from the data actually improved my LB and decreased my CV.", "votes": null }, { "id": "398925", "postDate": "10/04/2018 20:14:13", "content": "alright thanks ! That's what I was doing, but it requires one submission per feature so I was not very happy with that !
", "rawMarkdown": "alright thanks ! That's what I was doing, but it requires one submission per feature so I was not very happy with that !", "votes": null }, { "id": "398983", "postDate": "10/04/2018 22:58:18", "content": "I did quite the same, I have a validation holdout of 3 months. I see consistent change in cv between the two folds. But not in LB.
", "rawMarkdown": "I did quite the same, I have a validation holdout of 3 months. I see consistent change in cv between the two folds. But not in LB.", "votes": null }, { "id": "399082", "postDate": "10/05/2018 06:34:46", "content": "It's possible you're feature engineering consists of Data Leakage, maybe build a few partial dependency plots, and see if the correlations are too high on any of your features?
", "rawMarkdown": "It's possible you're feature engineering consists of Data Leakage, maybe build a few partial dependency plots, and see if the correlations are too high on any of your features?", "votes": null }, { "id": "399325", "postDate": "10/05/2018 15:54:14", "content": "Could you please elaborate on why month or alike would allow to overfit to the data? It seems to be obvious to many, but not to me.
I use stratified on purchased or not to make sure all folds have a reasonable (=same as it train) fraction of purchases. Otherwise there can be by chance a fold with no renevue
\n\nBut I have the same problem here- LB and CV going in opposite directions. My highest LB was a basic user-level model with no feature engineering other than usual aggregations over users :D
", "rawMarkdown": "I use stratified on purchased or not to make sure all folds have a reasonable (=same as it train) fraction of purchases. Otherwise there can be by chance a fold with no renevue\n\nBut I have the same problem here- LB and CV going in opposite directions. My highest LB was a basic user-level model with no feature engineering other than usual aggregations over users :D", "votes": null }, { "id": "399368", "postDate": "10/05/2018 17:11:46", "content": "To detect seasonality inside one period like a year you need more than one year of data that we don't have.
\n\nIf you detect a month related pattern, easy silly example, say that in february there's a lot of buying of calendars that have 29 days. Then you apply that pattern in next year... but unlike train, test set happen's not to be a leap year ---> you overfit.
\n\n(Of course you could still find useful month related patterns, it all depends on what the ground truth is about that possible seasonality)
", "rawMarkdown": "To detect seasonality inside one period like a year you need more than one year of data that we don't have. \n\nIf you detect a month related pattern, easy silly example, say that in february there's a lot of buying of calendars that have 29 days. Then you apply that pattern in next year... but unlike train, test set happen's not to be a leap year ---> you overfit. \n\n(Of course you could still find useful month related patterns, it all depends on what the ground truth is about that possible seasonality)", "votes": null }, { "id": "399537", "postDate": "10/06/2018 03:47:25", "content": "...
", "rawMarkdown": "...", "votes": null } ], "comments": [ { "id": 398474, "author_name": "suchith0312", "author_url": "", "post_date": "10/04/2018 06:55:04", "content": "I am facing the same issues. Till now I can't figure out what's happening.
\n\nUser level model:\nAt start my model has cv score of 1.59 and lb was 1.44.
\n\nWhen cv was 1.5632 I got a lb of 1.4388.
\n\nThen decreased it to 1.5593 and lb comes as 1.4403.
\n\nFor user level I am using stratified k fold ( stratification using target > 0 and target < 0).
\n\nDo you have any other validation strategy for user level models ?
", "votes": null, "replies": [ { "id": 398736, "author_name": "prashantkikani", "author_url": "", "post_date": "10/04/2018 14:09:26", "content": "@suchith0312 give a try to GroupKFold. As @ogrellier says, it avoid leakage.
", "votes": null, "replies": [] }, { "id": 398744, "author_name": "chocozzz", "author_url": "", "post_date": "10/04/2018 14:23:30", "content": "I use groupKfold but occur leakage
", "votes": null, "replies": [] }, { "id": 398746, "author_name": "suchith0312", "author_url": "", "post_date": "10/04/2018 14:27:16", "content": "I was talking about user level models. For user-level models, I think we don't need to use groupkfold.
", "votes": null, "replies": [] }, { "id": 398820, "author_name": "returnofsputnik", "author_url": "", "post_date": "10/04/2018 16:18:48", "content": "@suchith0312 i am using just kfold as well, not stratified because i dont know how to stratify it, unless you stratify it by 0/1 purchased or not. And i agree, there is no leakage in a user-level model
", "votes": null, "replies": [] }, { "id": 399329, "author_name": "mlisovyi", "author_url": "", "post_date": "10/05/2018 16:05:07", "content": "I use stratified on purchased or not to make sure all folds have a reasonable (=same as it train) fraction of purchases. Otherwise there can be by chance a fold with no renevue
\n\nBut I have the same problem here- LB and CV going in opposite directions. My highest LB was a basic user-level model with no feature engineering other than usual aggregations over users :D
", "votes": null, "replies": [] } ] }, { "id": 398733, "author_name": "prashantkikani", "author_url": "", "post_date": "10/04/2018 14:08:11", "content": "\n\nI have faced the same issue. I think CV is getting better, LB is getting worse !\nI think the issue is, the features we are creating, are differentiating train & test. So, the distribution of those features in train & test are not same.
\n\nMay be that's why they are beneficial in train data but worse in test data. \nBut if difference is not too high, we can trust CV.
", "votes": null, "replies": [ { "id": 398752, "author_name": "areveillon", "author_url": "", "post_date": "10/04/2018 14:36:33", "content": "Hi Prashant, how do you do your features selection in this case ? do you validate all your CV improvements with a submission ?
", "votes": null, "replies": [] }, { "id": 398842, "author_name": "prashantkikani", "author_url": "", "post_date": "10/04/2018 16:46:08", "content": "\n\nYep, \nIf I add some features, and CV/LB score got worse, remove that feature.
\n\nIf CV/LB goes up, keep it.
\n\nIf CV get better, LB worse a bit, keep it.
\n\nIf CV get worse, LB better, there may be some time related information included, so, normalize that feature and keep it.
", "votes": null, "replies": [] }, { "id": 398925, "author_name": "areveillon", "author_url": "", "post_date": "10/04/2018 20:14:13", "content": "alright thanks ! That's what I was doing, but it requires one submission per feature so I was not very happy with that !
", "votes": null, "replies": [] } ] }, { "id": 398734, "author_name": "areveillon", "author_url": "", "post_date": "10/04/2018 14:08:19", "content": "Hi Corey, the CV higher than LB seems to be the case for everybody, at least all public kernel that I saw.
\n\nFor the LB / CV moving in different directly a of people including me are facing this issue. But also some people say that it's not the case for them, and they managed to have some improvement in CV that translate in LB.
\n\nA simple example is the \"month\" feature, this seems to clearly overfit the training set and doesn't improve the LB. I have a lot of example like that where the feature improve my 5fold CV but not LB.
", "votes": null, "replies": [ { "id": 399325, "author_name": "mlisovyi", "author_url": "", "post_date": "10/05/2018 15:54:14", "content": "Could you please elaborate on why month or alike would allow to overfit to the data? It seems to be obvious to many, but not to me.
To detect seasonality inside one period like a year you need more than one year of data that we don't have.
\n\nIf you detect a month related pattern, easy silly example, say that in february there's a lot of buying of calendars that have 29 days. Then you apply that pattern in next year... but unlike train, test set happen's not to be a leap year ---> you overfit.
\n\n(Of course you could still find useful month related patterns, it all depends on what the ground truth is about that possible seasonality)
", "votes": null, "replies": [] } ] }, { "id": 398740, "author_name": "fatihozturk", "author_url": "", "post_date": "10/04/2018 14:14:19", "content": "May I ask was your 'large feature engineering' made mostly by totals.hits and totals.pageviews features or others?
", "votes": null, "replies": [ { "id": 398748, "author_name": "suchith0312", "author_url": "", "post_date": "10/04/2018 14:27:59", "content": "Yeah, My features include other stuff, but focuses on totals.pageviews and totals.hits. Is this creating a problem ?
", "votes": null, "replies": [] }, { "id": 398792, "author_name": "fatihozturk", "author_url": "", "post_date": "10/04/2018 15:33:28", "content": "I don't know but I'm very suspicious about them since this.
", "votes": null, "replies": [] }, { "id": 398819, "author_name": "returnofsputnik", "author_url": "", "post_date": "10/04/2018 16:16:23", "content": "@fatihozturk large feat eng. has been stuff not mentioned in any kernels or discussions as far as Im aware. It is stuff Ive independently thought of, and I have not done any feature engineering with totals.hits or totals.pageviews
", "votes": null, "replies": [] }, { "id": 398849, "author_name": "ogrellier", "author_url": "", "post_date": "10/04/2018 17:08:27", "content": "I agree with Shubin. CV going up and LB going down may just mean we overfit train less and generalization gets better. I've seen people using month but this would just help overfitting training data without any evidence of generalization capabilities.
\n\nAlso the amount of zeros probably increases the number of duplicate samples (on particular features) with different target values, which implies the model will try to average everything and a simple outlier predicted a bit worse may increase LB or CV score.
", "votes": null, "replies": [] } ] }, { "id": 398790, "author_name": "ogrellier", "author_url": "", "post_date": "10/04/2018 15:32:52", "content": "I have the same sort of issues with a huge CV decrease and an LB increase.
\n\nI then decided to split train in 2 folds (6 months each to sort of reproduce train/test) and the CV decrease in the first fold is reflected in the 2nd fold.
\n\nI'm not doing anything special and absolutely not using stuff in the 2nd fold (i.e. data leakage). Not sure what's going on here...
", "votes": null, "replies": [ { "id": 398918, "author_name": "areveillon", "author_url": "", "post_date": "10/04/2018 19:55:22", "content": "The only reason I could see to explain those differences would be some kind of periodical events that create different user behaviors between train and test. For instance :
\n\nTime data (month week of year etc) seems to clearly overfit. Some traffic source as well, I found one that had very high rate of revenue, but removing it from the data actually improved my LB and decreased my CV.
", "votes": null, "replies": [] }, { "id": 398983, "author_name": "tita1708", "author_url": "", "post_date": "10/04/2018 22:58:18", "content": "I did quite the same, I have a validation holdout of 3 months. I see consistent change in cv between the two folds. But not in LB.
", "votes": null, "replies": [] } ] }, { "id": 398816, "author_name": "lishubin", "author_url": "", "post_date": "10/04/2018 16:13:46", "content": "same here. i gave up on building any sorts of cv. i think the problem is we have 1 year data to predict next 6 months. it is very hard to set up a reliable time split cv. i tried to understand how LB is structured back then, phil told me kaggle has spent lots of time set up a realiable LB. i hope if LB is stratified by fullsivistorID and by their revenue, then LB will be my cv..
", "votes": null, "replies": [] }, { "id": 399082, "author_name": "jackdags", "author_url": "", "post_date": "10/05/2018 06:34:46", "content": "It's possible you're feature engineering consists of Data Leakage, maybe build a few partial dependency plots, and see if the correlations are too high on any of your features?
", "votes": null, "replies": [] }, { "id": 399537, "author_name": "evanren", "author_url": "", "post_date": "10/06/2018 03:47:25", "content": "...
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "398461": "My CV is high around 1.57, receive a LB of about of 1.43\n\nThen I improve CV to 1.55, LB go up to 1.44\n\nThis is counter intuitive. I'm not doing anything sketchy either, I'm just doing large feature engineering to some public kernels.\n\nAlso, is anyone else's CV a lot higher than the LB?", "398474": "I am facing the same issues. Till now I can't figure out what's happening. \n\nUser level model:\nAt start my model has cv score of 1.59 and lb was 1.44. \n\nWhen cv was 1.5632 I got a lb of 1.4388. \n\nThen decreased it to 1.5593 and lb comes as 1.4403.\n\nFor user level I am using stratified k fold ( stratification using target > 0 and target < 0).\n\nDo you have any other validation strategy for user level models ?", "398733": "returnofsputnik,\n\nI have faced the same issue. I think CV is getting better, LB is getting worse !\nI think the issue is, the features we are creating, are differentiating train & test. So, the distribution of those features in train & test are not same. \n\nMay be that's why they are beneficial in train data but worse in test data. \nBut if difference is not too high, we can trust CV.", "398734": "Hi Corey, the CV higher than LB seems to be the case for everybody, at least all public kernel that I saw.\n\nFor the LB / CV moving in different directly a of people including me are facing this issue. But also some people say that it's not the case for them, and they managed to have some improvement in CV that translate in LB.\n\nA simple example is the \"month\" feature, this seems to clearly overfit the training set and doesn't improve the LB. I have a lot of example like that where the feature improve my 5fold CV but not LB.", "398736": "suchith0312 give a try to GroupKFold. As @ogrellier says, it avoid leakage.", "398740": "May I ask was your 'large feature engineering' made mostly by totals.hits and totals.pageviews features or others?", "398744": "I use groupKfold but occur leakage", "398746": "I was talking about user level models. For user-level models, I think we don't need to use groupkfold.", "398748": "Yeah, My features include other stuff, but focuses on totals.pageviews and totals.hits. Is this creating a problem ?", "398752": "Hi Prashant, how do you do your features selection in this case ? do you validate all your CV improvements with a submission ?", "398790": "I have the same sort of issues with a huge CV decrease and an LB increase.\n\nI then decided to split train in 2 folds (6 months each to sort of reproduce train/test) and the CV decrease in the first fold is reflected in the 2nd fold.\n\nI'm not doing anything special and absolutely not using stuff in the 2nd fold (i.e. data leakage). Not sure what's going on here...", "398792": "I don't know but I'm very suspicious about them since [this][1].\n\n\n [1]: http://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66774#397417", "398816": "same here. i gave up on building any sorts of cv. i think the problem is we have 1 year data to predict next 6 months. it is very hard to set up a reliable time split cv. i tried to understand how LB is structured back then, phil told me kaggle has spent lots of time set up a realiable LB. i hope if LB is stratified by fullsivistorID and by their revenue, then LB will be my cv..", "398819": "fatihozturk large feat eng. has been stuff not mentioned in any kernels or discussions as far as Im aware. It is stuff Ive independently thought of, and I have not done any feature engineering with totals.hits or totals.pageviews", "398820": "suchith0312 i am using just kfold as well, not stratified because i dont know how to stratify it, unless you stratify it by 0/1 purchased or not. And i agree, there is no leakage in a user-level model", "398842": "areveillon, \n\nYep, \nIf I add some features, and CV/LB score got worse, remove that feature.\n\nIf CV/LB goes up, keep it.\n\nIf CV get better, LB worse a bit, keep it.\n\nIf CV get worse, LB better, there may be some time related information included, so, normalize that feature and keep it.", "398849": "I agree with Shubin. CV going up and LB going down may just mean we overfit train less and generalization gets better. I've seen people using month but this would just help overfitting training data without any evidence of generalization capabilities.\n\nAlso the amount of zeros probably increases the number of duplicate samples (on particular features) with different target values, which implies the model will try to average everything and a simple outlier predicted a bit worse may increase LB or CV score.", "398918": "The only reason I could see to explain those differences would be some kind of periodical events that create different user behaviors between train and test. For instance :\n\n - the release of a new product\n - a special promotional coupon from a specific source, and for a specific period of time\n - a new advertisement that was ran only for a short period of time\n - some marketing email campaign\n\nTime data (month week of year etc) seems to clearly overfit. Some traffic source as well, I found one that had very high rate of revenue, but removing it from the data actually improved my LB and decreased my CV.", "398925": "alright thanks ! That's what I was doing, but it requires one submission per feature so I was not very happy with that !", "398983": "I did quite the same, I have a validation holdout of 3 months. I see consistent change in cv between the two folds. But not in LB.", "399082": "It's possible you're feature engineering consists of Data Leakage, maybe build a few partial dependency plots, and see if the correlations are too high on any of your features?", "399325": "Could you please elaborate on why `month` or alike would allow to overfit to the data? It seems to be obvious to many, but not to me.", "399329": "I use stratified on purchased or not to make sure all folds have a reasonable (=same as it train) fraction of purchases. Otherwise there can be by chance a fold with no renevue\n\nBut I have the same problem here- LB and CV going in opposite directions. My highest LB was a basic user-level model with no feature engineering other than usual aggregations over users :D", "399368": "To detect seasonality inside one period like a year you need more than one year of data that we don't have. \n\nIf you detect a month related pattern, easy silly example, say that in february there's a lot of buying of calendars that have 29 days. Then you apply that pattern in next year... but unlike train, test set happen's not to be a leap year ---> you overfit. \n\n(Of course you could still find useful month related patterns, it all depends on what the ground truth is about that possible seasonality)", "399537": "..." }, "source": "meta" }