In the past few days I have used group k-fold to test my model. However, decrease on CV doesn't translate onto PB. So today I turned to use time-based validation, but again CV and PB are not in the same direction.
\n\nHere are my CV and PB:
\n\nsimple group k-fold:\nCV 1.6102 -> PB 1.4384 and\nCV 1.5874 -> PB 1.4406
\n\nsimple time-based:\nCV 1.6879 -> PB 1.4471 and\nCV 1.6768 -> PB 1.4531
\n\nI feel confused and don't know what to do. Any suggestion please ?
", "messages": [ { "id": "399244", "postDate": "10/05/2018 13:28:22", "content": "In the past few days I have used group k-fold to test my model. However, decrease on CV doesn't translate onto PB. So today I turned to use time-based validation, but again CV and PB are not in the same direction.
\n\nHere are my CV and PB:
\n\nsimple group k-fold:\nCV 1.6102 -> PB 1.4384 and\nCV 1.5874 -> PB 1.4406
\n\nsimple time-based:\nCV 1.6879 -> PB 1.4471 and\nCV 1.6768 -> PB 1.4531
\n\nI feel confused and don't know what to do. Any suggestion please ?
", "rawMarkdown": "In the past few days I have used group k-fold to test my model. However, decrease on CV doesn't translate onto PB. So today I turned to use time-based validation, but again CV and PB are not in the same direction.\n\nHere are my CV and PB:\n\nsimple group k-fold:\nCV 1.6102 -> PB 1.4384 and\nCV 1.5874 -> PB 1.4406\n\nsimple time-based:\nCV 1.6879 -> PB 1.4471 and\nCV 1.6768 -> PB 1.4531\n\nI feel confused and don't know what to do. Any suggestion please ?", "votes": null }, { "id": "399285", "postDate": "10/05/2018 14:46:20", "content": "Hi, i would guess that you are probably overfitting, i use groupkfold and get extremely consistent difference between cv and lb.\nTry to drop some time based features and hits/page views ratio that i found to be overfitting to train data
", "rawMarkdown": "Hi, i would guess that you are probably overfitting, i use groupkfold and get extremely consistent difference between cv and lb.\nTry to drop some time based features and hits/page views ratio that i found to be overfitting to train data", "votes": null }, { "id": "399347", "postDate": "10/05/2018 16:32:34", "content": "Thanks for your sharing. I have a question. How did you know that some time based features and hits/page views ratio are causing overfitting?
\n\nDid you check the difference of performances with/without these features?
", "rawMarkdown": "Thanks for your sharing. I have a question. How did you know that some time based features and hits/page views ratio are causing overfitting?\n\nDid you check the difference of performances with/without these features?", "votes": null }, { "id": "399360", "postDate": "10/05/2018 17:01:43", "content": "From my testing....After adding hits/pageviews ratio...The CV improve a lot while the LB drop around 0.01....
", "rawMarkdown": "From my testing....After adding hits/pageviews ratio...The CV improve a lot while the LB drop around 0.01....", "votes": null }, { "id": "399372", "postDate": "10/05/2018 17:18:24", "content": "Hi @YouHan,\nYes, i checked with and without.\nI will be more specific, using 'Day Of Month', 'Week Of Year' , and 'Hits\\Pageviews' have caused better CV and worse LB.\nAfter removing them, all my FE always kept very consistent LB-CV difference
", "rawMarkdown": "Hi @YouHan,\nYes, i checked with and without.\nI will be more specific, using 'Day Of Month', 'Week Of Year' , and 'Hits\\Pageviews' have caused better CV and worse LB.\nAfter removing them, all my FE always kept very consistent LB-CV difference", "votes": null }, { "id": "399382", "postDate": "10/05/2018 17:31:21", "content": "@AmirH. I appreciate the sharing of your experience. I understand what you did! I'm testing now to see the effect of removal of those features. If it works, I will share you!
", "rawMarkdown": "AmirH. I appreciate the sharing of your experience. I understand what you did! I'm testing now to see the effect of removal of those features. If it works, I will share you!", "votes": null }, { "id": "399384", "postDate": "10/05/2018 17:32:34", "content": "@Justin. Thanks for sharing!
", "rawMarkdown": "Justin. Thanks for sharing!", "votes": null }, { "id": "399533", "postDate": "10/06/2018 03:36:53", "content": "@AmirH. As testing the effect of time features and hits/pageviews, I got 0.002 boost!. Thanks for sharing!
", "rawMarkdown": "AmirH. As testing the effect of time features and hits/pageviews, I got 0.002 boost!. Thanks for sharing!", "votes": null }, { "id": "399553", "postDate": "10/06/2018 04:52:00", "content": "@AmirH. Thanks for your suggest, I will go to a try. But why hits/page views ratio causes overfit, the feature looks logical？
", "rawMarkdown": "AmirH. Thanks for your suggest, I will go to a try. But why hits/page views ratio causes overfit, the feature looks logical？", "votes": null }, { "id": "399554", "postDate": "10/06/2018 04:53:24", "content": "@Justin @YouHan thanks for your sharing!
", "rawMarkdown": "Justin @YouHan thanks for your sharing!", "votes": null }, { "id": "399581", "postDate": "10/06/2018 06:20:32", "content": "@YouHan great to hear!\nMay i ask how much boost does the categorical FE you added to oliviers kernel give you?
", "rawMarkdown": "YouHan great to hear!\nMay i ask how much boost does the categorical FE you added to oliviers kernel give you?", "votes": null }, { "id": "399582", "postDate": "10/06/2018 06:23:22", "content": "Hi @ethan.\nThis ratio is very significant in train set as shown in the specific discussion, but i suspect this behavior does not occur in test set for some reason, at least not in public LB set.\nThis is odd, but it is what it is
", "rawMarkdown": "Hi @ethan.\nThis ratio is very significant in train set as shown in the specific discussion, but i suspect this behavior does not occur in test set for some reason, at least not in public LB set.\nThis is odd, but it is what it is", "votes": null }, { "id": "399589", "postDate": "10/06/2018 07:07:23", "content": "@AmirH Actually, I added many categorical features, so it's not easy to say boost in exact number. Sorry for that!\nBut, for now, most of my features are categorical features using mean and frequency encoding.!
", "rawMarkdown": "AmirH Actually, I added many categorical features, so it's not easy to say boost in exact number. Sorry for that!\nBut, for now, most of my features are categorical features using mean and frequency encoding.!", "votes": null }, { "id": "399604", "postDate": "10/06/2018 08:14:51", "content": "time feature delete better?? i delete below code then LB & CV get worse
\n\n\nfor df in [train, test]:
\n df['date'] = pd.to_datetime(df['visitStartTime'], unit='s')
\n df['sess_date_dow'] = df['date'].dt.dayofweek
\n df['sess_date_hours'] = df['date'].dt.hour
\n df['sess_date_dom'] = df['date'].dt.day
\n
Really?\nI removed features from.day, month and hit/visit and then got a boost.
", "rawMarkdown": "Really?\nI removed features from.day, month and hit/visit and then got a boost.", "votes": null }, { "id": "399628", "postDate": "10/06/2018 09:58:33", "content": "Hi all,
\n\nSeems removing time related features and hit/ratio improves LB, but be aware of overfitting LB, as final score is evaluated by PB...
", "rawMarkdown": "Hi all, \n\nSeems removing time related features and hit/ratio improves LB, but be aware of overfitting LB, as final score is evaluated by PB...", "votes": null }, { "id": "399772", "postDate": "10/06/2018 18:13:31", "content": "+1 same for me
", "rawMarkdown": "1 same for me", "votes": null } ], "comments": [ { "id": 399285, "author_name": "tpthegreat", "author_url": "", "post_date": "10/05/2018 14:46:20", "content": "Hi, i would guess that you are probably overfitting, i use groupkfold and get extremely consistent difference between cv and lb.\nTry to drop some time based features and hits/page views ratio that i found to be overfitting to train data
", "votes": null, "replies": [ { "id": 399347, "author_name": "youhanlee", "author_url": "", "post_date": "10/05/2018 16:32:34", "content": "Thanks for your sharing. I have a question. How did you know that some time based features and hits/page views ratio are causing overfitting?
\n\nDid you check the difference of performances with/without these features?
", "votes": null, "replies": [] }, { "id": 399360, "author_name": "reputation18", "author_url": "", "post_date": "10/05/2018 17:01:43", "content": "From my testing....After adding hits/pageviews ratio...The CV improve a lot while the LB drop around 0.01....
", "votes": null, "replies": [] }, { "id": 399372, "author_name": "tpthegreat", "author_url": "", "post_date": "10/05/2018 17:18:24", "content": "Hi @YouHan,\nYes, i checked with and without.\nI will be more specific, using 'Day Of Month', 'Week Of Year' , and 'Hits\\Pageviews' have caused better CV and worse LB.\nAfter removing them, all my FE always kept very consistent LB-CV difference
", "votes": null, "replies": [] }, { "id": 399382, "author_name": "youhanlee", "author_url": "", "post_date": "10/05/2018 17:31:21", "content": "@AmirH. I appreciate the sharing of your experience. I understand what you did! I'm testing now to see the effect of removal of those features. If it works, I will share you!
", "votes": null, "replies": [] }, { "id": 399384, "author_name": "youhanlee", "author_url": "", "post_date": "10/05/2018 17:32:34", "content": "@Justin. Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 399533, "author_name": "youhanlee", "author_url": "", "post_date": "10/06/2018 03:36:53", "content": "@AmirH. As testing the effect of time features and hits/pageviews, I got 0.002 boost!. Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 399553, "author_name": "zhengyichen", "author_url": "", "post_date": "10/06/2018 04:52:00", "content": "@AmirH. Thanks for your suggest, I will go to a try. But why hits/page views ratio causes overfit, the feature looks logical？
", "votes": null, "replies": [] }, { "id": 399554, "author_name": "zhengyichen", "author_url": "", "post_date": "10/06/2018 04:53:24", "content": "@Justin @YouHan thanks for your sharing!
", "votes": null, "replies": [] }, { "id": 399581, "author_name": "tpthegreat", "author_url": "", "post_date": "10/06/2018 06:20:32", "content": "@YouHan great to hear!\nMay i ask how much boost does the categorical FE you added to oliviers kernel give you?
", "votes": null, "replies": [] }, { "id": 399582, "author_name": "tpthegreat", "author_url": "", "post_date": "10/06/2018 06:23:22", "content": "Hi @ethan.\nThis ratio is very significant in train set as shown in the specific discussion, but i suspect this behavior does not occur in test set for some reason, at least not in public LB set.\nThis is odd, but it is what it is
", "votes": null, "replies": [] }, { "id": 399589, "author_name": "youhanlee", "author_url": "", "post_date": "10/06/2018 07:07:23", "content": "@AmirH Actually, I added many categorical features, so it's not easy to say boost in exact number. Sorry for that!\nBut, for now, most of my features are categorical features using mean and frequency encoding.!
", "votes": null, "replies": [] }, { "id": 399604, "author_name": "chocozzz", "author_url": "", "post_date": "10/06/2018 08:14:51", "content": "time feature delete better?? i delete below code then LB & CV get worse
\n\n\nfor df in [train, test]:
\n df['date'] = pd.to_datetime(df['visitStartTime'], unit='s')
\n df['sess_date_dow'] = df['date'].dt.dayofweek
\n df['sess_date_hours'] = df['date'].dt.hour
\n df['sess_date_dom'] = df['date'].dt.day
\n
Really?\nI removed features from.day, month and hit/visit and then got a boost.
", "votes": null, "replies": [] }, { "id": 399628, "author_name": "joe1995", "author_url": "", "post_date": "10/06/2018 09:58:33", "content": "Hi all,
\n\nSeems removing time related features and hit/ratio improves LB, but be aware of overfitting LB, as final score is evaluated by PB...
", "votes": null, "replies": [] }, { "id": 399772, "author_name": "xavierbourretsicotte", "author_url": "", "post_date": "10/06/2018 18:13:31", "content": "+1 same for me
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "399244": "In the past few days I have used group k-fold to test my model. However, decrease on CV doesn't translate onto PB. So today I turned to use time-based validation, but again CV and PB are not in the same direction.\n\nHere are my CV and PB:\n\nsimple group k-fold:\nCV 1.6102 -> PB 1.4384 and\nCV 1.5874 -> PB 1.4406\n\nsimple time-based:\nCV 1.6879 -> PB 1.4471 and\nCV 1.6768 -> PB 1.4531\n\nI feel confused and don't know what to do. Any suggestion please ?", "399285": "Hi, i would guess that you are probably overfitting, i use groupkfold and get extremely consistent difference between cv and lb.\nTry to drop some time based features and hits/page views ratio that i found to be overfitting to train data", "399347": "Thanks for your sharing. I have a question. How did you know that some time based features and hits/page views ratio are causing overfitting?\n\nDid you check the difference of performances with/without these features?", "399360": "From my testing....After adding hits/pageviews ratio...The CV improve a lot while the LB drop around 0.01....", "399372": "Hi @YouHan,\nYes, i checked with and without.\nI will be more specific, using 'Day Of Month', 'Week Of Year' , and 'Hits\\Pageviews' have caused better CV and worse LB.\nAfter removing them, all my FE always kept very consistent LB-CV difference", "399382": "AmirH. I appreciate the sharing of your experience. I understand what you did! I'm testing now to see the effect of removal of those features. If it works, I will share you!", "399384": "Justin. Thanks for sharing!", "399533": "AmirH. As testing the effect of time features and hits/pageviews, I got 0.002 boost!. Thanks for sharing!", "399553": "AmirH. Thanks for your suggest, I will go to a try. But why hits/page views ratio causes overfit, the feature looks logical？", "399554": "Justin @YouHan thanks for your sharing!", "399581": "YouHan great to hear!\nMay i ask how much boost does the categorical FE you added to oliviers kernel give you?", "399582": "Hi @ethan.\nThis ratio is very significant in train set as shown in the specific discussion, but i suspect this behavior does not occur in test set for some reason, at least not in public LB set.\nThis is odd, but it is what it is", "399589": "AmirH Actually, I added many categorical features, so it's not easy to say boost in exact number. Sorry for that!\nBut, for now, most of my features are categorical features using mean and frequency encoding.!", "399604": "time feature delete better?? i delete below code then LB & CV get worse\n\n```\nfor df in [train, test]: \n df['date'] = pd.to_datetime(df['visitStartTime'], unit='s') \n df['sess_date_dow'] = df['date'].dt.dayofweek \n df['sess_date_hours'] = df['date'].dt.hour \n df['sess_date_dom'] = df['date'].dt.day \n```", "399626": "Really?\nI removed features from.day, month and hit/visit and then got a boost.", "399628": "Hi all, \n\nSeems removing time related features and hit/ratio improves LB, but be aware of overfitting LB, as final score is evaluated by PB...", "399772": "1 same for me" }, "source": "meta" }