Hello all,\nI just made a classifier to differentiate between train & test data.\nAfter dropping all date related features (including them will create leckage), classifier was 100% accurate (AUC : 1.0) in classifying between train & test data !!
\n\nSo, I ploted all the important features. here are those who clearly differentiate train & test data.
\n\n\nbrowser_category,\ncount_hits_per_network_domain,\nmonth_unique_user_count,\nday_unique_user_count,\nsum_hits_per_day,\nvar_hits_per_day,\nbrowser_os\n
Check the output plot of feature importance. LGBClassifier only need 10 features to differentiate train/test.
\n\nSo, are these 10 features are really creating CV/LB difference issues or they are creating leackage and I should drop them?
\n\nThanks.
", "messages": [ { "id": "392135", "postDate": "09/23/2018 04:32:51", "content": "Hello all,\nI just made a classifier to differentiate between train & test data.\nAfter dropping all date related features (including them will create leckage), classifier was 100% accurate (AUC : 1.0) in classifying between train & test data !!
\n\nSo, I ploted all the important features. here are those who clearly differentiate train & test data.
\n\n\nbrowser_category,\ncount_hits_per_network_domain,\nmonth_unique_user_count,\nday_unique_user_count,\nsum_hits_per_day,\nvar_hits_per_day,\nbrowser_os\n
Check the output plot of feature importance. LGBClassifier only need 10 features to differentiate train/test.
\n\nSo, are these 10 features are really creating CV/LB difference issues or they are creating leackage and I should drop them?
\n\nThanks.
", "rawMarkdown": "Hello all,\nI just made a [classifier][1] to differentiate between train & test data.\nAfter dropping all date related features (including them will create leckage), classifier was 100% accurate (AUC : 1.0) in classifying between train & test data !!\n\nSo, I ploted all the important features. here are those who clearly differentiate train & test data.\n\n```\nbrowser_category,\ncount_hits_per_network_domain,\nmonth_unique_user_count,\nday_unique_user_count,\nsum_hits_per_day,\nvar_hits_per_day,\nbrowser_os\n``` \n\nCheck the output plot of feature importance. LGBClassifier only need 10 features to differentiate train/test.\n\nSo, are these 10 features are really creating CV/LB difference issues or they are creating leackage and I should drop them?\n\nThanks. \n\n [1]: https://www.kaggle.com/prashantkikani/howmuch-train-test-data-are-different", "votes": null }, { "id": "392192", "postDate": "09/23/2018 08:43:52", "content": "Here we have a different count of a unique user in train and test. So, we will have a shift in all user-dependent sum or count features. I upload a little example of unique-user-per-date hist. Distributions are quite similar, but we can see a shift. To solve this issue you can use some normalization.
", "rawMarkdown": "Here we have a different count of a unique user in train and test. So, we will have a shift in all user-dependent sum or count features. I upload a little example of unique-user-per-date hist. Distributions are quite similar, but we can see a shift. To solve this issue you can use some normalization.", "votes": null }, { "id": "392198", "postDate": "09/23/2018 08:52:23", "content": "Interesting find, but i think i know why this is.\nTrain data has a whole year of data, Test data has less months.\nThe 2nd most influential feature is \"count_hits_per_network_domain\"\nwhich means it counts hits for the entire train\\test duration.\nbecause Train has more rows of data, this feature will have higher values.
\n\nTo really check this properly, try to have train and test have then same relative time difference, e.g - have train data until end of Apr-2017.\nThe recalculate the features and try this prediction, i assume the result will be less significant than 1.0 AUC =)
", "rawMarkdown": "Interesting find, but i think i know why this is.\nTrain data has a whole year of data, Test data has less months.\nThe 2nd most influential feature is \"count_hits_per_network_domain\"\nwhich means it counts hits for the entire train\\test duration.\nbecause Train has more rows of data, this feature will have higher values.\n\nTo really check this properly, try to have train and test have then same relative time difference, e.g - have train data until end of Apr-2017.\nThe recalculate the features and try this prediction, i assume the result will be less significant than 1.0 AUC =)", "votes": null }, { "id": "392298", "postDate": "09/23/2018 13:58:56", "content": "The time-aggregate features you derive like day_unique_user_count are likely acting as unique identifiers for specific days, which lgb can easily pick up on. That's why you are able to perfectly classify, but it doesn't really say anything about non time-based differences between train and test.
You had the absolute right idea to avoid leakage by excluding date features, but by including aggregates derived from dates you've given your model the same leakage information in an only slightly less direct way.
", "rawMarkdown": "The time-aggregate features you derive like `day_unique_user_count` are likely acting as unique identifiers for specific days, which lgb can easily pick up on. That's why you are able to perfectly classify, but it doesn't really say anything about non time-based differences between train and test. \n\nYou had the absolute right idea to avoid leakage by excluding date features, but by including aggregates derived from dates you've given your model the same leakage information in an only slightly less direct way.", "votes": null }, { "id": "392325", "postDate": "09/23/2018 14:37:15", "content": "hi @aquatic, day_unique_user_count is calculated by grouping at a day level and not at a date level. Hence this variable doesn't contain unique identifiers for specific days but rather what's the count on day 29 or day 1 i.e. unique identifiers for day number (which is similar having a categorical variable with 30 levels encoded) \nLet me know if this makes sense?
", "rawMarkdown": "hi @aquatic, day_unique_user_count is calculated by grouping at a day level and not at a date level. Hence this variable doesn't contain unique identifiers for specific days but rather what's the count on day 29 or day 1 i.e. unique identifiers for day number (which is similar having a categorical variable with 30 levels encoded) \nLet me know if this makes sense?", "votes": null } ], "comments": [ { "id": 392192, "author_name": "karachunmichael", "author_url": "", "post_date": "09/23/2018 08:43:52", "content": "Here we have a different count of a unique user in train and test. So, we will have a shift in all user-dependent sum or count features. I upload a little example of unique-user-per-date hist. Distributions are quite similar, but we can see a shift. To solve this issue you can use some normalization.
", "votes": null, "replies": [] }, { "id": 392198, "author_name": "tpthegreat", "author_url": "", "post_date": "09/23/2018 08:52:23", "content": "Interesting find, but i think i know why this is.\nTrain data has a whole year of data, Test data has less months.\nThe 2nd most influential feature is \"count_hits_per_network_domain\"\nwhich means it counts hits for the entire train\\test duration.\nbecause Train has more rows of data, this feature will have higher values.
\n\nTo really check this properly, try to have train and test have then same relative time difference, e.g - have train data until end of Apr-2017.\nThe recalculate the features and try this prediction, i assume the result will be less significant than 1.0 AUC =)
", "votes": null, "replies": [] }, { "id": 392298, "author_name": "aquatic", "author_url": "", "post_date": "09/23/2018 13:58:56", "content": "The time-aggregate features you derive like day_unique_user_count are likely acting as unique identifiers for specific days, which lgb can easily pick up on. That's why you are able to perfectly classify, but it doesn't really say anything about non time-based differences between train and test.
You had the absolute right idea to avoid leakage by excluding date features, but by including aggregates derived from dates you've given your model the same leakage information in an only slightly less direct way.
", "votes": null, "replies": [ { "id": 392325, "author_name": "sahil711", "author_url": "", "post_date": "09/23/2018 14:37:15", "content": "hi @aquatic, day_unique_user_count is calculated by grouping at a day level and not at a date level. Hence this variable doesn't contain unique identifiers for specific days but rather what's the count on day 29 or day 1 i.e. unique identifiers for day number (which is similar having a categorical variable with 30 levels encoded) \nLet me know if this makes sense?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "392135": "Hello all,\nI just made a [classifier][1] to differentiate between train & test data.\nAfter dropping all date related features (including them will create leckage), classifier was 100% accurate (AUC : 1.0) in classifying between train & test data !!\n\nSo, I ploted all the important features. here are those who clearly differentiate train & test data.\n\n```\nbrowser_category,\ncount_hits_per_network_domain,\nmonth_unique_user_count,\nday_unique_user_count,\nsum_hits_per_day,\nvar_hits_per_day,\nbrowser_os\n``` \n\nCheck the output plot of feature importance. LGBClassifier only need 10 features to differentiate train/test.\n\nSo, are these 10 features are really creating CV/LB difference issues or they are creating leackage and I should drop them?\n\nThanks. \n\n [1]: https://www.kaggle.com/prashantkikani/howmuch-train-test-data-are-different", "392192": "Here we have a different count of a unique user in train and test. So, we will have a shift in all user-dependent sum or count features. I upload a little example of unique-user-per-date hist. Distributions are quite similar, but we can see a shift. To solve this issue you can use some normalization.", "392198": "Interesting find, but i think i know why this is.\nTrain data has a whole year of data, Test data has less months.\nThe 2nd most influential feature is \"count_hits_per_network_domain\"\nwhich means it counts hits for the entire train\\test duration.\nbecause Train has more rows of data, this feature will have higher values.\n\nTo really check this properly, try to have train and test have then same relative time difference, e.g - have train data until end of Apr-2017.\nThe recalculate the features and try this prediction, i assume the result will be less significant than 1.0 AUC =)", "392298": "The time-aggregate features you derive like `day_unique_user_count` are likely acting as unique identifiers for specific days, which lgb can easily pick up on. That's why you are able to perfectly classify, but it doesn't really say anything about non time-based differences between train and test. \n\nYou had the absolute right idea to avoid leakage by excluding date features, but by including aggregates derived from dates you've given your model the same leakage information in an only slightly less direct way.", "392325": "hi @aquatic, day_unique_user_count is calculated by grouping at a day level and not at a date level. Hence this variable doesn't contain unique identifiers for specific days but rather what's the count on day 29 or day 1 i.e. unique identifiers for day number (which is similar having a categorical variable with 30 levels encoded) \nLet me know if this makes sense?" }, "source": "meta" }