Some of the best (public) kernels use visitStartTime in UNIX format as a core feature of the LGBM with k-fold cross validation on the training set.
But from my understanding of how the decision trees splits such a continuous numerical feature, how would the model deal with the visitStartTime values from the test set ? These will always be significantly larger (since test set takes place a year later) and may trip the decision splits which will go to the right leaf instead of the left leaf for example...
visitStartTime leads to a worse LB scorevisitStartTime also worsens LB score slightlyvisitStartTime from the test set by a year worsens the LB score significantlyWhen using visitStartTime the gap between training and validation CV RMSE and the LB score is large and uncorrelated. For example in my best kernel I get
Whats going on with this feature ? And could it be that leading public kernels are just weirdly overfit ?
", "messages": [ { "id": 396526, "postDate": "2018-10-01T00:46:16.573Z", "content": "Some of the best (public) kernels use visitStartTime in UNIX format as a core feature of the LGBM with k-fold cross validation on the training set.
But from my understanding of how the decision trees splits such a continuous numerical feature, how would the model deal with the visitStartTime values from the test set ? These will always be significantly larger (since test set takes place a year later) and may trip the decision splits which will go to the right leaf instead of the left leaf for example...
visitStartTime leads to a worse LB scorevisitStartTime also worsens LB score slightlyvisitStartTime from the test set by a year worsens the LB score significantlyWhen using visitStartTime the gap between training and validation CV RMSE and the LB score is large and uncorrelated. For example in my best kernel I get
Whats going on with this feature ? And could it be that leading public kernels are just weirdly overfit ?
", "rawMarkdown": "## VisitStartTime\n\nSome of the best (public) kernels use `visitStartTime` in UNIX format as a core feature of the LGBM with k-fold cross validation on the training set. \n\nBut from my understanding of how the decision trees splits such a continuous numerical feature, how would the model deal with the `visitStartTime` values from the test set ? These will always be significantly larger (since test set takes place a year later) and may trip the decision splits which will go to the right leaf instead of the left leaf for example... \n\n## Some observations\n\n- Adding more \"cyclical\" date time features such as day of the week, day of the year etc.. and removing `visitStartTime` leads to a worse LB score\n- Adding date time features **together** with `visitStartTime` also worsens LB score slightly\n- Lagging `visitStartTime` from the test set by a year worsens the LB score significantly\n- When using `visitStartTime` the gap between training and validation CV RMSE and the LB score is large and uncorrelated. For example in my best kernel I get\n\n - Session level CV score: 1.6238689196657545\n - User level CV score: 1.6007747104870507\n - LB score 1.4398\n\nWhats going on with this feature ? And could it be that leading public kernels are just weirdly overfit ? \n", "votes": 10 }, { "id": 396814, "postDate": "2018-10-01T12:19:30.870Z", "content": "I agree the visitStartTime behavior feels strange, but I think one possibility is that it helps to explain some trend over time. In that case, treating the test set as all being at the end period of the train data (tree methods will do this) is a crude proxy, but still better than completely ignoring the trend. Probably worth exploring this in more detail.
\n\nAlso, I'd suggest being careful about some of the \"cyclical\" date time features you mention - I think it'd be quite easy to overfit to those. For example, if a special deal or similar specific effect is in place on certain days or weeks in the training data, the model should not predict on test data as if the same effect is automatically in place at the same time of the year.
", "rawMarkdown": "I agree the visitStartTime behavior feels strange, but I think one possibility is that it helps to explain some trend over time. In that case, treating the test set as all being at the end period of the train data (tree methods will do this) is a crude proxy, but still better than completely ignoring the trend. Probably worth exploring this in more detail.\n\nAlso, I'd suggest being careful about some of the \"cyclical\" date time features you mention - I think it'd be quite easy to overfit to those. For example, if a special deal or similar specific effect is in place on certain days or weeks in the training data, the model should not predict on test data as if the same effect is automatically in place at the same time of the year.", "votes": 2 }, { "id": 396995, "postDate": "2018-10-01T18:22:07.057Z", "content": "@Joe_Eddy Decision trees are notoriously bad at dealing with time series data unless some form of pre-processing is done before hand. Gradient boosted (and bagged) trees perform somewhat better but I am not convinced that the trees trained on the training data with continuous visitStartTime (and thus split into histogram bins) are able to deal with the Test set visitStartTime which will always be greater.
\n\nThis means that at every split on visitStartTime, the model will take the right hand side decision, effectively rendering the visitStartTime split useless... because the decision is already made for all the test data set.
\n\nOne potential explanation is that what this does is some form of randomized pruning of the tree when predicting on the test set, as we are effectively removing all the decisions based on visitStartTime. Since not all trees contain splits on this field, only some trees are pruned and others not
\n\nPerhaps this helps with dealing with overfitting in some unplanned way ?
\n\nLooking forward to other ideas.
", "rawMarkdown": "@Joe_Eddy Decision trees are notoriously **bad** at dealing with time series data unless some form of pre-processing is done before hand. Gradient boosted (and bagged) trees perform somewhat better but I am not convinced that the trees trained on the training data with continuous visitStartTime (and thus split into histogram bins) are able to deal with the Test set visitStartTime which will always be greater. \n\nThis means that at every split on visitStartTime, the model will take the right hand side decision, effectively rendering the visitStartTime split useless... because the decision is already made for all the test data set. \n\nOne potential explanation is that what this does is some form of randomized pruning of the tree when predicting on the test set, as we are effectively removing all the decisions based on visitStartTime. Since not all trees contain splits on this field, only some trees are pruned and others not\n\nPerhaps this helps with dealing with overfitting in some unplanned way ? \n\nLooking forward to other ideas. ", "replies": [ { "id": 397026, "postDate": "2018-10-01T19:13:05.670Z", "content": "I think you missed the point of my comment - \"treating the test set as all being at the end period of the train data (tree methods will do this) is a crude proxy, but still better than completely ignoring the trend\". The tree problems you mention are exactly why I said crude proxy. My point is that if visitStartTime hypothetically helps capture something like e.g. an inflationary price trend, even partially accounting for that trend could be a great improvement over ignoring it. If things just cost more at the end of train than at the start, and cost even more as test proceeds, it's still better to treat test as the end of train in this trend than to ignore the trend completely. It's not the case that using visitStartTime effectively removes all decisions based on it, because including it still materially impacts how the model predicts on the test data (in this case, yes on the entire test data, so it is the same material decision for the entire test data).
\n\nI've not thoroughly explored the data for particular time-based trends yet, the point is that a meaningful trend would be a potential explanation of why including visitStartTime seems to help tree-based models. And yes, there is probably a better approach to capturing any trend than just treating all of test as happening at the end of train.
", "rawMarkdown": "I think you missed the point of my comment - \"treating the test set as all being at the end period of the train data (tree methods will do this) is a crude proxy, but still better than completely ignoring the trend\". The tree problems you mention are exactly why I said *crude proxy*. My point is that if visitStartTime hypothetically helps capture something like e.g. an inflationary price trend, even partially accounting for that trend could be a great improvement over ignoring it. If things just cost more at the end of train than at the start, and cost even more as test proceeds, it's still better to treat test as the end of train in this trend than to ignore the trend completely. It's **not** the case that using visitStartTime effectively removes all decisions based on it, because including it still materially impacts how the model predicts on the test data (in this case, yes on the entire test data, so it is the same material decision for the entire test data).\n\nI've not thoroughly explored the data for particular time-based trends yet, the point is that a meaningful trend would be a potential explanation of why including visitStartTime seems to help tree-based models. And yes, there is probably a better approach to capturing any trend than just treating all of test as happening at the end of train.\n\n ", "votes": 3 }, { "id": 397045, "postDate": "2018-10-01T19:59:41.523Z", "content": "Fair points - what kind of date / time features are you including atm ?
", "rawMarkdown": "Fair points - what kind of date / time features are you including atm ? " } ] }, { "id": 396985, "postDate": "2018-10-01T18:03:40.547Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 396814, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-10-01T12:19:30.870000", "content": "I agree the visitStartTime behavior feels strange, but I think one possibility is that it helps to explain some trend over time. In that case, treating the test set as all being at the end period of the train data (tree methods will do this) is a crude proxy, but still better than completely ignoring the trend. Probably worth exploring this in more detail.
\n\nAlso, I'd suggest being careful about some of the \"cyclical\" date time features you mention - I think it'd be quite easy to overfit to those. For example, if a special deal or similar specific effect is in place on certain days or weeks in the training data, the model should not predict on test data as if the same effect is automatically in place at the same time of the year.
", "votes": 2, "replies": [] }, { "id": 396995, "author_name": "xaviermaxime", "author_url": "", "post_date": "2018-10-01T18:22:07.057000", "content": "@Joe_Eddy Decision trees are notoriously bad at dealing with time series data unless some form of pre-processing is done before hand. Gradient boosted (and bagged) trees perform somewhat better but I am not convinced that the trees trained on the training data with continuous visitStartTime (and thus split into histogram bins) are able to deal with the Test set visitStartTime which will always be greater.
\n\nThis means that at every split on visitStartTime, the model will take the right hand side decision, effectively rendering the visitStartTime split useless... because the decision is already made for all the test data set.
\n\nOne potential explanation is that what this does is some form of randomized pruning of the tree when predicting on the test set, as we are effectively removing all the decisions based on visitStartTime. Since not all trees contain splits on this field, only some trees are pruned and others not
\n\nPerhaps this helps with dealing with overfitting in some unplanned way ?
\n\nLooking forward to other ideas.
", "votes": 0, "replies": [ { "id": 397026, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-10-01T19:13:05.670000", "content": "I think you missed the point of my comment - \"treating the test set as all being at the end period of the train data (tree methods will do this) is a crude proxy, but still better than completely ignoring the trend\". The tree problems you mention are exactly why I said crude proxy. My point is that if visitStartTime hypothetically helps capture something like e.g. an inflationary price trend, even partially accounting for that trend could be a great improvement over ignoring it. If things just cost more at the end of train than at the start, and cost even more as test proceeds, it's still better to treat test as the end of train in this trend than to ignore the trend completely. It's not the case that using visitStartTime effectively removes all decisions based on it, because including it still materially impacts how the model predicts on the test data (in this case, yes on the entire test data, so it is the same material decision for the entire test data).
\n\nI've not thoroughly explored the data for particular time-based trends yet, the point is that a meaningful trend would be a potential explanation of why including visitStartTime seems to help tree-based models. And yes, there is probably a better approach to capturing any trend than just treating all of test as happening at the end of train.
", "votes": 3, "replies": [] }, { "id": 397045, "author_name": "Xavier Bourret Sicotte", "author_url": "", "post_date": "2018-10-01T19:59:41.523000", "content": "Fair points - what kind of date / time features are you including atm ?
", "votes": 0, "replies": [] } ] }, { "id": 396985, "author_name": "", "author_url": "", "post_date": "2018-10-01T18:03:40.547000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "396526": "## VisitStartTime\n\nSome of the best (public) kernels use `visitStartTime` in UNIX format as a core feature of the LGBM with k-fold cross validation on the training set. \n\nBut from my understanding of how the decision trees splits such a continuous numerical feature, how would the model deal with the `visitStartTime` values from the test set ? These will always be significantly larger (since test set takes place a year later) and may trip the decision splits which will go to the right leaf instead of the left leaf for example... \n\n## Some observations\n\n- Adding more \"cyclical\" date time features such as day of the week, day of the year etc.. and removing `visitStartTime` leads to a worse LB score\n- Adding date time features **together** with `visitStartTime` also worsens LB score slightly\n- Lagging `visitStartTime` from the test set by a year worsens the LB score significantly\n- When using `visitStartTime` the gap between training and validation CV RMSE and the LB score is large and uncorrelated. For example in my best kernel I get\n\n - Session level CV score: 1.6238689196657545\n - User level CV score: 1.6007747104870507\n - LB score 1.4398\n\nWhats going on with this feature ? And could it be that leading public kernels are just weirdly overfit ? \n", "396814": "I agree the visitStartTime behavior feels strange, but I think one possibility is that it helps to explain some trend over time. In that case, treating the test set as all being at the end period of the train data (tree methods will do this) is a crude proxy, but still better than completely ignoring the trend. Probably worth exploring this in more detail.\n\nAlso, I'd suggest being careful about some of the \"cyclical\" date time features you mention - I think it'd be quite easy to overfit to those. For example, if a special deal or similar specific effect is in place on certain days or weeks in the training data, the model should not predict on test data as if the same effect is automatically in place at the same time of the year.", "396995": "@Joe_Eddy Decision trees are notoriously **bad** at dealing with time series data unless some form of pre-processing is done before hand. Gradient boosted (and bagged) trees perform somewhat better but I am not convinced that the trees trained on the training data with continuous visitStartTime (and thus split into histogram bins) are able to deal with the Test set visitStartTime which will always be greater. \n\nThis means that at every split on visitStartTime, the model will take the right hand side decision, effectively rendering the visitStartTime split useless... because the decision is already made for all the test data set. \n\nOne potential explanation is that what this does is some form of randomized pruning of the tree when predicting on the test set, as we are effectively removing all the decisions based on visitStartTime. Since not all trees contain splits on this field, only some trees are pruned and others not\n\nPerhaps this helps with dealing with overfitting in some unplanned way ? \n\nLooking forward to other ideas. ", "396985": "" } }