I've been playing with this data for longer than I care to admit and it certainly poses some unique challenges. Here are a few insights I've come up with that may help others:
\n\nI completely agree with everyone who has written that the competition has very little business value, both because of a poorly chosen metric and the ability to peek into the future. I'm treating it simply as a data puzzle rather than a business problem.
I'm also seeing a large discrepancy in CV and LB scores. Currently my best model scores 1.526 on visitor-level 5-fold CV and 1.4084 on the LB (my best LB score [1.4065] comes from a little post-processing). Assuming no error in the metric, the only explanation for this that I can think of is that the test set contains a much lower percentage of visitors with purchases. You can mimic this behavior by randomly eliminating about 20% of the visitors with purchases from your validation set. This should give you scores much closer to your LB scores.
Concerning CV and LB scores moving in opposite directions, I think this is mostly due to overfitting. As others have noted, any features that enable the model to distinguish one day from another are likely to overfit. This is a tough problem since there appear to be real seasonality differences (e.g., more purchases right before Christmas), but I have yet to figure out a way to reliably distinguish between real and spurious seasonality differences. One good way to eliminate date features that are causing overfitting is to perform feature selection with a time-based validation set (e.g., last 2 months). Any features that make the time-based CV score worse should be eliminated, even if they would help a kfold-based cv. In addition to the date features, there are likely other features that are causing overfitting due to differences in distribution between the train and sets.
I've been playing with this data for longer than I care to admit and it certainly poses some unique challenges. Here are a few insights I've come up with that may help others:
\n\nI completely agree with everyone who has written that the competition has very little business value, both because of a poorly chosen metric and the ability to peek into the future. I'm treating it simply as a data puzzle rather than a business problem.
I'm also seeing a large discrepancy in CV and LB scores. Currently my best model scores 1.526 on visitor-level 5-fold CV and 1.4084 on the LB (my best LB score [1.4065] comes from a little post-processing). Assuming no error in the metric, the only explanation for this that I can think of is that the test set contains a much lower percentage of visitors with purchases. You can mimic this behavior by randomly eliminating about 20% of the visitors with purchases from your validation set. This should give you scores much closer to your LB scores.
Concerning CV and LB scores moving in opposite directions, I think this is mostly due to overfitting. As others have noted, any features that enable the model to distinguish one day from another are likely to overfit. This is a tough problem since there appear to be real seasonality differences (e.g., more purchases right before Christmas), but I have yet to figure out a way to reliably distinguish between real and spurious seasonality differences. One good way to eliminate date features that are causing overfitting is to perform feature selection with a time-based validation set (e.g., last 2 months). Any features that make the time-based CV score worse should be eliminated, even if they would help a kfold-based cv. In addition to the date features, there are likely other features that are causing overfitting due to differences in distribution between the train and sets.
Quick question: would a feature such as \"time to next visit\" be at risk of leakage ?
", "rawMarkdown": "Quick question: would a feature such as \"time to next visit\" be at risk of leakage ? ", "votes": 1, "replies": [ { "id": 400125, "postDate": "2018-10-07T16:39:45.950Z", "content": "Leakage - not necessarily (you are not using the target). That being said, it might get tricky in two \"edge\" cases: 1. user visits towards the end of the test set 2. if test set has more users with single visit (NA for such feature) than train.
", "rawMarkdown": "Leakage - not necessarily (you are not using the target). That being said, it might get tricky in two \"edge\" cases: 1. user visits towards the end of the test set 2. if test set has more users with single visit (NA for such feature) than train.", "votes": 2 } ] }, { "id": 400366, "postDate": "2018-10-08T07:16:17.640Z", "content": "I have seen this topic before :D
\n\nThanks for sharing your hints!
", "rawMarkdown": "I have seen this topic before :D\n\nThanks for sharing your hints!", "votes": 2 }, { "id": 399170, "postDate": "2018-10-05T10:55:43.250Z", "content": "I got similar CV to LB by testing on np.log1p(revenues) <18.3
\n\nExplaining disparity between CV and the LB
", "rawMarkdown": "I got similar CV to LB by testing on np.log1p(revenues) <18.3\n\n[Explaining disparity between CV and the LB][1]\n\n\n [1]: https://www.kaggle.com/scirpus/explaining-disparity-between-cv-and-the-lb", "votes": 2, "replies": [ { "id": 399214, "postDate": "2018-10-05T12:31:22.507Z", "content": "@scirpus, It's true that you could just remove visitors with the biggest values and then you would only need to remove about 1% instead of 20% to match cv to LB. But that seems sort of strange. It would make more sense to set a ceiling on the total transaction revenue. If you do this, the ceiling value is about 16.5. I think I tested some ceiling values previously and found that they don't help the LB score, but I'm not sure now whether the tests were before or after the metric correction. Has anyone tested this?
", "rawMarkdown": "@scirpus, It's true that you could just remove visitors with the biggest values and then you would only need to remove about 1% instead of 20% to match cv to LB. But that seems sort of strange. It would make more sense to set a ceiling on the total transaction revenue. If you do this, the ceiling value is about 16.5. I think I tested some ceiling values previously and found that they don't help the LB score, but I'm not sure now whether the tests were before or after the metric correction. Has anyone tested this?", "votes": 3 }, { "id": 399223, "postDate": "2018-10-05T12:57:48.177Z", "content": "Yeah I train on everything and just use the threshold as an additional scoring metric. Hopefully using this and the full if they both improve I am going in the right direction.
\n\nThe RMSE metric is a bit odd as we know < 0 is wrong and values between 0 and x are wrong so I am looking at Prob(positivescore)*Regression(between 10 and 25) combined to get the final RMSE - I can do this with genetic programming but I haven't a clue how to co-evolve this using LGBM or XGBoost.
\n\nThe ceiling thing sounds good as a few rare big values can skew the results quite badly
", "rawMarkdown": "Yeah I train on everything and just use the threshold as an additional scoring metric. Hopefully using this and the full if they both improve I am going in the right direction.\n\nThe RMSE metric is a bit odd as we know < 0 is wrong and values between 0 and x are wrong so I am looking at Prob(positivescore)*Regression(between 10 and 25) combined to get the final RMSE - I can do this with genetic programming but I haven't a clue how to co-evolve this using LGBM or XGBoost.\n\nThe ceiling thing sounds good as a few rare big values can skew the results quite badly\n\n\n\n" }, { "id": 399332, "postDate": "2018-10-05T16:12:05.680Z", "content": "To update, I just burnt a submission setting the ceiling at 17 and the LB score was slightly worse, so I don't think this is the issue.
", "rawMarkdown": "To update, I just burnt a submission setting the ceiling at 17 and the LB score was slightly worse, so I don't think this is the issue.", "votes": 3 }, { "id": 399400, "postDate": "2018-10-05T17:54:32.123Z", "content": "Wish I had seen this post before burning one doing the same!
\n\nThanks for sharing though, I have more confidence in my results now
", "rawMarkdown": "Wish I had seen this post before burning one doing the same! \n\nThanks for sharing though, I have more confidence in my results now", "votes": 1 } ] }, { "id": 401767, "postDate": "2018-10-10T17:21:50.967Z", "content": "Really appreciate your sharing! Great insights!
", "rawMarkdown": "Really appreciate your sharing! Great insights!" }, { "id": 400897, "postDate": "2018-10-09T05:18:42.470Z", "content": "Cool stuff!
", "rawMarkdown": "Cool stuff!" }, { "id": 400271, "postDate": "2018-10-08T02:53:37.040Z", "content": "There are few duplicates in the session Id, do you think they should be removed?
", "rawMarkdown": "There are few duplicates in the session Id, do you think they should be removed?", "replies": [ { "id": 400359, "postDate": "2018-10-08T07:02:14.023Z", "content": "They're not really duplicates -- they're sessions that start on one date and continue into the following day. I made a change in my code recently to merge them into a single session before further processing. It doesn't seem to have any effect on the LB score but it makes me feel better ;-).
", "rawMarkdown": "They're not really duplicates -- they're sessions that start on one date and continue into the following day. I made a change in my code recently to merge them into a single session before further processing. It doesn't seem to have any effect on the LB score but it makes me feel better ;-).", "votes": 5 }, { "id": 400367, "postDate": "2018-10-08T07:19:45.153Z", "content": "@Silogram,\nI spotted some of them that have transactionRevenue for BOTH the rows that are the same session.\nThat's something i found to be very odd.\nboth of them having different transaction revenue value
", "rawMarkdown": "@Silogram,\nI spotted some of them that have transactionRevenue for BOTH the rows that are the same session.\nThat's something i found to be very odd.\nboth of them having different transaction revenue value", "votes": 1 }, { "id": 401586, "postDate": "2018-10-10T10:20:08.403Z", "content": "@AmirH: I haven't checked these but it could be that in a session crossing a day barrier a visitor makes two different purchases, one in day1 and the other in day2. When the session is split, it will generate sessions with the same ID but different transaction values. Just an idea...
\n\nBtw, I'll be off the grid for the next week or so. Good luck to everyone!
", "rawMarkdown": "@AmirH: I haven't checked these but it could be that in a session crossing a day barrier a visitor makes two different purchases, one in day1 and the other in day2. When the session is split, it will generate sessions with the same ID but different transaction values. Just an idea...\n\nBtw, I'll be off the grid for the next week or so. Good luck to everyone!", "votes": 2 } ] }, { "id": 400195, "postDate": "2018-10-07T19:55:40.167Z", "content": "Hi Silogram, thanks a lot for sharing these important thoughts. Can you share some hints regarding how you do post-processing, for example I tried following -
\n\nBut, these are not that helpful.
", "rawMarkdown": "Hi Silogram, thanks a lot for sharing these important thoughts. Can you share some hints regarding how you do post-processing, for example I tried following -\n\n 1. Using missing value of bounces\n 2. Converting all predictions (before applying log1p) to their corresponding closest numbers multiple of 10000.\n\nBut, these are not that helpful.", "replies": [ { "id": 400355, "postDate": "2018-10-08T06:56:44.177Z", "content": "I tried a variety of linear transformations to probe the test set and get a better idea of how the test labels are distributed. But I'm no longer using this technique and don't really recommend it since it overfits to the public test set and may fail on the private set.
", "rawMarkdown": "I tried a variety of linear transformations to probe the test set and get a better idea of how the test labels are distributed. But I'm no longer using this technique and don't really recommend it since it overfits to the public test set and may fail on the private set.", "votes": 2 } ] }, { "id": 400031, "postDate": "2018-10-07T12:18:05.873Z", "content": "Tanks for sharing . expected update
", "rawMarkdown": "Tanks for sharing . expected update" }, { "id": 399996, "postDate": "2018-10-07T10:22:48.550Z", "content": "Thanks for the insights Silogram. A basic question concerning point 1:
\n\nDoes google analytics have the revenue information of all shops subscribed to ga?\nIf the answer is \"no\" then could ga use the model (or insights derived from it) to estimate the \"value\" of a shop subscribed to ga? (However, of course a real future value of a shop would be more useful)
", "rawMarkdown": "Thanks for the insights Silogram. A basic question concerning point 1:\n\nDoes google analytics have the revenue information of _all_ shops subscribed to ga?\nIf the answer is \"no\" then could ga use the model (or insights derived from it) to estimate the \"value\" of a shop subscribed to ga? (However, of course a real _future_ value of a shop would be more useful)", "replies": [ { "id": 400067, "postDate": "2018-10-07T13:48:48.323Z", "content": "@Kinimo, I don't think a model based on data from only one shop would generalize well to other shops.
", "rawMarkdown": "@Kinimo, I don't think a model based on data from only one shop would generalize well to other shops." } ] }, { "id": 399725, "postDate": "2018-10-06T15:32:00.810Z", "content": "Silogram, a big thank you for another good read of yours !
", "rawMarkdown": "Silogram, a big thank you for another good read of yours !" }, { "id": 399426, "postDate": "2018-10-05T18:54:00.777Z", "content": "Thank you so much Silogram :) I was eagerly waiting for this
", "rawMarkdown": "Thank you so much Silogram :) I was eagerly waiting for this" }, { "id": 399355, "postDate": "2018-10-05T16:53:41.590Z", "content": "@Silogram, Thank you for sharing! I am wondering are you training your model based on session level or user level?
", "rawMarkdown": "@Silogram, Thank you for sharing! I am wondering are you training your model based on session level or user level? ", "replies": [ { "id": 399356, "postDate": "2018-10-05T16:57:02.693Z", "content": "And on the other hand...I am kind of afraid if the remaining test data has more users with purchase..... It will definitely a huge shake when the competition end.....@Silogram
", "rawMarkdown": "And on the other hand...I am kind of afraid if the remaining test data has more users with purchase..... It will definitely a huge shake when the competition end.....@Silogram" }, { "id": 399371, "postDate": "2018-10-05T17:18:16.670Z", "content": "Both. I have three level-1 models. Two are session based and one is user-based. The final predictions are an ensemble of the three models.
\n\nI hate to say it, but I actually think the LB is more reliable than our local CV scores in this competition. The fact that LB scores are so far away from anything we can get with any kind of CV split suggests that the test set is much different from the train set. I guess it's always possible that Kaggle has split the public and private sets in some bizarre way that is causing this behavior, but I'm assuming that's not the case.
", "rawMarkdown": "Both. I have three level-1 models. Two are session based and one is user-based. The final predictions are an ensemble of the three models.\n\nI hate to say it, but I actually think the LB is more reliable than our local CV scores in this competition. The fact that LB scores are so far away from anything we can get with any kind of CV split suggests that the test set is much different from the train set. I guess it's always possible that Kaggle has split the public and private sets in some bizarre way that is causing this behavior, but I'm assuming that's not the case.", "votes": 6 }, { "id": 399374, "postDate": "2018-10-05T17:21:45.130Z", "content": "Yeah...Have no choice but to trust the LB now...lol...
", "rawMarkdown": "Yeah...Have no choice but to trust the LB now...lol..." }, { "id": 399410, "postDate": "2018-10-05T18:12:05.837Z", "content": "Silogram, may I ask is your 1.526 CV comes from one of those three models or is it ensembled CV score?
", "rawMarkdown": "Silogram, may I ask is your 1.526 CV comes from one of those three models or is it ensembled CV score?", "votes": 2 }, { "id": 399413, "postDate": "2018-10-05T18:25:31.103Z", "content": "Silogram, I think for user based model, aggregating train sessions to fit users' one year revenue is not comparable with aggregating test sessions to predict users' 9 month revenue on test. Is that right?
", "rawMarkdown": "Silogram, I think for user based model, aggregating train sessions to fit users' one year revenue is not comparable with aggregating test sessions to predict users' 9 month revenue on test. Is that right? ", "votes": 1 }, { "id": 399425, "postDate": "2018-10-05T18:53:40.720Z", "content": "@fatihöztürk Ensemble score.
", "rawMarkdown": "@fatihöztürk Ensemble score.", "votes": 1 }, { "id": 399427, "postDate": "2018-10-05T18:56:00.003Z", "content": "@dylan_chen: No, you can't just sum up the session predictions. Even if the test duration were the same as the train duration, simple summing wouldn't work very well.
", "rawMarkdown": "@dylan_chen: No, you can't just sum up the session predictions. Even if the test duration were the same as the train duration, simple summing wouldn't work very well.", "votes": 2 }, { "id": 399657, "postDate": "2018-10-06T12:07:23.263Z", "rawMarkdown": "", "isDeleted": true }, { "id": 399899, "postDate": "2018-10-07T04:06:44.940Z", "content": "thanks silogram. i have a question. how to use 2 session levels ..? same features different fold strategy? or using different features ?
", "rawMarkdown": "thanks silogram. i have a question. how to use 2 session levels ..? same features different fold strategy? or using different features ?" }, { "id": 400071, "postDate": "2018-10-07T13:54:19.210Z", "content": "@hyun woo kim, Typically one would ensemble with many models varying all of the following:
\n\nThis competition is somewhat unusual inasmuch as it might also help to vary the fold strategy, but I haven't tried that yet.
", "rawMarkdown": "@hyun woo kim, Typically one would ensemble with many models varying all of the following:\n\n- features\n- objective function and/or label transformations\n- algorithm\n\nThis competition is somewhat unusual inasmuch as it might also help to vary the fold strategy, but I haven't tried that yet.", "votes": 5 }, { "id": 400072, "postDate": "2018-10-07T13:59:25.693Z", "content": "Thanks for answering. i try it your answer. thanks for helping me :)
", "rawMarkdown": "Thanks for answering. i try it your answer. thanks for helping me :) " }, { "id": 400116, "postDate": "2018-10-07T16:17:50.210Z", "content": "Silogram, first of all, thanks for your tips. I improved a lot after your post. I have doubts about your ensemble strategy. I tried stack 4 models (2 lgb user/session and 2 xgb user/session) using a linear model. My cv (5 folds) reduced almost 0.01 but my LB score was the same. After that, I tried a simple blending using 2 user level models. I set the weights based on LB scores (better lb more weight) and get an little improvement. Would you mind to share any insight? Thanks in advance!
", "rawMarkdown": "Silogram, first of all, thanks for your tips. I improved a lot after your post. I have doubts about your ensemble strategy. I tried stack 4 models (2 lgb user/session and 2 xgb user/session) using a linear model. My cv (5 folds) reduced almost 0.01 but my LB score was the same. After that, I tried a simple blending using 2 user level models. I set the weights based on LB scores (better lb more weight) and get an little improvement. Would you mind to share any insight? Thanks in advance!", "votes": 2 }, { "id": 400288, "postDate": "2018-10-08T03:49:20.193Z", "content": "@Silogram Not sure if my question makes sense. I wonder how do you ensemble your two level-1 session model if summing up session predictions is not an option.
\n\nOr are you using your user-based model to learn how to sum session predictions similar to this kernel?\nhttps://www.kaggle.com/ogrellier/teach-lightgbm-to-sum-predictions
", "rawMarkdown": "@Silogram Not sure if my question makes sense. I wonder how do you ensemble your two level-1 session model if summing up session predictions is not an option.\n\nOr are you using your user-based model to learn how to sum session predictions similar to this kernel?\nhttps://www.kaggle.com/ogrellier/teach-lightgbm-to-sum-predictions\n", "votes": 1 }, { "id": 400353, "postDate": "2018-10-08T06:52:39.203Z", "content": "@Pauo Felipe & James Chong, To ensemble level-1 predictions, you can train a new model that uses the level-1 predictions as features. For the session-based models, you need to first generate features at the user level by aggregating the session predictions (e.g., mean, median, sum, max, min, etc.). Note that because leakage is a risk for this type of architecture, you need to be careful not to overfit the level-2 model.
", "rawMarkdown": "@Pauo Felipe & James Chong, To ensemble level-1 predictions, you can train a new model that uses the level-1 predictions as features. For the session-based models, you need to first generate features at the user level by aggregating the session predictions (e.g., mean, median, sum, max, min, etc.). Note that because leakage is a risk for this type of architecture, you need to be careful not to overfit the level-2 model.", "votes": 5 }, { "id": 400399, "postDate": "2018-10-08T08:46:58.677Z", "content": "@Silogram, Thanks for previous replying. As you said, I find the level-2 model overfit the level-1 predictions too much since I use level-1 predictions as features. And it is difficult to keep CV consitent with LB. So do you think it is more stable to use simple blending for this problem?
", "rawMarkdown": "@Silogram, Thanks for previous replying. As you said, I find the level-2 model overfit the level-1 predictions too much since I use level-1 predictions as features. And it is difficult to keep CV consitent with LB. So do you think it is more stable to use simple blending for this problem?" }, { "id": 400558, "postDate": "2018-10-08T14:44:57.473Z", "content": "@dylan_chen: Yes, aligning CV with LB is the biggest challenge in this competition. It's still a mystery to me...
", "rawMarkdown": "@dylan_chen: Yes, aligning CV with LB is the biggest challenge in this competition. It's still a mystery to me..." }, { "id": 400617, "postDate": "2018-10-08T16:21:32.653Z", "content": "Thanks, Silogram! I will try to find out a way to no overfit when training the 2-level model!
", "rawMarkdown": "Thanks, Silogram! I will try to find out a way to no overfit when training the 2-level model! " }, { "id": 402018, "postDate": "2018-10-11T03:42:17.960Z", "content": "Thanks Silogram for your great inputs.
\n\nFew follow up questions
\n\nThanks a ton in advance
", "rawMarkdown": "Thanks Silogram for your great inputs.\n\nFew follow up questions\n\n 1. How did you come with the idea of making models based on session based and user based?\n 2. If 2 of the models on layer-1 are session based, will the prediction will be aggregate in layer-2, as in layer-2 you have 1 user based prediction? I am assuming in layer-2 all will be based on user based before it trained by a model ?\n 3. Which Fold strategy you have used, K / Group ?\n 4. Have you used RF/ET / NN ? How do i know, these will not be usefull for this type of problems / competition ?\n 5. How do i know my model is overfitting ?\n 6. Whether same feature is used for all the 3 models at level-1 ? If different features are use, how do we know which feature should i used for which model before ensemble?\n \nThanks a ton in advance" } ] }, { "id": 399324, "postDate": "2018-10-05T15:49:13.860Z", "content": "I'm one of those \"lucky\" ones with potential data leakage. So far I struggle to understand how to trace down the source of the leak. Your third proposal sounds like something that I have not tried yet. Could you elaborate on it for the case of user-level models? I do not build visit-level models and at the visitor level time split is unclear to me.
", "rawMarkdown": "I'm one of those \"lucky\" ones with potential data leakage. So far I struggle to understand how to trace down the source of the leak. Your third proposal sounds like something that I have not tried yet. Could you elaborate on it for the case of user-level models? I do not build visit-level models and at the visitor level time split is unclear to me.", "replies": [ { "id": 399340, "postDate": "2018-10-05T16:21:43.510Z", "content": "Yes, there's no way to do a time-based split at the user level, so you just need to remove all date-based features. If your model includes visitStartTime, for example, this will overfit badly. My models have just two date-based features -- hour and year. Note that at the user level, you can generate additional features based on visitStartTime that do not overfit. For example, the number of seconds between the first and last session is a valid feature.
", "rawMarkdown": "Yes, there's no way to do a time-based split at the user level, so you just need to remove all date-based features. If your model includes visitStartTime, for example, this will overfit badly. My models have just two date-based features -- hour and year. Note that at the user level, you can generate additional features based on visitStartTime that do not overfit. For example, the number of seconds between the first and last session is a valid feature.", "votes": 11 }, { "id": 399350, "postDate": "2018-10-05T16:36:15.933Z", "content": "Thanks for the insights. They are very useful, as were those in the HomeCredit comp.
\n\nCould you elaborate on why time-related features are prong to overfitting. This is my first time series competition, so even basic issues are not apparent to me. By time-related features I do not mean exact date and not aggregates per date, but some relative times, like day of the week, number of days to closest Hanukkah, hour of the day, etc.
\n\nBtw, why year is not a problematic feature, i.e. why do you use it? Isn't year very different between train and test?
", "rawMarkdown": "Thanks for the insights. They are very useful, as were those in the HomeCredit comp.\n\nCould you elaborate on why time-related features are prong to overfitting. This is my first time series competition, so even basic issues are not apparent to me. By time-related features I do not mean exact date and not aggregates per date, but some relative times, like day of the week, number of days to closest Hanukkah, hour of the day, etc. \n\nBtw, why year is not a problematic feature, i.e. why do you use it? Isn't year very different between train and test?", "votes": 1 }, { "id": 399364, "postDate": "2018-10-05T17:11:07.073Z", "content": "If you look at a graph of the mean ratio of purchase to visit by day, you'll see that there's quite a bit of variation. This variation can have three different sources:
\n\nWe want our models to ignore #1 and #3 but take into account #2. The problem we have in this competition is that there's no way for the model to distinguish between #2 and #3. If the model can figure out which day a session belongs to, it will infer the probability of the user purchasing something on that day. This can be a strong feature and make the cv score improve, but the model can't generalize the feature to apply to dates in the test set. GBMs are very clever about combining features so if you include month and dayofweek, for example, the model can easily narrow down the day to one of 4 or 5 days of the year.
\n\nHour and dayofweek by themselves are probably OK, but days before Hannukah is a problem. To see why, suppose that in 2016, on one or more days before Hannukah, Google had a 50% sale. This would cause the purchase probability to increase on those days in the training set, and the model will take this into account. But if Google doesn't repeat the sale in 2017, the model will badly over-estimate purchases on these days in the test set. So again, the model can't 'know' whether the increase in sales occurred because a holiday was approaching (something that repeats regularly) or by an unpredictable factor (e.g., the sale).
\n\nRegarding the year feature, it's just a very imperfect way of including long-term trends in the model (e.g., is the purchase probability increasing or decreasing year over year). It's true that the year 2018 appears only in the test set so the model will treat it as being more like 2017 than 2016. It seems to help the cv and LB scores a little but it's not a strong feature.
", "rawMarkdown": "If you look at a graph of the mean ratio of purchase to visit by day, you'll see that there's quite a bit of variation. This variation can have three different sources:\n\n1. Randomness\n2. Seasonality (e.g., days before Christmas or Hanukkah)\n3. Unknown factors (e.g., special promotions, new product introductions, price changes, payment system failures, etc.)\n\nWe want our models to ignore #1 and #3 but take into account #2. The problem we have in this competition is that there's no way for the model to distinguish between #2 and #3. If the model can figure out which day a session belongs to, it will infer the probability of the user purchasing something on that day. This can be a strong feature and make the cv score improve, but the model can't generalize the feature to apply to dates in the test set. GBMs are very clever about combining features so if you include month and dayofweek, for example, the model can easily narrow down the day to one of 4 or 5 days of the year. \n\nHour and dayofweek by themselves are probably OK, but days before Hannukah is a problem. To see why, suppose that in 2016, on one or more days before Hannukah, Google had a 50% sale. This would cause the purchase probability to increase on those days in the training set, and the model will take this into account. But if Google doesn't repeat the sale in 2017, the model will badly over-estimate purchases on these days in the test set. So again, the model can't 'know' whether the increase in sales occurred because a holiday was approaching (something that repeats regularly) or by an unpredictable factor (e.g., the sale).\n\nRegarding the year feature, it's just a very imperfect way of including long-term trends in the model (e.g., is the purchase probability increasing or decreasing year over year). It's true that the year 2018 appears only in the test set so the model will treat it as being more like 2017 than 2016. It seems to help the cv and LB scores a little but it's not a strong feature.", "votes": 22 }, { "id": 399600, "postDate": "2018-10-06T07:56:25.463Z", "content": "In 2016 Hanukkah and Christmas were both at December 25 But in 2017 Hanukkah was at December 13. I think this is the reason you get discrepancy between the revenues.
", "rawMarkdown": "In 2016 Hanukkah and Christmas were both at December 25 But in 2017 Hanukkah was at December 13. I think this is the reason you get discrepancy between the revenues.", "votes": 1 }, { "id": 399716, "postDate": "2018-10-06T15:00:41.123Z", "content": "@psilogram\nThanks for detailed explanation. This makes the situation with temporal features more clear. Let me try to generalise the example that you gave to check if I for if right:
\n\nIs it correct to say that the underlying problem lies in having high cardinality features (e.g. day) with fluctuations between categories (e.g. between days X and Y) that do not generalise well due to influence of factors of type 3 from your list? If yes, does it mean that coarser features (with lower cardinality), like quarter or weekday generally should be safer to avoid overfitting?
One question that is not clear to me is why do we not see signs of overfitting in CV. Or rather, what type of CV to use to be able to find this sort of overfitting? (For session level I see the ti?e-series split as an option, but what to do on user level?)
", "rawMarkdown": "@psilogram\nThanks for detailed explanation. This makes the situation with temporal features more clear. Let me try to generalise the example that you gave to check if I for if right:\n\n Is it correct to say that the underlying problem lies in having high cardinality features (e.g. `day`) with fluctuations between categories (e.g. between days X and Y) that do not generalise well due to influence of factors of type 3 from your list? If yes, does it mean that coarser features (with lower cardinality), like quarter or weekday generally should be safer to avoid overfitting? \n\nOne question that is not clear to me is why do we not see signs of overfitting in CV. Or rather, what type of CV to use to be able to find this sort of overfitting? (For session level I see the ti?e-series split as an option, but what to do on user level?)", "votes": 1 }, { "id": 399746, "postDate": "2018-10-06T16:33:15.770Z", "content": "Yes and no. The high cardinality itself isn't really the problem. It's the unknown factors. Suppose, for example, that in 2016 Google had a sale on swag every Monday, and then in August 2017 they changed the sale day to Tuesday. Including dayofweek as a feature would work great in the cv but fail miserably in the test set. (This is just an example -- I think that actually dayofweek is safe to use.)
\n\nUsing a feature like quarter raises additional problems. While it might capture coarse seasonality trends, it will work against longer-term trends (e.g., if purchase probability for a particular group is gradually increasing or decreasing across the entire train and test set period).
\n\nAs far as building a visitor-based CV system to guard against this kind of overfitting, you might be able to separate visitors by time based on their first or last sessions. You would still have some overlap for visitors whose visits span more than a couple months, but it might separate the folds enough to show which date features are overfitting. I haven't tried this yet so I would be interested to hear back from you if you do try it.
", "rawMarkdown": "Yes and no. The high cardinality itself isn't really the problem. It's the unknown factors. Suppose, for example, that in 2016 Google had a sale on swag every Monday, and then in August 2017 they changed the sale day to Tuesday. Including dayofweek as a feature would work great in the cv but fail miserably in the test set. (This is just an example -- I think that actually dayofweek is safe to use.)\n\nUsing a feature like quarter raises additional problems. While it might capture coarse seasonality trends, it will work against longer-term trends (e.g., if purchase probability for a particular group is gradually increasing or decreasing across the entire train and test set period). \n\nAs far as building a visitor-based CV system to guard against this kind of overfitting, you might be able to separate visitors by time based on their first or last sessions. You would still have some overlap for visitors whose visits span more than a couple months, but it might separate the folds enough to show which date features are overfitting. I haven't tried this yet so I would be interested to hear back from you if you do try it.", "votes": 3 }, { "id": 399783, "postDate": "2018-10-06T18:45:20.630Z", "content": "Many thanks. This sounds clear, but I will most likely come back to you, once i get to actual implementation :D
", "rawMarkdown": "Many thanks. This sounds clear, but I will most likely come back to you, once i get to actual implementation :D" }, { "id": 401119, "postDate": "2018-10-09T13:13:43.077Z", "content": "Thanks for the insights Silogram. Can you advise how are you handling the categorical variables grouping at a user level, for example if you have log ins from multiple browsers.
", "rawMarkdown": "Thanks for the insights Silogram. Can you advise how are you handling the categorical variables grouping at a user level, for example if you have log ins from multiple browsers." }, { "id": 401171, "postDate": "2018-10-09T14:49:04.627Z", "content": "That is a great idea actually, if you have code for calculating this I would be looking forward to seeing it :) share the love !
", "rawMarkdown": "That is a great idea actually, if you have code for calculating this I would be looking forward to seeing it :) share the love !" }, { "id": 401590, "postDate": "2018-10-10T10:26:57.933Z", "content": "I actually tried this, splitting the training set into 12 folds where each fold contains visitors whose last visit is in months 1-12. The results are really interesting, with CV scores ranging from 1.18 for month 11 to 1.86 for month 5. The CV scores are linearly aligned with the mean log transaction value (e.g., as the mean value rises per month, the cv also goes up). The big variance in monthly CV scores makes me feel better about the LB scores.
\n\nAnyway, I'll be off the grid for the next week or so. Good luck.
", "rawMarkdown": "I actually tried this, splitting the training set into 12 folds where each fold contains visitors whose last visit is in months 1-12. The results are really interesting, with CV scores ranging from 1.18 for month 11 to 1.86 for month 5. The CV scores are linearly aligned with the mean log transaction value (e.g., as the mean value rises per month, the cv also goes up). The big variance in monthly CV scores makes me feel better about the LB scores.\n\nAnyway, I'll be off the grid for the next week or so. Good luck.", "votes": 3 } ] }, { "id": 399173, "postDate": "2018-10-05T10:58:34.313Z", "content": "Hey Silogram, great insights thanks. Its really great and motivating for us. I just wonder what do you mean by visitor-level 5 folds cv ? Is it something like a groupKFold cv or just standard cv trained at sessions level? Thanks
", "rawMarkdown": "Hey Silogram, great insights thanks. Its really great and motivating for us. I just wonder what do you mean by visitor-level 5 folds cv ? Is it something like a groupKFold cv or just standard cv trained at sessions level? Thanks", "replies": [ { "id": 399215, "postDate": "2018-10-05T12:32:20.630Z", "content": "@Adri, I just mean normal kfold after aggregating sessions by fullVisitorId.
", "rawMarkdown": "@Adri, I just mean normal kfold after aggregating sessions by fullVisitorId.", "votes": 2 } ] }, { "id": 399137, "postDate": "2018-10-05T09:18:47.573Z", "content": "Thanks for the helpful insights Silogram. Do you have any idea about this specific issue as well? https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66774
", "rawMarkdown": "Thanks for the helpful insights Silogram. Do you have any idea about this specific issue as well? https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66774", "replies": [ { "id": 399150, "postDate": "2018-10-05T09:45:43.013Z", "content": "I haven't tested this by removing my hits-visits ratio features, so I can't really confirm this problem for my own models, but it's certainly mysterious and troubling.
", "rawMarkdown": "I haven't tested this by removing my hits-visits ratio features, so I can't really confirm this problem for my own models, but it's certainly mysterious and troubling. " }, { "id": 399269, "postDate": "2018-10-05T14:20:02.990Z", "content": "If you look at the ratio of (hits - page visits)/hits it becomes very strange towards the end of the test period. Maybe the have changed the technical specification of the site.
", "rawMarkdown": "If you look at the ratio of (hits - page visits)/hits it becomes very strange towards the end of the test period. Maybe the have changed the technical specification of the site.", "votes": 9 }, { "id": 399272, "postDate": "2018-10-05T14:23:25.083Z", "content": "Expect breakfast pirates shakeup stats to be epic!
", "rawMarkdown": "Expect breakfast pirates shakeup stats to be epic!", "votes": 2 }, { "id": 399334, "postDate": "2018-10-05T16:16:33.097Z", "content": "To update, I can confirm the same behavior that others have reported: removing all hits-views ratio features makes my cv score substantially worse (1.527 -> 1.539) but improves my LB score (1.408 -> 1.405). I think @narsil may be on to something.
", "rawMarkdown": "To update, I can confirm the same behavior that others have reported: removing all hits-views ratio features makes my cv score substantially worse (1.527 -> 1.539) but improves my LB score (1.408 -> 1.405). I think @narsil may be on to something.", "votes": 2 }, { "id": 399512, "postDate": "2018-10-06T00:25:47.927Z", "content": "Also confirming that by removing hit/pageview ratio features, CV gets worse by 0.015 but LB gets better by 0.007. But should we worry about overfitting to the LB?
", "rawMarkdown": "Also confirming that by removing hit/pageview ratio features, CV gets worse by 0.015 but LB gets better by 0.007. But should we worry about overfitting to the LB?" }, { "id": 399760, "postDate": "2018-10-06T17:43:09.367Z", "content": "I can confirm same effect: removing pageview/hit feature makes the validation score worse, but improves LB by almost 0.01.
", "rawMarkdown": "I can confirm same effect: removing pageview/hit feature makes the validation score worse, but improves LB by almost 0.01.", "votes": 3 } ] }, { "id": 399134, "postDate": "2018-10-05T09:13:24.727Z", "content": "@psilogram I really wondered how top LB positions were improving scores and might have figured out something unique. I agree on
\n\n\n\n\na much lower percentage of visitors with purchases. You can mimic this\n behavior by randomly eliminating about 20% of the visitors with\n purchases from your validation set. This should give you scores much\n closer to your LB scores
\n
Thanks for sharing this with us!
", "rawMarkdown": " @psilogram I really wondered how top LB positions were improving scores and might have figured out something unique. I agree on \n\n> a much lower percentage of visitors with purchases. You can mimic this\n> behavior by randomly eliminating about 20% of the visitors with\n> purchases from your validation set. This should give you scores much\n> closer to your LB scores\n\nThanks for sharing this with us!" }, { "id": 399129, "postDate": "2018-10-05T09:04:42.647Z", "content": "Thanks for sharing your insight. I totally agree with your 3 insights because I've tried to find time-independent features!
", "rawMarkdown": "Thanks for sharing your insight. I totally agree with your 3 insights because I've tried to find time-independent features!" }, { "id": 688425, "postDate": "2019-12-05T15:15:50.527Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3303973, "postDate": "2025-10-19T12:45:23.860Z", "content": "Great share!!!
", "rawMarkdown": "Great share!!! " }, { "id": 404515, "postDate": "2018-10-15T21:25:53.330Z", "content": "Interesting view. Thanks for sharing!
", "rawMarkdown": "Interesting view. Thanks for sharing!" }, { "id": 403266, "postDate": "2018-10-13T07:59:16.177Z", "content": "Thanks for sharing :D
", "rawMarkdown": "Thanks for sharing :D" }, { "id": 401558, "postDate": "2018-10-10T09:14:12.460Z", "content": "Thanks for sharing your insight.
", "rawMarkdown": "Thanks for sharing your insight." }, { "id": 401447, "postDate": "2018-10-10T05:02:51.163Z", "content": "Thanks for the insight Silogram !
", "rawMarkdown": "Thanks for the insight Silogram !" }, { "id": 400717, "postDate": "2018-10-08T19:08:02.177Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing" }, { "id": 399853, "postDate": "2018-10-06T23:55:40.913Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." }, { "id": 399831, "postDate": "2018-10-06T22:02:38.640Z", "content": "Thanks for the share
", "rawMarkdown": "Thanks for the share" } ], "comments": [ { "id": 400118, "author_name": "Xavier Bourret Sicotte", "author_url": "", "post_date": "2018-10-07T16:29:48.813000", "content": "Quick question: would a feature such as \"time to next visit\" be at risk of leakage ?
", "votes": 1, "replies": [ { "id": 400125, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-10-07T16:39:45.950000", "content": "Leakage - not necessarily (you are not using the target). That being said, it might get tricky in two \"edge\" cases: 1. user visits towards the end of the test set 2. if test set has more users with single visit (NA for such feature) than train.
", "votes": 2, "replies": [] } ] }, { "id": 400366, "author_name": "Meyk", "author_url": "", "post_date": "2018-10-08T07:16:17.640000", "content": "I have seen this topic before :D
\n\nThanks for sharing your hints!
", "votes": 2, "replies": [] }, { "id": 399170, "author_name": "Scirpus", "author_url": "", "post_date": "2018-10-05T10:55:43.250000", "content": "I got similar CV to LB by testing on np.log1p(revenues) <18.3
\n\nExplaining disparity between CV and the LB
", "votes": 2, "replies": [ { "id": 399214, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T12:31:22.507000", "content": "@scirpus, It's true that you could just remove visitors with the biggest values and then you would only need to remove about 1% instead of 20% to match cv to LB. But that seems sort of strange. It would make more sense to set a ceiling on the total transaction revenue. If you do this, the ceiling value is about 16.5. I think I tested some ceiling values previously and found that they don't help the LB score, but I'm not sure now whether the tests were before or after the metric correction. Has anyone tested this?
", "votes": 3, "replies": [] }, { "id": 399223, "author_name": "Scirpus", "author_url": "", "post_date": "2018-10-05T12:57:48.177000", "content": "Yeah I train on everything and just use the threshold as an additional scoring metric. Hopefully using this and the full if they both improve I am going in the right direction.
\n\nThe RMSE metric is a bit odd as we know < 0 is wrong and values between 0 and x are wrong so I am looking at Prob(positivescore)*Regression(between 10 and 25) combined to get the final RMSE - I can do this with genetic programming but I haven't a clue how to co-evolve this using LGBM or XGBoost.
\n\nThe ceiling thing sounds good as a few rare big values can skew the results quite badly
", "votes": 0, "replies": [] }, { "id": 399332, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T16:12:05.680000", "content": "To update, I just burnt a submission setting the ceiling at 17 and the LB score was slightly worse, so I don't think this is the issue.
", "votes": 3, "replies": [] }, { "id": 399400, "author_name": "GolemMaking", "author_url": "", "post_date": "2018-10-05T17:54:32.123000", "content": "Wish I had seen this post before burning one doing the same!
\n\nThanks for sharing though, I have more confidence in my results now
", "votes": 1, "replies": [] } ] }, { "id": 401767, "author_name": "", "author_url": "", "post_date": "2018-10-10T17:21:50.967000", "content": "Really appreciate your sharing! Great insights!
", "votes": 0, "replies": [] }, { "id": 400897, "author_name": "Jaylen A. Tingle", "author_url": "", "post_date": "2018-10-09T05:18:42.470000", "content": "Cool stuff!
", "votes": 0, "replies": [] }, { "id": 400271, "author_name": "Vamsi369", "author_url": "", "post_date": "2018-10-08T02:53:37.040000", "content": "There are few duplicates in the session Id, do you think they should be removed?
", "votes": 0, "replies": [ { "id": 400359, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-08T07:02:14.023000", "content": "They're not really duplicates -- they're sessions that start on one date and continue into the following day. I made a change in my code recently to merge them into a single session before further processing. It doesn't seem to have any effect on the LB score but it makes me feel better ;-).
", "votes": 5, "replies": [] }, { "id": 400367, "author_name": "AmirH", "author_url": "", "post_date": "2018-10-08T07:19:45.153000", "content": "@Silogram,\nI spotted some of them that have transactionRevenue for BOTH the rows that are the same session.\nThat's something i found to be very odd.\nboth of them having different transaction revenue value
", "votes": 1, "replies": [] }, { "id": 401586, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-10T10:20:08.403000", "content": "@AmirH: I haven't checked these but it could be that in a session crossing a day barrier a visitor makes two different purchases, one in day1 and the other in day2. When the session is split, it will generate sessions with the same ID but different transaction values. Just an idea...
\n\nBtw, I'll be off the grid for the next week or so. Good luck to everyone!
", "votes": 2, "replies": [] } ] }, { "id": 400195, "author_name": "Indranil Bhattacharya", "author_url": "", "post_date": "2018-10-07T19:55:40.167000", "content": "Hi Silogram, thanks a lot for sharing these important thoughts. Can you share some hints regarding how you do post-processing, for example I tried following -
\n\nBut, these are not that helpful.
", "votes": 0, "replies": [ { "id": 400355, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-08T06:56:44.177000", "content": "I tried a variety of linear transformations to probe the test set and get a better idea of how the test labels are distributed. But I'm no longer using this technique and don't really recommend it since it overfits to the public test set and may fail on the private set.
", "votes": 2, "replies": [] } ] }, { "id": 400031, "author_name": "Zhengzong", "author_url": "", "post_date": "2018-10-07T12:18:05.873000", "content": "Tanks for sharing . expected update
", "votes": 0, "replies": [] }, { "id": 399996, "author_name": "Kinimo", "author_url": "", "post_date": "2018-10-07T10:22:48.550000", "content": "Thanks for the insights Silogram. A basic question concerning point 1:
\n\nDoes google analytics have the revenue information of all shops subscribed to ga?\nIf the answer is \"no\" then could ga use the model (or insights derived from it) to estimate the \"value\" of a shop subscribed to ga? (However, of course a real future value of a shop would be more useful)
", "votes": 0, "replies": [ { "id": 400067, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-07T13:48:48.323000", "content": "@Kinimo, I don't think a model based on data from only one shop would generalize well to other shops.
", "votes": 0, "replies": [] } ] }, { "id": 399725, "author_name": "olivier", "author_url": "", "post_date": "2018-10-06T15:32:00.810000", "content": "Silogram, a big thank you for another good read of yours !
", "votes": 0, "replies": [] }, { "id": 399426, "author_name": "Usman Abbas", "author_url": "", "post_date": "2018-10-05T18:54:00.777000", "content": "Thank you so much Silogram :) I was eagerly waiting for this
", "votes": 0, "replies": [] }, { "id": 399355, "author_name": "TensorFrozen", "author_url": "", "post_date": "2018-10-05T16:53:41.590000", "content": "@Silogram, Thank you for sharing! I am wondering are you training your model based on session level or user level?
", "votes": 0, "replies": [ { "id": 399356, "author_name": "TensorFrozen", "author_url": "", "post_date": "2018-10-05T16:57:02.693000", "content": "And on the other hand...I am kind of afraid if the remaining test data has more users with purchase..... It will definitely a huge shake when the competition end.....@Silogram
", "votes": 0, "replies": [] }, { "id": 399371, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T17:18:16.670000", "content": "Both. I have three level-1 models. Two are session based and one is user-based. The final predictions are an ensemble of the three models.
\n\nI hate to say it, but I actually think the LB is more reliable than our local CV scores in this competition. The fact that LB scores are so far away from anything we can get with any kind of CV split suggests that the test set is much different from the train set. I guess it's always possible that Kaggle has split the public and private sets in some bizarre way that is causing this behavior, but I'm assuming that's not the case.
", "votes": 6, "replies": [] }, { "id": 399374, "author_name": "TensorFrozen", "author_url": "", "post_date": "2018-10-05T17:21:45.130000", "content": "Yeah...Have no choice but to trust the LB now...lol...
", "votes": 0, "replies": [] }, { "id": 399410, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-10-05T18:12:05.837000", "content": "Silogram, may I ask is your 1.526 CV comes from one of those three models or is it ensembled CV score?
", "votes": 2, "replies": [] }, { "id": 399413, "author_name": "fortune cookie", "author_url": "", "post_date": "2018-10-05T18:25:31.103000", "content": "Silogram, I think for user based model, aggregating train sessions to fit users' one year revenue is not comparable with aggregating test sessions to predict users' 9 month revenue on test. Is that right?
", "votes": 1, "replies": [] }, { "id": 399425, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T18:53:40.720000", "content": "@fatihöztürk Ensemble score.
", "votes": 1, "replies": [] }, { "id": 399427, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T18:56:00.003000", "content": "@dylan_chen: No, you can't just sum up the session predictions. Even if the test duration were the same as the train duration, simple summing wouldn't work very well.
", "votes": 2, "replies": [] }, { "id": 399657, "author_name": "", "author_url": "", "post_date": "2018-10-06T12:07:23.263000", "content": "", "votes": 0, "replies": [] }, { "id": 399899, "author_name": "choco", "author_url": "", "post_date": "2018-10-07T04:06:44.940000", "content": "thanks silogram. i have a question. how to use 2 session levels ..? same features different fold strategy? or using different features ?
", "votes": 0, "replies": [] }, { "id": 400071, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-07T13:54:19.210000", "content": "@hyun woo kim, Typically one would ensemble with many models varying all of the following:
\n\nThis competition is somewhat unusual inasmuch as it might also help to vary the fold strategy, but I haven't tried that yet.
", "votes": 5, "replies": [] }, { "id": 400072, "author_name": "choco", "author_url": "", "post_date": "2018-10-07T13:59:25.693000", "content": "Thanks for answering. i try it your answer. thanks for helping me :)
", "votes": 0, "replies": [] }, { "id": 400116, "author_name": "Paulo Felipe", "author_url": "", "post_date": "2018-10-07T16:17:50.210000", "content": "Silogram, first of all, thanks for your tips. I improved a lot after your post. I have doubts about your ensemble strategy. I tried stack 4 models (2 lgb user/session and 2 xgb user/session) using a linear model. My cv (5 folds) reduced almost 0.01 but my LB score was the same. After that, I tried a simple blending using 2 user level models. I set the weights based on LB scores (better lb more weight) and get an little improvement. Would you mind to share any insight? Thanks in advance!
", "votes": 2, "replies": [] }, { "id": 400288, "author_name": "James Chong", "author_url": "", "post_date": "2018-10-08T03:49:20.193000", "content": "@Silogram Not sure if my question makes sense. I wonder how do you ensemble your two level-1 session model if summing up session predictions is not an option.
\n\nOr are you using your user-based model to learn how to sum session predictions similar to this kernel?\nhttps://www.kaggle.com/ogrellier/teach-lightgbm-to-sum-predictions
", "votes": 1, "replies": [] }, { "id": 400353, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-08T06:52:39.203000", "content": "@Pauo Felipe & James Chong, To ensemble level-1 predictions, you can train a new model that uses the level-1 predictions as features. For the session-based models, you need to first generate features at the user level by aggregating the session predictions (e.g., mean, median, sum, max, min, etc.). Note that because leakage is a risk for this type of architecture, you need to be careful not to overfit the level-2 model.
", "votes": 5, "replies": [] }, { "id": 400399, "author_name": "fortune cookie", "author_url": "", "post_date": "2018-10-08T08:46:58.677000", "content": "@Silogram, Thanks for previous replying. As you said, I find the level-2 model overfit the level-1 predictions too much since I use level-1 predictions as features. And it is difficult to keep CV consitent with LB. So do you think it is more stable to use simple blending for this problem?
", "votes": 0, "replies": [] }, { "id": 400558, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-08T14:44:57.473000", "content": "@dylan_chen: Yes, aligning CV with LB is the biggest challenge in this competition. It's still a mystery to me...
", "votes": 0, "replies": [] }, { "id": 400617, "author_name": "Paulo Felipe", "author_url": "", "post_date": "2018-10-08T16:21:32.653000", "content": "Thanks, Silogram! I will try to find out a way to no overfit when training the 2-level model!
", "votes": 0, "replies": [] }, { "id": 402018, "author_name": "MLS", "author_url": "", "post_date": "2018-10-11T03:42:17.960000", "content": "Thanks Silogram for your great inputs.
\n\nFew follow up questions
\n\nThanks a ton in advance
", "votes": 0, "replies": [] } ] }, { "id": 399324, "author_name": "Misha Lisovyi", "author_url": "", "post_date": "2018-10-05T15:49:13.860000", "content": "I'm one of those \"lucky\" ones with potential data leakage. So far I struggle to understand how to trace down the source of the leak. Your third proposal sounds like something that I have not tried yet. Could you elaborate on it for the case of user-level models? I do not build visit-level models and at the visitor level time split is unclear to me.
", "votes": 0, "replies": [ { "id": 399340, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T16:21:43.510000", "content": "Yes, there's no way to do a time-based split at the user level, so you just need to remove all date-based features. If your model includes visitStartTime, for example, this will overfit badly. My models have just two date-based features -- hour and year. Note that at the user level, you can generate additional features based on visitStartTime that do not overfit. For example, the number of seconds between the first and last session is a valid feature.
", "votes": 11, "replies": [] }, { "id": 399350, "author_name": "Misha Lisovyi", "author_url": "", "post_date": "2018-10-05T16:36:15.933000", "content": "Thanks for the insights. They are very useful, as were those in the HomeCredit comp.
\n\nCould you elaborate on why time-related features are prong to overfitting. This is my first time series competition, so even basic issues are not apparent to me. By time-related features I do not mean exact date and not aggregates per date, but some relative times, like day of the week, number of days to closest Hanukkah, hour of the day, etc.
\n\nBtw, why year is not a problematic feature, i.e. why do you use it? Isn't year very different between train and test?
", "votes": 1, "replies": [] }, { "id": 399364, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T17:11:07.073000", "content": "If you look at a graph of the mean ratio of purchase to visit by day, you'll see that there's quite a bit of variation. This variation can have three different sources:
\n\nWe want our models to ignore #1 and #3 but take into account #2. The problem we have in this competition is that there's no way for the model to distinguish between #2 and #3. If the model can figure out which day a session belongs to, it will infer the probability of the user purchasing something on that day. This can be a strong feature and make the cv score improve, but the model can't generalize the feature to apply to dates in the test set. GBMs are very clever about combining features so if you include month and dayofweek, for example, the model can easily narrow down the day to one of 4 or 5 days of the year.
\n\nHour and dayofweek by themselves are probably OK, but days before Hannukah is a problem. To see why, suppose that in 2016, on one or more days before Hannukah, Google had a 50% sale. This would cause the purchase probability to increase on those days in the training set, and the model will take this into account. But if Google doesn't repeat the sale in 2017, the model will badly over-estimate purchases on these days in the test set. So again, the model can't 'know' whether the increase in sales occurred because a holiday was approaching (something that repeats regularly) or by an unpredictable factor (e.g., the sale).
\n\nRegarding the year feature, it's just a very imperfect way of including long-term trends in the model (e.g., is the purchase probability increasing or decreasing year over year). It's true that the year 2018 appears only in the test set so the model will treat it as being more like 2017 than 2016. It seems to help the cv and LB scores a little but it's not a strong feature.
", "votes": 22, "replies": [] }, { "id": 399600, "author_name": "HaimFeldman", "author_url": "", "post_date": "2018-10-06T07:56:25.463000", "content": "In 2016 Hanukkah and Christmas were both at December 25 But in 2017 Hanukkah was at December 13. I think this is the reason you get discrepancy between the revenues.
", "votes": 1, "replies": [] }, { "id": 399716, "author_name": "Misha Lisovyi", "author_url": "", "post_date": "2018-10-06T15:00:41.123000", "content": "@psilogram\nThanks for detailed explanation. This makes the situation with temporal features more clear. Let me try to generalise the example that you gave to check if I for if right:
\n\nIs it correct to say that the underlying problem lies in having high cardinality features (e.g. day) with fluctuations between categories (e.g. between days X and Y) that do not generalise well due to influence of factors of type 3 from your list? If yes, does it mean that coarser features (with lower cardinality), like quarter or weekday generally should be safer to avoid overfitting?
One question that is not clear to me is why do we not see signs of overfitting in CV. Or rather, what type of CV to use to be able to find this sort of overfitting? (For session level I see the ti?e-series split as an option, but what to do on user level?)
", "votes": 1, "replies": [] }, { "id": 399746, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-06T16:33:15.770000", "content": "Yes and no. The high cardinality itself isn't really the problem. It's the unknown factors. Suppose, for example, that in 2016 Google had a sale on swag every Monday, and then in August 2017 they changed the sale day to Tuesday. Including dayofweek as a feature would work great in the cv but fail miserably in the test set. (This is just an example -- I think that actually dayofweek is safe to use.)
\n\nUsing a feature like quarter raises additional problems. While it might capture coarse seasonality trends, it will work against longer-term trends (e.g., if purchase probability for a particular group is gradually increasing or decreasing across the entire train and test set period).
\n\nAs far as building a visitor-based CV system to guard against this kind of overfitting, you might be able to separate visitors by time based on their first or last sessions. You would still have some overlap for visitors whose visits span more than a couple months, but it might separate the folds enough to show which date features are overfitting. I haven't tried this yet so I would be interested to hear back from you if you do try it.
", "votes": 3, "replies": [] }, { "id": 399783, "author_name": "Misha Lisovyi", "author_url": "", "post_date": "2018-10-06T18:45:20.630000", "content": "Many thanks. This sounds clear, but I will most likely come back to you, once i get to actual implementation :D
", "votes": 0, "replies": [] }, { "id": 401119, "author_name": " MartinPetrov", "author_url": "", "post_date": "2018-10-09T13:13:43.077000", "content": "Thanks for the insights Silogram. Can you advise how are you handling the categorical variables grouping at a user level, for example if you have log ins from multiple browsers.
", "votes": 0, "replies": [] }, { "id": 401171, "author_name": "Xavier Bourret Sicotte", "author_url": "", "post_date": "2018-10-09T14:49:04.627000", "content": "That is a great idea actually, if you have code for calculating this I would be looking forward to seeing it :) share the love !
", "votes": 0, "replies": [] }, { "id": 401590, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-10T10:26:57.933000", "content": "I actually tried this, splitting the training set into 12 folds where each fold contains visitors whose last visit is in months 1-12. The results are really interesting, with CV scores ranging from 1.18 for month 11 to 1.86 for month 5. The CV scores are linearly aligned with the mean log transaction value (e.g., as the mean value rises per month, the cv also goes up). The big variance in monthly CV scores makes me feel better about the LB scores.
\n\nAnyway, I'll be off the grid for the next week or so. Good luck.
", "votes": 3, "replies": [] } ] }, { "id": 399173, "author_name": "Adri", "author_url": "", "post_date": "2018-10-05T10:58:34.313000", "content": "Hey Silogram, great insights thanks. Its really great and motivating for us. I just wonder what do you mean by visitor-level 5 folds cv ? Is it something like a groupKFold cv or just standard cv trained at sessions level? Thanks
", "votes": 0, "replies": [ { "id": 399215, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T12:32:20.630000", "content": "@Adri, I just mean normal kfold after aggregating sessions by fullVisitorId.
", "votes": 2, "replies": [] } ] }, { "id": 399137, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-10-05T09:18:47.573000", "content": "Thanks for the helpful insights Silogram. Do you have any idea about this specific issue as well? https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66774
", "votes": 0, "replies": [ { "id": 399150, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T09:45:43.013000", "content": "I haven't tested this by removing my hits-visits ratio features, so I can't really confirm this problem for my own models, but it's certainly mysterious and troubling.
", "votes": 0, "replies": [] }, { "id": 399269, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2018-10-05T14:20:02.990000", "content": "If you look at the ratio of (hits - page visits)/hits it becomes very strange towards the end of the test period. Maybe the have changed the technical specification of the site.
", "votes": 9, "replies": [] }, { "id": 399272, "author_name": "Scirpus", "author_url": "", "post_date": "2018-10-05T14:23:25.083000", "content": "Expect breakfast pirates shakeup stats to be epic!
", "votes": 2, "replies": [] }, { "id": 399334, "author_name": "Silogram", "author_url": "", "post_date": "2018-10-05T16:16:33.097000", "content": "To update, I can confirm the same behavior that others have reported: removing all hits-views ratio features makes my cv score substantially worse (1.527 -> 1.539) but improves my LB score (1.408 -> 1.405). I think @narsil may be on to something.
", "votes": 2, "replies": [] }, { "id": 399512, "author_name": "evers", "author_url": "", "post_date": "2018-10-06T00:25:47.927000", "content": "Also confirming that by removing hit/pageview ratio features, CV gets worse by 0.015 but LB gets better by 0.007. But should we worry about overfitting to the LB?
", "votes": 0, "replies": [] }, { "id": 399760, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-10-06T17:43:09.367000", "content": "I can confirm same effect: removing pageview/hit feature makes the validation score worse, but improves LB by almost 0.01.
", "votes": 3, "replies": [] } ] }, { "id": 399134, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-10-05T09:13:24.727000", "content": "@psilogram I really wondered how top LB positions were improving scores and might have figured out something unique. I agree on
\n\n\n\n\na much lower percentage of visitors with purchases. You can mimic this\n behavior by randomly eliminating about 20% of the visitors with\n purchases from your validation set. This should give you scores much\n closer to your LB scores
\n
Thanks for sharing this with us!
", "votes": 0, "replies": [] }, { "id": 399129, "author_name": "Youhan Lee", "author_url": "", "post_date": "2018-10-05T09:04:42.647000", "content": "Thanks for sharing your insight. I totally agree with your 3 insights because I've tried to find time-independent features!
", "votes": 0, "replies": [] }, { "id": 688425, "author_name": "", "author_url": "", "post_date": "2019-12-05T15:15:50.527000", "content": "", "votes": 0, "replies": [] }, { "id": 3303973, "author_name": "Muhammad Ehsan", "author_url": "", "post_date": "2025-10-19T12:45:23.860000", "content": "Great share!!!
", "votes": 0, "replies": [] }, { "id": 404515, "author_name": "Facundo Iannello", "author_url": "", "post_date": "2018-10-15T21:25:53.330000", "content": "Interesting view. Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 403266, "author_name": "Rom Uddamvathanak", "author_url": "", "post_date": "2018-10-13T07:59:16.177000", "content": "Thanks for sharing :D
", "votes": 0, "replies": [] }, { "id": 401558, "author_name": "我是小蚂蚁", "author_url": "", "post_date": "2018-10-10T09:14:12.460000", "content": "Thanks for sharing your insight.
", "votes": 0, "replies": [] }, { "id": 401447, "author_name": "Tung", "author_url": "", "post_date": "2018-10-10T05:02:51.163000", "content": "Thanks for the insight Silogram !
", "votes": 0, "replies": [] }, { "id": 400717, "author_name": "Abhigyan Shivam", "author_url": "", "post_date": "2018-10-08T19:08:02.177000", "content": "Thanks for sharing
", "votes": 0, "replies": [] }, { "id": 399853, "author_name": "Taiwo Ayanleye", "author_url": "", "post_date": "2018-10-06T23:55:40.913000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] }, { "id": 399831, "author_name": "Alaeddine Zekri", "author_url": "", "post_date": "2018-10-06T22:02:38.640000", "content": "Thanks for the share
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "399126": "I've been playing with this data for longer than I care to admit and it certainly poses some unique challenges. Here are a few insights I've come up with that may help others:\n\n1. I completely agree with everyone who has written that the competition has very little business value, both because of a poorly chosen metric and the ability to peek into the future. I'm treating it simply as a data puzzle rather than a business problem.\n\n2. I'm also seeing a large discrepancy in CV and LB scores. Currently my best model scores 1.526 on visitor-level 5-fold CV and 1.4084 on the LB (my best LB score [1.4065] comes from a little post-processing). Assuming no error in the metric, the only explanation for this that I can think of is that the test set contains a much lower percentage of visitors with purchases. You can mimic this behavior by randomly eliminating about 20% of the visitors with purchases from your validation set. This should give you scores much closer to your LB scores.\n\n3. Concerning CV and LB scores moving in opposite directions, I think this is mostly due to overfitting. As others have noted, any features that enable the model to distinguish one day from another are likely to overfit. This is a tough problem since there appear to be real seasonality differences (e.g., more purchases right before Christmas), but I have yet to figure out a way to reliably distinguish between real and spurious seasonality differences. One good way to eliminate date features that are causing overfitting is to perform feature selection with a time-based validation set (e.g., last 2 months). Any features that make the time-based CV score worse should be eliminated, even if they would help a kfold-based cv. In addition to the date features, there are likely other features that are causing overfitting due to differences in distribution between the train and sets. ", "400118": "Quick question: would a feature such as \"time to next visit\" be at risk of leakage ? ", "400366": "I have seen this topic before :D\n\nThanks for sharing your hints!", "399170": "I got similar CV to LB by testing on np.log1p(revenues) <18.3\n\n[Explaining disparity between CV and the LB][1]\n\n\n [1]: https://www.kaggle.com/scirpus/explaining-disparity-between-cv-and-the-lb", "401767": "Really appreciate your sharing! Great insights!", "400897": "Cool stuff!", "400271": "There are few duplicates in the session Id, do you think they should be removed?", "400195": "Hi Silogram, thanks a lot for sharing these important thoughts. Can you share some hints regarding how you do post-processing, for example I tried following -\n\n 1. Using missing value of bounces\n 2. Converting all predictions (before applying log1p) to their corresponding closest numbers multiple of 10000.\n\nBut, these are not that helpful.", "400031": "Tanks for sharing . expected update", "399996": "Thanks for the insights Silogram. A basic question concerning point 1:\n\nDoes google analytics have the revenue information of _all_ shops subscribed to ga?\nIf the answer is \"no\" then could ga use the model (or insights derived from it) to estimate the \"value\" of a shop subscribed to ga? (However, of course a real _future_ value of a shop would be more useful)", "399725": "Silogram, a big thank you for another good read of yours !", "399426": "Thank you so much Silogram :) I was eagerly waiting for this", "399355": "@Silogram, Thank you for sharing! I am wondering are you training your model based on session level or user level? ", "399324": "I'm one of those \"lucky\" ones with potential data leakage. So far I struggle to understand how to trace down the source of the leak. Your third proposal sounds like something that I have not tried yet. Could you elaborate on it for the case of user-level models? I do not build visit-level models and at the visitor level time split is unclear to me.", "399173": "Hey Silogram, great insights thanks. Its really great and motivating for us. I just wonder what do you mean by visitor-level 5 folds cv ? Is it something like a groupKFold cv or just standard cv trained at sessions level? Thanks", "399137": "Thanks for the helpful insights Silogram. Do you have any idea about this specific issue as well? https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66774", "399134": " @psilogram I really wondered how top LB positions were improving scores and might have figured out something unique. I agree on \n\n> a much lower percentage of visitors with purchases. You can mimic this\n> behavior by randomly eliminating about 20% of the visitors with\n> purchases from your validation set. This should give you scores much\n> closer to your LB scores\n\nThanks for sharing this with us!", "399129": "Thanks for sharing your insight. I totally agree with your 3 insights because I've tried to find time-independent features!", "688425": "", "3303973": "Great share!!! ", "404515": "Interesting view. Thanks for sharing!", "403266": "Thanks for sharing :D", "401558": "Thanks for sharing your insight.", "401447": "Thanks for the insight Silogram !", "400717": "Thanks for sharing", "399853": "Thanks for sharing.", "399831": "Thanks for the share" } }