![\"R²](\"http://i67.tinypic.com/28tg6x1.png\")
As competition final results are not known, it's also hard to evaluate which approach will turn out to be the best and which is worth sharing. But anyway - I don't think I'll remember all the details of this competition after 2 month, so it would be interesting to hear some thoughts already now, what was your approach?
\n\nSome quick thoughts from my side:
\n\nGood luck!
", "messages": [ { "id": "430948", "postDate": "12/01/2018 10:02:59", "content": "As competition final results are not known, it's also hard to evaluate which approach will turn out to be the best and which is worth sharing. But anyway - I don't think I'll remember all the details of this competition after 2 month, so it would be interesting to hear some thoughts already now, what was your approach?
\n\nSome quick thoughts from my side:
\n\nGood luck!
", "rawMarkdown": "As competition final results are not known, it's also hard to evaluate which approach will turn out to be the best and which is worth sharing. But anyway - I don't think I'll remember all the details of this competition after 2 month, so it would be interesting to hear some thoughts already now, what was your approach?\n\nSome quick thoughts from my side:\n\n 1. I think this is quite a big lottery, given we have to guess only\n couple of returning visitors with purchases, so I didn't spent too much time on it and only built one LightGBM model.\n 2. I exported some basic data from Demo account for gap 15.oct. - 30.nov. But it is hard to match Demo account data 100% with given train/test data so having Demo account data only in test set actually hurts the model. That's why I had to fill gaps with Demo account exports also for train data.\n 3. Regarding features the most important ones was:\n - Time in days since the last visit;\n - Total number of purchases in the last year;\n - Total number of purchases in the last 1.5 month from Demo account;\n - Statistics on different event types (e.g. if customer has added something in basket, there is a chance he will buy it sooner or later);\n - Some others...\n 4. This metric is sensitive to scaling so I scaled final predictions up/down for my final submissions - one to be more conservative just to beat all-zero and second to be more risky and to fight for higher places in case of luck.\n\nGood luck!", "votes": null }, { "id": "431044", "postDate": "12/01/2018 14:14:07", "content": "Thanks for this. I also didn't spend that much time on this competition for the same reason. Looking on Google Analytics demo account there were 2 big falls in total monthly revenue across all customers. Last year, and earlier this year, monthly revenue was around $500,000 then it fell to a stable average of around $120,000 for 4 months consecutively then it fell again so in November it was $3269.78. I found this in the Google Analytics Demo Account - Conversions - E-Commerce - Overview then change the dates and choose the value by month. I think there could be a mistake in the way that the data is reported. As you said, it is really just guesswork. Going forward, we don't know if new marketing campaigns are going to start or hot new products will be released.
\n\nI didn't do everything that you described but what I did different to you was:\n+to sum revenue in consecutive 2 month windows per user going back as far as possible. This led to 11 2-month periods which I then modelled using LSTM. \n+After that I also calculated some features such as the count of periods in which there was revenue and added the last recorded value for some categoricals then modelled again mainly using LGBM. \n+I also ran this as a classification problem and then used the output of that as an input to regression.
\n\nI tried to do a relatively conservative option to beat all zeros but in hindsight I could have been more conservative than I was. Time will tell I guess.
", "rawMarkdown": "Thanks for this. I also didn't spend that much time on this competition for the same reason. Looking on Google Analytics demo account there were 2 big falls in total monthly revenue across all customers. Last year, and earlier this year, monthly revenue was around $500,000 then it fell to a stable average of around $120,000 for 4 months consecutively then it fell again so in November it was $3269.78. I found this in the Google Analytics Demo Account - Conversions - E-Commerce - Overview then change the dates and choose the value by month. I think there could be a mistake in the way that the data is reported. As you said, it is really just guesswork. Going forward, we don't know if new marketing campaigns are going to start or hot new products will be released.\n\nI didn't do everything that you described but what I did different to you was:\n+to sum revenue in consecutive 2 month windows per user going back as far as possible. This led to 11 2-month periods which I then modelled using LSTM. \n+After that I also calculated some features such as the count of periods in which there was revenue and added the last recorded value for some categoricals then modelled again mainly using LGBM. \n+I also ran this as a classification problem and then used the output of that as an input to regression.\n\nI tried to do a relatively conservative option to beat all zeros but in hindsight I could have been more conservative than I was. Time will tell I guess.", "votes": null }, { "id": "431129", "postDate": "12/01/2018 16:50:41", "content": "I had the hardest time finding the motivation to do anything in this competition, but on the last day I decided to put in a couple of lottery tickets.
\n\nMy modeling dataset approach seems to be the one that everyone else converged to: create a bunch of non-overlapping slices of two-month totals, created only for users who shopped in a 165-day window 45 days before the slice begins. I did not get any data from the demo account.
\n\nGiven that in this modeling dataset there are only ~800 positive target values, I gambled on foregoing machine learning approaches. In my experience, machine learning can fall on its face when the data starts getting extremely sparse and skewed, and hopefully this will be the case here. I instead built a simple GAM model using mgcv package in R, and wound up using about 10 predictors (nothing fancy at all, just the usual suspects).
\n\nI'm sure I could've done a lot more, though I'm not sure whether doing more would start overfitting by feature engineering.
", "rawMarkdown": "I had the hardest time finding the motivation to do anything in this competition, but on the last day I decided to put in a couple of lottery tickets. \n\nMy modeling dataset approach seems to be the one that everyone else converged to: create a bunch of non-overlapping slices of two-month totals, created only for users who shopped in a 165-day window 45 days before the slice begins. I did not get any data from the demo account.\n\nGiven that in this modeling dataset there are only ~800 positive target values, I gambled on foregoing machine learning approaches. In my experience, machine learning can fall on its face when the data starts getting extremely sparse and skewed, and hopefully this will be the case here. I instead built a simple GAM model using mgcv package in R, and wound up using about 10 predictors (nothing fancy at all, just the usual suspects).\n\nI'm sure I could've done a lot more, though I'm not sure whether doing more would start overfitting by feature engineering.", "votes": null }, { "id": "431646", "postDate": "12/02/2018 17:05:15", "content": "Ok, so I used R. Here's my source code (and the data prep step).
\n\nThis was my first competition, and it took some time to get the basics right. Luckily I only joined in the v2 phase. As others have mentioned, the data was very sparse, and I found it very hard to beat the baseline (predict the historic mean) by any margin. However, I was able to get reasonably consistent results with a suitable validation strategy:
\n\ncut_offs = ymd(\"2017-01-15\") %m+% months(0:17) # earliest we can start and have 5.5 months data (to make a consistent recent visitor pool)\n\ntrain_cut_offs = cut_offs[cut_offs < max(cut_offs) %m-% months(2)] \n\nvalid_cut_offs = train_cut_offs %m+% months(2)\n\ntrain_sets = lapply(train_cut_offs, function(t) which(fm$cut_off %within% interval(t %m-% months(12), t))) \n\nvalid_sets = lapply(valid_cut_offs, function(t) which(fm$cut_off == t))\nIn other words, I had a cut off and new forecast every month, 18 times. I then built 18 sets of features using an expanding window, from the data’s origin to the cut off. This worked better for me than a rolling window. More on why later. The cut off is the analog to the 15th October 2018: the point at which the training data stops. I did not use the demo account to infill 16th-31st October, as I was trying to infill all the missing data, but of course @alijs execution above is much smarter.
\n\nFor cross-validation, I would train each model with the feature set from the cut off month and the 11 months before, so I had a rolling window of 12 feature sets. (Note, each feature set is built from an expanding window, so they overlap.) This was an important step to augment the data. There’s so few conversions of recent visitors in any 2 month period, it’s hard to train a model of any complexity using a single cut off. I chose 12 months somewhat arbitrarily, as 5 or 8 months worked well too, arguing to myself it would smooth out any seasonality.
\n\nThe metric I used for cross-validation was R². It scales the MSE vs the baseline: predict the historic mean. I felt this was helpful, as the MSE moved quite a bit over time, and this helped me understand why: the mean moved a lot, presumably due to promotional activity (see @MichaelP's comments). As you can see from this chart, the model was way off last Spring, so my entry relies on this promotional activity (or lack thereof) not happening again this Winter!
\n\n
I am willing to bet the winning entry will incorporate both short term knowledge of promotional activity, and long term general behaviour of visitors, especially repeat purchasers.
\n\nIn terms of features, I took inspiration from Buy ‘Til You Die models. They have sufficient statistics of: the number of purchases, and the time since the first and the last purchase. This is why I used an expanding window: these models rely on having the entire history of transactions to determine whether the customer is still alive: they may not have transacted the last 6 months, but this could be just because they tend to purchase infrequently, say once every 8 months on average. These sufficient statistics were the main time features I used, both for visits and transactions (ie all visits, and those visits with positive transactionRevenue). I also added average pageviews per visit and average transaction revenue (ie sum of transactionRevenue over count of transactions).
I mean-encoded a few of the more useful categorical variables, but not using target encoding. The target was too sparse for this, so I encoded the historical conversion rate instead (total transactions / total visits for the specific value of the categorical).
\n\nDuring the competition, I thought I had found an important data leak. In the v2 train data, the organisers had removed the source column for a large number of visits. Comparing this to the v1 data, I could see these were consistently identifying internal websites for Google employees. I reasoned that Google employees were much more likely to transact, which is why they had redacted them in this way. So I added the redacted source back to the data set as a feature, for these visits, as well as an indicator. In practice, they did not improve the model's performance once I included all my time features, described above, so I removed them.
To speed up my development cycle, I rebalanced the “classes”. I took the whole history and removed 90% of the non-transacting visitors. For the final submission I used the full set, of course. I say \"classes\" as I treated the comp as a regression, not a classification as others have mentioned. My reasoning was the positive cases were very sparse, so building two models was pushing it!
\n\nI used Catboost as it outperformed LightGBM, despite the fact that I had mean-encoded all the categoricals before handing the data to the model (Catboost specialises in mean-encoding categoricals, amongst other things). I didn’t try Xgboost. I didn’t ensemble or stack — I haven’t had time to learn this skill yet!
\n\nI also totally failed to pick 2 submissions for the private leaderboard! I thought I had to do that after stage 1 finished, but I now realise it’s something you need to do before, to avoid gaming. Anyway, luckily I’d only ever made 3 submissions, and the second highest scoring one (on the stage 1 leaderboard) is the one I wanted to submit, and it will go through automatically. As Mr Miyagi would say, “You beginner luck!”
", "rawMarkdown": "Ok, so I used R. Here's my [source code][1] (and the [data prep step][2]). \n\nThis was my first competition, and it took some time to get the basics right. Luckily I only joined in the v2 phase. As others have mentioned, the data was very sparse, and I found it very hard to beat the baseline (predict the historic mean) by any margin. However, I was able to get reasonably consistent results with a suitable **validation strategy**:\n\n cut_offs = ymd(\"2017-01-15\") %m+% months(0:17) # earliest we can start and have 5.5 months data (to make a consistent recent visitor pool)\n \n train_cut_offs = cut_offs[cut_offs < max(cut_offs) %m-% months(2)] \n \n valid_cut_offs = train_cut_offs %m+% months(2)\n\n train_sets = lapply(train_cut_offs, function(t) which(fm$cut_off %within% interval(t %m-% months(12), t))) \n \n valid_sets = lapply(valid_cut_offs, function(t) which(fm$cut_off == t))\n\nIn other words, I had a cut off and new forecast every month, 18 times. I then built 18 sets of features using an expanding window, from the data’s origin to the cut off. This worked better for me than a rolling window. More on why later. The cut off is the analog to the 15th October 2018: the point at which the training data stops. I did not use the demo account to infill 16th-31st October, as I was trying to infill *all* the missing data, but of course @alijs execution above is much smarter.\n\nFor cross-validation, I would train each model with the feature set from the cut off month and the 11 months before, so I had a rolling window of 12 feature sets. (Note, each feature set is built from an expanding window, so they overlap.) This was an important step to **augment the data**. There’s so few conversions of recent visitors in any 2 month period, it’s hard to train a model of any complexity using a single cut off. I chose 12 months somewhat arbitrarily, as 5 or 8 months worked well too, arguing to myself it would smooth out any seasonality.\n\nThe **metric** I used for cross-validation was R². It scales the MSE vs the baseline: predict the historic mean. I felt this was helpful, as the MSE moved quite a bit over time, and this helped me understand why: the mean moved a lot, presumably due to promotional activity (see @MichaelP's comments). As you can see from this chart, the model was way off last Spring, so my entry relies on this promotional activity (or lack thereof) not happening again this Winter!\n\n![R² by target interval end dates (cut off plus 3.5 months)][3]\n\nI am willing to bet the winning entry will incorporate both short term knowledge of promotional activity, and long term general behaviour of visitors, especially repeat purchasers.\n\nIn terms of **features**, I took inspiration from *[Buy ‘Til You Die][4]* models. They have [sufficient statistics][5] of: the number of purchases, and the time since the first and the last purchase. This is why I used an expanding window: these models rely on having the entire history of transactions to determine whether the customer is still alive: they may not have transacted the last 6 months, but this could be just because they tend to purchase infrequently, say once every 8 months on average. These sufficient statistics were the main time features I used, both for visits and transactions (ie all visits, and those visits with positive `transactionRevenue`). I also added average pageviews per visit and average transaction revenue (ie sum of `transactionRevenue` over count of transactions).\n\nI **mean-encoded** a few of the more useful categorical variables, but *not* using target encoding. The target was too sparse for this, so I encoded the historical conversion rate instead (total transactions / total visits for the specific value of the categorical).\n\nDuring the competition, I thought I had found an important **data leak**. In the v2 train data, the organisers had removed the `source` column for a large number of visits. Comparing this to the v1 data, I could see these were consistently identifying internal websites for Google employees. I reasoned that Google employees were much more likely to transact, which is why they had redacted them in this way. So I added the redacted `source` back to the data set as a feature, for these visits, as well as an indicator. In practice, they did not improve the model's performance once I included all my time features, described above, so I removed them.\n\nTo speed up my development cycle, I **rebalanced** the “classes”. I took the whole history and removed 90% of the non-transacting visitors. For the final submission I used the full set, of course. I say \"classes\" as I treated the comp as a regression, not a classification as others have mentioned. My reasoning was the positive cases were very sparse, so building two models was pushing it!\n\nI used **Catboost** as it outperformed LightGBM, despite the fact that I had mean-encoded all the categoricals before handing the data to the model (Catboost specialises in mean-encoding categoricals, amongst other things). I didn’t try Xgboost. I didn’t ensemble or stack — I haven’t had time to learn this skill yet!\n\nI also totally failed to pick 2 submissions for the private leaderboard! I thought I had to do that after stage 1 finished, but I now realise it’s something you need to do before, to avoid gaming. Anyway, luckily I’d only ever made 3 submissions, and the second highest scoring one (on the stage 1 leaderboard) is the one I wanted to submit, and it will go through automatically. As **Mr Miyagi** would say, [“You beginner luck!”][6]\n\n\n [1]: https://www.kaggle.com/burritodan/gstore-2-final\n [2]: https://www.kaggle.com/burritodan/gstore-1-flatten\n [3]: http://i67.tinypic.com/28tg6x1.png\n [4]: http://www.brucehardie.com/papers.html\n [5]: https://en.wikipedia.org/wiki/Sufficient_statistic\n [6]: https://www.youtube.com/watch?v=J1gAHil89Z4", "votes": null }, { "id": "432012", "postDate": "12/03/2018 08:50:32", "content": "@Dmitriy I would love to see your GAM model. I recently read up on the gamboost package in R and thought it looked cool. With some partial dependence plots, I could see some of my features had exponential decay in their response, or other curves that might be easy to fit, which might be a promising start for an additive model. But, I didn't have time to look into the feature interactions, which if complex, could confound an additive model.
![\"Features](\"http://i68.tinypic.com/b88vh5.png\")
@Dmitriy I think your assessment of the applicability of machine learning methods is quite fair. At least in my case the results of LGBM were quite terrible, due to the extremely low number of positive labels. I tried to combat that with formulating it as a classification problem and then only using rows with positive predictions for regression, but this didn't help much. Neither did class weights or changing cost functions.\nIn the end I conclude that I spent way too much time on feature engineering, whereas I should have looked more into outlier detection methods.
", "rawMarkdown": "[@Dmitriy](https://www.kaggle.com/dmitriyguller) I think your assessment of the applicability of machine learning methods is quite fair. At least in my case the results of LGBM were quite terrible, due to the extremely low number of positive labels. I tried to combat that with formulating it as a classification problem and then only using rows with positive predictions for regression, but this didn't help much. Neither did class weights or changing cost functions.\nIn the end I conclude that I spent way too much time on feature engineering, whereas I should have looked more into outlier detection methods.", "votes": null }, { "id": "432128", "postDate": "12/03/2018 12:44:42", "content": "Thank you @Burrito Dan for sharing your approach and code. Our approaches are somewhat similar. I am wondering if you checked the performance of your model for a non-rebalanced dataset?\nIn my case, undersampling the non-purchasing users led to much better performance in training/validation. However, using that model on a complete holdout set for predictions performed abysmally.
", "rawMarkdown": "Thank you [@Burrito Dan](https://www.kaggle.com/burritodan) for sharing your approach and code. Our approaches are somewhat similar. I am wondering if you checked the performance of your model for a non-rebalanced dataset?\nIn my case, undersampling the non-purchasing users led to much better performance in training/validation. However, using that model on a complete holdout set for predictions performed abysmally.", "votes": null }, { "id": "432623", "postDate": "12/04/2018 05:33:42", "content": "@motirzbach thanks! Yes, undersampling improved the performance on the validation sets. For the submission, I retrained the model on the full sample though. Although this performed less well than the undersampled version, it was happy with the same features, keeping them roughly in the same order in my early tests of this. I have to say, I didn't check this consistently throughout development, nor for the final submission...
\n\nFor the final submission:
\n\nRMSE 0.302 +/- 0.0714\nRsquared 0.0303 +/- 0.016\nFor the undersampled set (use SUBMISSION = FALSE in my code):
RMSE 1.77 +/- 0.352\nRsquared 0.0766 +/- 0.0233\nThe metrics above are the mean and SD for the validation strategy I described in my post. Note, Caret calculates Rsquared using correlation (I used the default).
", "rawMarkdown": "motirzbach thanks! Yes, undersampling improved the performance on the validation sets. For the submission, I retrained the model on the full sample though. Although this performed less well than the undersampled version, it was happy with the same features, keeping them roughly in the same order in my early tests of this. I have to say, I didn't check this consistently throughout development, nor for the final submission...\n\nFor the final submission:\n\n RMSE 0.302 +/- 0.0714\n Rsquared 0.0303 +/- 0.016\n\nFor the undersampled set (use `SUBMISSION = FALSE` in my code):\n\n RMSE 1.77 +/- 0.352\n Rsquared 0.0766 +/- 0.0233\n\nThe metrics above are the mean and SD for the validation strategy I described in my post. Note, Caret [calculates Rsquared using correlation][1] (I used the default).\n\n\n [1]: https://stats.stackexchange.com/questions/230913/how-caret-calculates-r-squared", "votes": null } ], "comments": [ { "id": 431044, "author_name": "datahobbit", "author_url": "", "post_date": "12/01/2018 14:14:07", "content": "Thanks for this. I also didn't spend that much time on this competition for the same reason. Looking on Google Analytics demo account there were 2 big falls in total monthly revenue across all customers. Last year, and earlier this year, monthly revenue was around $500,000 then it fell to a stable average of around $120,000 for 4 months consecutively then it fell again so in November it was $3269.78. I found this in the Google Analytics Demo Account - Conversions - E-Commerce - Overview then change the dates and choose the value by month. I think there could be a mistake in the way that the data is reported. As you said, it is really just guesswork. Going forward, we don't know if new marketing campaigns are going to start or hot new products will be released.
\n\nI didn't do everything that you described but what I did different to you was:\n+to sum revenue in consecutive 2 month windows per user going back as far as possible. This led to 11 2-month periods which I then modelled using LSTM. \n+After that I also calculated some features such as the count of periods in which there was revenue and added the last recorded value for some categoricals then modelled again mainly using LGBM. \n+I also ran this as a classification problem and then used the output of that as an input to regression.
\n\nI tried to do a relatively conservative option to beat all zeros but in hindsight I could have been more conservative than I was. Time will tell I guess.
", "votes": null, "replies": [] }, { "id": 431129, "author_name": "dmitriyguller", "author_url": "", "post_date": "12/01/2018 16:50:41", "content": "I had the hardest time finding the motivation to do anything in this competition, but on the last day I decided to put in a couple of lottery tickets.
\n\nMy modeling dataset approach seems to be the one that everyone else converged to: create a bunch of non-overlapping slices of two-month totals, created only for users who shopped in a 165-day window 45 days before the slice begins. I did not get any data from the demo account.
\n\nGiven that in this modeling dataset there are only ~800 positive target values, I gambled on foregoing machine learning approaches. In my experience, machine learning can fall on its face when the data starts getting extremely sparse and skewed, and hopefully this will be the case here. I instead built a simple GAM model using mgcv package in R, and wound up using about 10 predictors (nothing fancy at all, just the usual suspects).
\n\nI'm sure I could've done a lot more, though I'm not sure whether doing more would start overfitting by feature engineering.
", "votes": null, "replies": [ { "id": 432012, "author_name": "burritodan", "author_url": "", "post_date": "12/03/2018 08:50:32", "content": "@Dmitriy I would love to see your GAM model. I recently read up on the gamboost package in R and thought it looked cool. With some partial dependence plots, I could see some of my features had exponential decay in their response, or other curves that might be easy to fit, which might be a promising start for an additive model. But, I didn't have time to look into the feature interactions, which if complex, could confound an additive model.
@Dmitriy I think your assessment of the applicability of machine learning methods is quite fair. At least in my case the results of LGBM were quite terrible, due to the extremely low number of positive labels. I tried to combat that with formulating it as a classification problem and then only using rows with positive predictions for regression, but this didn't help much. Neither did class weights or changing cost functions.\nIn the end I conclude that I spent way too much time on feature engineering, whereas I should have looked more into outlier detection methods.
", "votes": null, "replies": [] } ] }, { "id": 431646, "author_name": "burritodan", "author_url": "", "post_date": "12/02/2018 17:05:15", "content": "Ok, so I used R. Here's my source code (and the data prep step).
\n\nThis was my first competition, and it took some time to get the basics right. Luckily I only joined in the v2 phase. As others have mentioned, the data was very sparse, and I found it very hard to beat the baseline (predict the historic mean) by any margin. However, I was able to get reasonably consistent results with a suitable validation strategy:
\n\ncut_offs = ymd(\"2017-01-15\") %m+% months(0:17) # earliest we can start and have 5.5 months data (to make a consistent recent visitor pool)\n\ntrain_cut_offs = cut_offs[cut_offs < max(cut_offs) %m-% months(2)] \n\nvalid_cut_offs = train_cut_offs %m+% months(2)\n\ntrain_sets = lapply(train_cut_offs, function(t) which(fm$cut_off %within% interval(t %m-% months(12), t))) \n\nvalid_sets = lapply(valid_cut_offs, function(t) which(fm$cut_off == t))\nIn other words, I had a cut off and new forecast every month, 18 times. I then built 18 sets of features using an expanding window, from the data’s origin to the cut off. This worked better for me than a rolling window. More on why later. The cut off is the analog to the 15th October 2018: the point at which the training data stops. I did not use the demo account to infill 16th-31st October, as I was trying to infill all the missing data, but of course @alijs execution above is much smarter.
\n\nFor cross-validation, I would train each model with the feature set from the cut off month and the 11 months before, so I had a rolling window of 12 feature sets. (Note, each feature set is built from an expanding window, so they overlap.) This was an important step to augment the data. There’s so few conversions of recent visitors in any 2 month period, it’s hard to train a model of any complexity using a single cut off. I chose 12 months somewhat arbitrarily, as 5 or 8 months worked well too, arguing to myself it would smooth out any seasonality.
\n\nThe metric I used for cross-validation was R². It scales the MSE vs the baseline: predict the historic mean. I felt this was helpful, as the MSE moved quite a bit over time, and this helped me understand why: the mean moved a lot, presumably due to promotional activity (see @MichaelP's comments). As you can see from this chart, the model was way off last Spring, so my entry relies on this promotional activity (or lack thereof) not happening again this Winter!
\n\n
I am willing to bet the winning entry will incorporate both short term knowledge of promotional activity, and long term general behaviour of visitors, especially repeat purchasers.
\n\nIn terms of features, I took inspiration from Buy ‘Til You Die models. They have sufficient statistics of: the number of purchases, and the time since the first and the last purchase. This is why I used an expanding window: these models rely on having the entire history of transactions to determine whether the customer is still alive: they may not have transacted the last 6 months, but this could be just because they tend to purchase infrequently, say once every 8 months on average. These sufficient statistics were the main time features I used, both for visits and transactions (ie all visits, and those visits with positive transactionRevenue). I also added average pageviews per visit and average transaction revenue (ie sum of transactionRevenue over count of transactions).
I mean-encoded a few of the more useful categorical variables, but not using target encoding. The target was too sparse for this, so I encoded the historical conversion rate instead (total transactions / total visits for the specific value of the categorical).
\n\nDuring the competition, I thought I had found an important data leak. In the v2 train data, the organisers had removed the source column for a large number of visits. Comparing this to the v1 data, I could see these were consistently identifying internal websites for Google employees. I reasoned that Google employees were much more likely to transact, which is why they had redacted them in this way. So I added the redacted source back to the data set as a feature, for these visits, as well as an indicator. In practice, they did not improve the model's performance once I included all my time features, described above, so I removed them.
To speed up my development cycle, I rebalanced the “classes”. I took the whole history and removed 90% of the non-transacting visitors. For the final submission I used the full set, of course. I say \"classes\" as I treated the comp as a regression, not a classification as others have mentioned. My reasoning was the positive cases were very sparse, so building two models was pushing it!
\n\nI used Catboost as it outperformed LightGBM, despite the fact that I had mean-encoded all the categoricals before handing the data to the model (Catboost specialises in mean-encoding categoricals, amongst other things). I didn’t try Xgboost. I didn’t ensemble or stack — I haven’t had time to learn this skill yet!
\n\nI also totally failed to pick 2 submissions for the private leaderboard! I thought I had to do that after stage 1 finished, but I now realise it’s something you need to do before, to avoid gaming. Anyway, luckily I’d only ever made 3 submissions, and the second highest scoring one (on the stage 1 leaderboard) is the one I wanted to submit, and it will go through automatically. As Mr Miyagi would say, “You beginner luck!”
", "votes": null, "replies": [ { "id": 432128, "author_name": "moritzbach", "author_url": "", "post_date": "12/03/2018 12:44:42", "content": "Thank you @Burrito Dan for sharing your approach and code. Our approaches are somewhat similar. I am wondering if you checked the performance of your model for a non-rebalanced dataset?\nIn my case, undersampling the non-purchasing users led to much better performance in training/validation. However, using that model on a complete holdout set for predictions performed abysmally.
", "votes": null, "replies": [] }, { "id": 432623, "author_name": "burritodan", "author_url": "", "post_date": "12/04/2018 05:33:42", "content": "@motirzbach thanks! Yes, undersampling improved the performance on the validation sets. For the submission, I retrained the model on the full sample though. Although this performed less well than the undersampled version, it was happy with the same features, keeping them roughly in the same order in my early tests of this. I have to say, I didn't check this consistently throughout development, nor for the final submission...
\n\nFor the final submission:
\n\nRMSE 0.302 +/- 0.0714\nRsquared 0.0303 +/- 0.016\nFor the undersampled set (use SUBMISSION = FALSE in my code):
RMSE 1.77 +/- 0.352\nRsquared 0.0766 +/- 0.0233\nThe metrics above are the mean and SD for the validation strategy I described in my post. Note, Caret calculates Rsquared using correlation (I used the default).
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "430948": "As competition final results are not known, it's also hard to evaluate which approach will turn out to be the best and which is worth sharing. But anyway - I don't think I'll remember all the details of this competition after 2 month, so it would be interesting to hear some thoughts already now, what was your approach?\n\nSome quick thoughts from my side:\n\n 1. I think this is quite a big lottery, given we have to guess only\n couple of returning visitors with purchases, so I didn't spent too much time on it and only built one LightGBM model.\n 2. I exported some basic data from Demo account for gap 15.oct. - 30.nov. But it is hard to match Demo account data 100% with given train/test data so having Demo account data only in test set actually hurts the model. That's why I had to fill gaps with Demo account exports also for train data.\n 3. Regarding features the most important ones was:\n - Time in days since the last visit;\n - Total number of purchases in the last year;\n - Total number of purchases in the last 1.5 month from Demo account;\n - Statistics on different event types (e.g. if customer has added something in basket, there is a chance he will buy it sooner or later);\n - Some others...\n 4. This metric is sensitive to scaling so I scaled final predictions up/down for my final submissions - one to be more conservative just to beat all-zero and second to be more risky and to fight for higher places in case of luck.\n\nGood luck!", "431044": "Thanks for this. I also didn't spend that much time on this competition for the same reason. Looking on Google Analytics demo account there were 2 big falls in total monthly revenue across all customers. Last year, and earlier this year, monthly revenue was around $500,000 then it fell to a stable average of around $120,000 for 4 months consecutively then it fell again so in November it was $3269.78. I found this in the Google Analytics Demo Account - Conversions - E-Commerce - Overview then change the dates and choose the value by month. I think there could be a mistake in the way that the data is reported. As you said, it is really just guesswork. Going forward, we don't know if new marketing campaigns are going to start or hot new products will be released.\n\nI didn't do everything that you described but what I did different to you was:\n+to sum revenue in consecutive 2 month windows per user going back as far as possible. This led to 11 2-month periods which I then modelled using LSTM. \n+After that I also calculated some features such as the count of periods in which there was revenue and added the last recorded value for some categoricals then modelled again mainly using LGBM. \n+I also ran this as a classification problem and then used the output of that as an input to regression.\n\nI tried to do a relatively conservative option to beat all zeros but in hindsight I could have been more conservative than I was. Time will tell I guess.", "431129": "I had the hardest time finding the motivation to do anything in this competition, but on the last day I decided to put in a couple of lottery tickets. \n\nMy modeling dataset approach seems to be the one that everyone else converged to: create a bunch of non-overlapping slices of two-month totals, created only for users who shopped in a 165-day window 45 days before the slice begins. I did not get any data from the demo account.\n\nGiven that in this modeling dataset there are only ~800 positive target values, I gambled on foregoing machine learning approaches. In my experience, machine learning can fall on its face when the data starts getting extremely sparse and skewed, and hopefully this will be the case here. I instead built a simple GAM model using mgcv package in R, and wound up using about 10 predictors (nothing fancy at all, just the usual suspects).\n\nI'm sure I could've done a lot more, though I'm not sure whether doing more would start overfitting by feature engineering.", "431646": "Ok, so I used R. Here's my [source code][1] (and the [data prep step][2]). \n\nThis was my first competition, and it took some time to get the basics right. Luckily I only joined in the v2 phase. As others have mentioned, the data was very sparse, and I found it very hard to beat the baseline (predict the historic mean) by any margin. However, I was able to get reasonably consistent results with a suitable **validation strategy**:\n\n cut_offs = ymd(\"2017-01-15\") %m+% months(0:17) # earliest we can start and have 5.5 months data (to make a consistent recent visitor pool)\n \n train_cut_offs = cut_offs[cut_offs < max(cut_offs) %m-% months(2)] \n \n valid_cut_offs = train_cut_offs %m+% months(2)\n\n train_sets = lapply(train_cut_offs, function(t) which(fm$cut_off %within% interval(t %m-% months(12), t))) \n \n valid_sets = lapply(valid_cut_offs, function(t) which(fm$cut_off == t))\n\nIn other words, I had a cut off and new forecast every month, 18 times. I then built 18 sets of features using an expanding window, from the data’s origin to the cut off. This worked better for me than a rolling window. More on why later. The cut off is the analog to the 15th October 2018: the point at which the training data stops. I did not use the demo account to infill 16th-31st October, as I was trying to infill *all* the missing data, but of course @alijs execution above is much smarter.\n\nFor cross-validation, I would train each model with the feature set from the cut off month and the 11 months before, so I had a rolling window of 12 feature sets. (Note, each feature set is built from an expanding window, so they overlap.) This was an important step to **augment the data**. There’s so few conversions of recent visitors in any 2 month period, it’s hard to train a model of any complexity using a single cut off. I chose 12 months somewhat arbitrarily, as 5 or 8 months worked well too, arguing to myself it would smooth out any seasonality.\n\nThe **metric** I used for cross-validation was R². It scales the MSE vs the baseline: predict the historic mean. I felt this was helpful, as the MSE moved quite a bit over time, and this helped me understand why: the mean moved a lot, presumably due to promotional activity (see @MichaelP's comments). As you can see from this chart, the model was way off last Spring, so my entry relies on this promotional activity (or lack thereof) not happening again this Winter!\n\n![R² by target interval end dates (cut off plus 3.5 months)][3]\n\nI am willing to bet the winning entry will incorporate both short term knowledge of promotional activity, and long term general behaviour of visitors, especially repeat purchasers.\n\nIn terms of **features**, I took inspiration from *[Buy ‘Til You Die][4]* models. They have [sufficient statistics][5] of: the number of purchases, and the time since the first and the last purchase. This is why I used an expanding window: these models rely on having the entire history of transactions to determine whether the customer is still alive: they may not have transacted the last 6 months, but this could be just because they tend to purchase infrequently, say once every 8 months on average. These sufficient statistics were the main time features I used, both for visits and transactions (ie all visits, and those visits with positive `transactionRevenue`). I also added average pageviews per visit and average transaction revenue (ie sum of `transactionRevenue` over count of transactions).\n\nI **mean-encoded** a few of the more useful categorical variables, but *not* using target encoding. The target was too sparse for this, so I encoded the historical conversion rate instead (total transactions / total visits for the specific value of the categorical).\n\nDuring the competition, I thought I had found an important **data leak**. In the v2 train data, the organisers had removed the `source` column for a large number of visits. Comparing this to the v1 data, I could see these were consistently identifying internal websites for Google employees. I reasoned that Google employees were much more likely to transact, which is why they had redacted them in this way. So I added the redacted `source` back to the data set as a feature, for these visits, as well as an indicator. In practice, they did not improve the model's performance once I included all my time features, described above, so I removed them.\n\nTo speed up my development cycle, I **rebalanced** the “classes”. I took the whole history and removed 90% of the non-transacting visitors. For the final submission I used the full set, of course. I say \"classes\" as I treated the comp as a regression, not a classification as others have mentioned. My reasoning was the positive cases were very sparse, so building two models was pushing it!\n\nI used **Catboost** as it outperformed LightGBM, despite the fact that I had mean-encoded all the categoricals before handing the data to the model (Catboost specialises in mean-encoding categoricals, amongst other things). I didn’t try Xgboost. I didn’t ensemble or stack — I haven’t had time to learn this skill yet!\n\nI also totally failed to pick 2 submissions for the private leaderboard! I thought I had to do that after stage 1 finished, but I now realise it’s something you need to do before, to avoid gaming. Anyway, luckily I’d only ever made 3 submissions, and the second highest scoring one (on the stage 1 leaderboard) is the one I wanted to submit, and it will go through automatically. As **Mr Miyagi** would say, [“You beginner luck!”][6]\n\n\n [1]: https://www.kaggle.com/burritodan/gstore-2-final\n [2]: https://www.kaggle.com/burritodan/gstore-1-flatten\n [3]: http://i67.tinypic.com/28tg6x1.png\n [4]: http://www.brucehardie.com/papers.html\n [5]: https://en.wikipedia.org/wiki/Sufficient_statistic\n [6]: https://www.youtube.com/watch?v=J1gAHil89Z4", "432012": "Dmitriy I would love to see your GAM model. I recently read up on the `gamboost` package in R and thought it looked cool. With some partial dependence plots, I could see some of my features had exponential decay in their response, or other curves that might be easy to fit, which might be a promising start for an additive model. But, I didn't have time to look into the feature interactions, which if complex, could confound an additive model.\n\n![Features 1][1]\n\n [1]: http://i68.tinypic.com/b88vh5.png", "432050": "[@Dmitriy](https://www.kaggle.com/dmitriyguller) I think your assessment of the applicability of machine learning methods is quite fair. At least in my case the results of LGBM were quite terrible, due to the extremely low number of positive labels. I tried to combat that with formulating it as a classification problem and then only using rows with positive predictions for regression, but this didn't help much. Neither did class weights or changing cost functions.\nIn the end I conclude that I spent way too much time on feature engineering, whereas I should have looked more into outlier detection methods.", "432128": "Thank you [@Burrito Dan](https://www.kaggle.com/burritodan) for sharing your approach and code. Our approaches are somewhat similar. I am wondering if you checked the performance of your model for a non-rebalanced dataset?\nIn my case, undersampling the non-purchasing users led to much better performance in training/validation. However, using that model on a complete holdout set for predictions performed abysmally.", "432623": "motirzbach thanks! Yes, undersampling improved the performance on the validation sets. For the submission, I retrained the model on the full sample though. Although this performed less well than the undersampled version, it was happy with the same features, keeping them roughly in the same order in my early tests of this. I have to say, I didn't check this consistently throughout development, nor for the final submission...\n\nFor the final submission:\n\n RMSE 0.302 +/- 0.0714\n Rsquared 0.0303 +/- 0.016\n\nFor the undersampled set (use `SUBMISSION = FALSE` in my code):\n\n RMSE 1.77 +/- 0.352\n Rsquared 0.0766 +/- 0.0233\n\nThe metrics above are the mean and SD for the validation strategy I described in my post. Note, Caret [calculates Rsquared using correlation][1] (I used the default).\n\n\n [1]: https://stats.stackexchange.com/questions/230913/how-caret-calculates-r-squared" }, "source": "meta" }