Hi everyone,
\n\nWhile this is a really fun and interesting competition, I wonder if what we are trying to achieve makes business sense.
\n\nA lot of the data (e.g. number of visits, pageviews) we are using to predict consumer spend is only known after the fact. If the store has the data in the whole test set, then they already know how much a customer has spent.
\n\nIt might have been more useful to create a model that can be used with the data available up to when the model is used (e.g. at the beginning of a new visit), without knowing everything about future visits. However, it is understandable that this would be hard to set up on Kaggle.
\n\nThoughts?
", "messages": [ { "id": "391327", "postDate": "09/21/2018 15:56:23", "content": "Hi everyone,
\n\nWhile this is a really fun and interesting competition, I wonder if what we are trying to achieve makes business sense.
\n\nA lot of the data (e.g. number of visits, pageviews) we are using to predict consumer spend is only known after the fact. If the store has the data in the whole test set, then they already know how much a customer has spent.
\n\nIt might have been more useful to create a model that can be used with the data available up to when the model is used (e.g. at the beginning of a new visit), without knowing everything about future visits. However, it is understandable that this would be hard to set up on Kaggle.
\n\nThoughts?
", "rawMarkdown": "Hi everyone,\n\nWhile this is a really fun and interesting competition, I wonder if what we are trying to achieve makes business sense.\n\nA lot of the data (e.g. number of visits, pageviews) we are using to predict consumer spend is only known after the fact. If the store has the data in the whole test set, then they already know how much a customer has spent. \n\nIt might have been more useful to create a model that can be used with the data available up to when the model is used (e.g. at the beginning of a new visit), without knowing everything about future visits. However, it is understandable that this would be hard to set up on Kaggle.\n\nThoughts?", "votes": null }, { "id": "391378", "postDate": "09/21/2018 18:00:41", "content": "There are at least 3 ways to think about the application:
\n\n1) Funnel analysis (a user visiting home page -> product page -> payment page etc..),
\n\n2) Retention analysis (if an old paid user visits & pays again)
\n\n3) New customer acquisition (lookalike audience)
", "rawMarkdown": "There are at least 3 ways to think about the application:\n \n1) Funnel analysis (a user visiting home page -> product page -> payment page etc..), \n\n2) Retention analysis (if an old paid user visits & pays again)\n\n3) New customer acquisition (lookalike audience)", "votes": null }, { "id": "391412", "postDate": "09/21/2018 18:46:44", "content": "No, I think this is a toy competition with a prize. We pretty much know the entire future information, except for one variable. The evaluation metric is also a very weird one, and not something that you would use on this data if you actually had money riding on the result.
", "rawMarkdown": "No, I think this is a toy competition with a prize. We pretty much know the entire future information, except for one variable. The evaluation metric is also a very weird one, and not something that you would use on this data if you actually had money riding on the result.", "votes": null }, { "id": "391736", "postDate": "09/22/2018 10:11:29", "content": "I think it's a 'placeholder competition' - since the google's takeover we see a decline of non-google competition and a lot of artificial competitions created by google itself. Although it doesn't bother me - they even give us free coupons every now and then - it may suggest a deeper problem of the platform.
", "rawMarkdown": "I think it's a 'placeholder competition' - since the google's takeover we see a decline of non-google competition and a lot of artificial competitions created by google itself. Although it doesn't bother me - they even give us free coupons every now and then - it may suggest a deeper problem of the platform.", "votes": null }, { "id": "391854", "postDate": "09/22/2018 15:08:59", "content": "I hope the organizers will address this question, while the some useful learnings from the competition are possible, I agree with the question.
", "rawMarkdown": "I hope the organizers will address this question, while the some useful learnings from the competition are possible, I agree with the question.", "votes": null }, { "id": "391903", "postDate": "09/22/2018 17:09:32", "content": "A gap between what has been sold and what should have been sold based on the model may reveal a problem with the products visited, customer targeting, technology, etc.\nIt is a possible use a posteriori of these statistics.
", "rawMarkdown": "A gap between what has been sold and what should have been sold based on the model may reveal a problem with the products visited, customer targeting, technology, etc.\nIt is a possible use a posteriori of these statistics.", "votes": null }, { "id": "392882", "postDate": "09/24/2018 14:38:47", "content": "I think the default rule on Kaggle is \"anything goes\" and this sort of useless leakage has caused several competitions to be abandoned in the past.\n You're right that the winning model will not be implementable but if it's helpful, you could think of this competition as Google's way to save on their recruitment budget rather than an actually useful analysis.
", "rawMarkdown": "I think the default rule on Kaggle is \"anything goes\" and this sort of useless leakage has caused several competitions to be abandoned in the past.\n You're right that the winning model will not be implementable but if it's helpful, you could think of this competition as Google's way to save on their recruitment budget rather than an actually useful analysis.", "votes": null }, { "id": "393695", "postDate": "09/25/2018 19:06:12", "content": "In general in data science it is good to ask this question as thoughts about the business/domain may help with feature engineering even if the problem does make business sense. It is especially relevant to ask the question with time series data. What you are asking is a valid question, and, on the whole I agree with you.
\n\nHowever, thinking about it, here is a shot at a potential use case. Say that, for future data, a visit happens and does not produce revenue. Let's say at the end of the competition the winner's have produced a good model. That model could then be retrained with all data up to the present. So now the fit part of the model is done. The predict part of the model could be run immediately after any new visit after that point. Or it could be run periodically for new visits after that point e.g., every minute, hour, day etc. I will refer to all of those as near real-time rather than the months old predictions in this comp test set. Lets say that near real-time prediction was that the visit would produce significant revenue. Clearly, in a strict sense, the model was wrong on that specific prediction. But, it could be that the model detected something that made the visitor likely to make a purchase. Perhaps the user was undecided or got distracted. Maybe a timely follow-up ad or email might persuade the visitor to make that purchase after all. Likely it wouldn't be worth serving that ad if the prediction was that the user would spend 0.01 cents but would be worth serving the ad if the prediction was that they would spend $1000.
", "rawMarkdown": "In general in data science it is good to ask this question as thoughts about the business/domain may help with feature engineering even if the problem does make business sense. It is especially relevant to ask the question with time series data. What you are asking is a valid question, and, on the whole I agree with you.\n\nHowever, thinking about it, here is a shot at a potential use case. Say that, for future data, a visit happens and does not produce revenue. Let's say at the end of the competition the winner's have produced a good model. That model could then be retrained with all data up to the present. So now the fit part of the model is done. The predict part of the model could be run immediately after any new visit after that point. Or it could be run periodically for new visits after that point e.g., every minute, hour, day etc. I will refer to all of those as near real-time rather than the months old predictions in this comp test set. Lets say that near real-time prediction was that the visit would produce significant revenue. Clearly, in a strict sense, the model was wrong on that specific prediction. But, it could be that the model detected something that made the visitor likely to make a purchase. Perhaps the user was undecided or got distracted. Maybe a timely follow-up ad or email might persuade the visitor to make that purchase after all. Likely it wouldn't be worth serving that ad if the prediction was that the user would spend 0.01 cents but would be worth serving the ad if the prediction was that they would spend $1000.", "votes": null }, { "id": "394042", "postDate": "09/26/2018 08:55:45", "content": "Well elaboration of points on how a model could be potentially used in future case which I agree with
", "rawMarkdown": "Well elaboration of points on how a model could be potentially used in future case which I agree with", "votes": null }, { "id": "396263", "postDate": "09/30/2018 10:58:41", "content": "\"The predict part of the model could be run immediately after any new visit after that point\". Except you can't, the winning model is most like to use future information as an input.
", "rawMarkdown": "\"The predict part of the model could be run immediately after any new visit after that point\". Except you can't, the winning model is most like to use future information as an input.", "votes": null }, { "id": "396333", "postDate": "09/30/2018 13:49:02", "content": "> Except you can't, the winning model is most like to use future information as an input.
\n\nI expressed in my post that I shared the concerns raised by the original post over the nature of this competition and was generally in agreement with them. Time series competitions often work well when the results unfold over time after the end date of the competition as in the Web Traffic competition last year. That way no future information leaks accidental or intentional can take place.
\n\nHowever, the organisers have made a prize fund of $45,000 available for this competition. So, presumably they think that there is something worthwhile to be gathered from the results. I am describing a potential business case for that since the original question was about the business sense of this competition. Whatever future leaks there may or may not be in this data, or in the way that models are constructed, the organisers will still have to try to extract some value for their prize money, plus their time investment. Taking both session level models and user level models, I will describe how I think the organisers could implement a winning model without using future leaks.
\n\nThe competition ends mid November 2018. Let's say the organisers spend a while going through the results. By late January 2019 they are ready to retrain the winning model (or a combination of high-placed models).
\n\nFor a session-level predicting model they could:
\n\nA user-level model prediction could be made with periodic runs, also in the way I described. Again using 31st January 2019 as an example, the organisers would:
\n\nThere are at least 3 ways to think about the application:
\n\n1) Funnel analysis (a user visiting home page -> product page -> payment page etc..),
\n\n2) Retention analysis (if an old paid user visits & pays again)
\n\n3) New customer acquisition (lookalike audience)
", "votes": null, "replies": [] }, { "id": 391412, "author_name": "dmitriyguller", "author_url": "", "post_date": "09/21/2018 18:46:44", "content": "No, I think this is a toy competition with a prize. We pretty much know the entire future information, except for one variable. The evaluation metric is also a very weird one, and not something that you would use on this data if you actually had money riding on the result.
", "votes": null, "replies": [] }, { "id": 391736, "author_name": "adamsfei", "author_url": "", "post_date": "09/22/2018 10:11:29", "content": "I think it's a 'placeholder competition' - since the google's takeover we see a decline of non-google competition and a lot of artificial competitions created by google itself. Although it doesn't bother me - they even give us free coupons every now and then - it may suggest a deeper problem of the platform.
", "votes": null, "replies": [] }, { "id": 391854, "author_name": "chinta", "author_url": "", "post_date": "09/22/2018 15:08:59", "content": "I hope the organizers will address this question, while the some useful learnings from the competition are possible, I agree with the question.
", "votes": null, "replies": [] }, { "id": 391903, "author_name": "emorej", "author_url": "", "post_date": "09/22/2018 17:09:32", "content": "A gap between what has been sold and what should have been sold based on the model may reveal a problem with the products visited, customer targeting, technology, etc.\nIt is a possible use a posteriori of these statistics.
", "votes": null, "replies": [] }, { "id": 392882, "author_name": "desk165", "author_url": "", "post_date": "09/24/2018 14:38:47", "content": "I think the default rule on Kaggle is \"anything goes\" and this sort of useless leakage has caused several competitions to be abandoned in the past.\n You're right that the winning model will not be implementable but if it's helpful, you could think of this competition as Google's way to save on their recruitment budget rather than an actually useful analysis.
", "votes": null, "replies": [] }, { "id": 393695, "author_name": "datahobbit", "author_url": "", "post_date": "09/25/2018 19:06:12", "content": "In general in data science it is good to ask this question as thoughts about the business/domain may help with feature engineering even if the problem does make business sense. It is especially relevant to ask the question with time series data. What you are asking is a valid question, and, on the whole I agree with you.
\n\nHowever, thinking about it, here is a shot at a potential use case. Say that, for future data, a visit happens and does not produce revenue. Let's say at the end of the competition the winner's have produced a good model. That model could then be retrained with all data up to the present. So now the fit part of the model is done. The predict part of the model could be run immediately after any new visit after that point. Or it could be run periodically for new visits after that point e.g., every minute, hour, day etc. I will refer to all of those as near real-time rather than the months old predictions in this comp test set. Lets say that near real-time prediction was that the visit would produce significant revenue. Clearly, in a strict sense, the model was wrong on that specific prediction. But, it could be that the model detected something that made the visitor likely to make a purchase. Perhaps the user was undecided or got distracted. Maybe a timely follow-up ad or email might persuade the visitor to make that purchase after all. Likely it wouldn't be worth serving that ad if the prediction was that the user would spend 0.01 cents but would be worth serving the ad if the prediction was that they would spend $1000.
", "votes": null, "replies": [ { "id": 394042, "author_name": "admond1994", "author_url": "", "post_date": "09/26/2018 08:55:45", "content": "Well elaboration of points on how a model could be potentially used in future case which I agree with
", "votes": null, "replies": [] }, { "id": 396263, "author_name": "louistian", "author_url": "", "post_date": "09/30/2018 10:58:41", "content": "\"The predict part of the model could be run immediately after any new visit after that point\". Except you can't, the winning model is most like to use future information as an input.
", "votes": null, "replies": [] }, { "id": 396333, "author_name": "datahobbit", "author_url": "", "post_date": "09/30/2018 13:49:02", "content": "> Except you can't, the winning model is most like to use future information as an input.
\n\nI expressed in my post that I shared the concerns raised by the original post over the nature of this competition and was generally in agreement with them. Time series competitions often work well when the results unfold over time after the end date of the competition as in the Web Traffic competition last year. That way no future information leaks accidental or intentional can take place.
\n\nHowever, the organisers have made a prize fund of $45,000 available for this competition. So, presumably they think that there is something worthwhile to be gathered from the results. I am describing a potential business case for that since the original question was about the business sense of this competition. Whatever future leaks there may or may not be in this data, or in the way that models are constructed, the organisers will still have to try to extract some value for their prize money, plus their time investment. Taking both session level models and user level models, I will describe how I think the organisers could implement a winning model without using future leaks.
\n\nThe competition ends mid November 2018. Let's say the organisers spend a while going through the results. By late January 2019 they are ready to retrain the winning model (or a combination of high-placed models).
\n\nFor a session-level predicting model they could:
\n\nA user-level model prediction could be made with periodic runs, also in the way I described. Again using 31st January 2019 as an example, the organisers would:
\n\n