Among many other things, one thing that makes Kaggle great is that we get to using real-life data to solve a real-life problem.
\n\nHowever, due to the way this competition is framed, there is a fundamental temporal feature leakage problem embedded in this competition that makes it very unrealistic. If I want to predict a user's spending on day n. Knowing whether and when a user will come back in a later time is obviously useful. However, in the real-life situation, you would not know any future web traffic in advance because, well, it's in the future. Competitors are therefore forced to use this leaked feature in their solution.
If we look at this problem at a different angle, the test data includes all of the web traffic data for the next n days. If we already know the web traffic, then we would have to know any transactions already in real life. Then what the point of predicting things that are already known?
UPDATE: \nJust saw the Two Sigma competition. This competition should have been done in the same way as a kernel only competition to avoid leakage.
", "messages": [ { "id": "396089", "postDate": "09/30/2018 02:25:19", "content": "Among many other things, one thing that makes Kaggle great is that we get to using real-life data to solve a real-life problem.
\n\nHowever, due to the way this competition is framed, there is a fundamental temporal feature leakage problem embedded in this competition that makes it very unrealistic. If I want to predict a user's spending on day n. Knowing whether and when a user will come back in a later time is obviously useful. However, in the real-life situation, you would not know any future web traffic in advance because, well, it's in the future. Competitors are therefore forced to use this leaked feature in their solution.
If we look at this problem at a different angle, the test data includes all of the web traffic data for the next n days. If we already know the web traffic, then we would have to know any transactions already in real life. Then what the point of predicting things that are already known?
UPDATE: \nJust saw the Two Sigma competition. This competition should have been done in the same way as a kernel only competition to avoid leakage.
", "rawMarkdown": "Among many other things, one thing that makes Kaggle great is that we get to using real-life data to solve a real-life problem. \n\nHowever, due to the way this competition is framed, there is a fundamental temporal feature leakage problem embedded in this competition that makes it very unrealistic. If I want to predict a user's spending on day `n`. Knowing whether and when a user will come back in a later time is obviously useful. However, in the real-life situation, you would not know any future web traffic in advance because, well, it's in the future. Competitors are therefore forced to use this **leaked** feature in their solution. \n\nIf we look at this problem at a different angle, the test data includes all of the web traffic data for the next `n` days. If we already know the web traffic, then we would have to know any transactions already in real life. Then what the point of predicting things that are already known?\n\nUPDATE: \nJust saw the Two Sigma competition. This competition should have been done in the same way as a kernel only competition to avoid leakage.", "votes": null }, { "id": "396108", "postDate": "09/30/2018 02:57:27", "content": "I don't think the competition hosts care if it's unrealistic. Recall the goal of this competition:
\n\n\n\n\nRStudio, the developer of free and open tools for R and enterprise-ready products for teams to scale and share work, has partnered with Google Cloud and Kaggle to demonstrate the business impact that thorough data analysis can have.
\n
They're looking for people to prove that data analysis can improve ML performance; their goal is not to look for the best model and then implement it in real life. I think this is a toy problem to see the upper limits of data analysis. One big reason I believe they're doing this is to prove that R is very useful in data analysis, which can then encourage people to use RStudio, as opposed to using Python Jupyter notebook for example. RStudio can cite this competition when trying to sell their enterprise products.
\n\nTherefore I agree that this competition is unrealistic, but I don't think the competition hosts care about that
", "rawMarkdown": "I don't think the competition hosts care if it's unrealistic. Recall the goal of this competition:\n\n> RStudio, the developer of free and open tools for R and enterprise-ready products for teams to scale and share work, has partnered with Google Cloud and Kaggle to demonstrate the business impact that thorough data analysis can have.\n\nThey're looking for people to prove that data analysis can improve ML performance; their goal is *not* to look for the best model and then implement it in real life. I think this is a toy problem to see the upper limits of data analysis. One big reason I believe they're doing this is to prove that R is very useful in data analysis, which can then encourage people to use RStudio, as opposed to using Python Jupyter notebook for example. RStudio can cite this competition when trying to sell their enterprise products. \n\nTherefore I agree that this competition is unrealistic, but I don't think the competition hosts care about that", "votes": null }, { "id": "396120", "postDate": "09/30/2018 03:19:24", "content": "I'm with you. Also relevant is the fact that so many fields are \"not available in demo dataset\". Any website owner who agrees to share data with Google Analytics can get metrics/features that we don't have here. Landing pages, ad key words, and so on. I don't know how much these things matter, but it does seem that this competition is a spinoff of how we would help a company analyze web purchases in the real world.
", "rawMarkdown": "I'm with you. Also relevant is the fact that so many fields are \"not available in demo dataset\". Any website owner who agrees to share data with Google Analytics can get metrics/features that we don't have here. Landing pages, ad key words, and so on. I don't know how much these things matter, but it does seem that this competition is a spinoff of how we would help a company analyze web purchases in the real world.", "votes": null }, { "id": "396123", "postDate": "09/30/2018 03:25:06", "content": "@CoreyLevinson
\n\nThanks for your comments.
\n\nI believe the best way to demonstrate the business impact that thorough data analysis and the machine learning is to use it to solve a real-life problem.
\n\nA flawed toy question does the opposite. I use RStudio's product myself so it is not my intention to offend anyone. So apologies in advance. But at least for me, have a question like this shows a lack of understanding of the fundamental of machine learning.
", "rawMarkdown": "CoreyLevinson\n\nThanks for your comments.\n \nI believe the best way to demonstrate the business impact that thorough data analysis and the machine learning is to use it to solve a real-life problem. \n\nA flawed toy question does the opposite. I use RStudio's product myself so it is not my intention to offend anyone. So apologies in advance. But at least for me, have a question like this shows a lack of understanding of the fundamental of machine learning.", "votes": null }, { "id": "396124", "postDate": "09/30/2018 03:31:07", "content": "I totally agree, but maybe RStudio did not have a problem they wanted to solve, or did not have access to a reliable dataset, and therefore kindly asked Google if they had any random dataset to practice a competition on, to which Google responded with the Merchandise Store dataset.
\n\nMaybe a better competition would be a database of users who visit RStudio website, how many of them download the software, how many of them make an enterprise purchase, etc.
\n\nTo your last sentence: that's up for the customer to decide. I think you can still have intense data analysis in a flawed problem. I don't think you can use this competition to cite LightGBM or xgboost or NN as very strong models, because it's not a real-life problem, but the data analysis is still valid, in my opinion.
", "rawMarkdown": "I totally agree, but maybe RStudio did not have a problem they wanted to solve, or did not have access to a reliable dataset, and therefore kindly asked Google if they had any random dataset to practice a competition on, to which Google responded with the Merchandise Store dataset.\n\nMaybe a better competition would be a database of users who visit RStudio website, how many of them download the software, how many of them make an enterprise purchase, etc.\n\nTo your last sentence: that's up for the customer to decide. I think you can still have intense data analysis in a flawed problem. I don't think you can use this competition to cite LightGBM or xgboost or NN as very strong models, because it's not a real-life problem, but the data analysis is still valid, in my opinion.", "votes": null }, { "id": "396126", "postDate": "09/30/2018 03:40:49", "content": "I have no particular issue with the dataset. My problem is with the question.
\n\nA much better question would be given only the historical sessions information and ask to predict the users' transactional value in the next n day. In another word, no session information in the test dataset.
\n\nI am not even asking for more or different information. All I am saying is less information should be disclosed in the first place.
", "rawMarkdown": "I have no particular issue with the dataset. My problem is with the question. \n\nA much better question would be given only the historical sessions information and ask to predict the users' transactional value in the next n day. In another word, no session information in the test dataset. \n\nI am not even asking for more or different information. All I am saying is less information should be disclosed in the first place.", "votes": null }, { "id": "396200", "postDate": "09/30/2018 08:10:30", "content": "It has been discussed before here
", "rawMarkdown": "It has been discussed before [here][1]\n\n\n [1]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66440", "votes": null }, { "id": "396245", "postDate": "09/30/2018 10:13:01", "content": "This is a problem that I have encountered many times (at work and in Kaggle competitions).
\n\nOne possible solution is to only use features that are available at the time of the sample.
\n\nFor example, if you know the web activity of a person in 2018-08-10 then you can use all possible features up to that date (but nothing in the future) even though future information is available in the training dataset.
\n\nAlso, using sklearn TimeSeriesSplit is a good practice. Also avoid shuffling your train data so that leakage isn't introduced.
\n\nFinally, some models are sequential by construction: RNNs for example and \"classic\" statistical methods (ARIMA et al).
\n\nThe Two Sigma approach is indeed great.
\n\nI hope this helps!
", "rawMarkdown": "This is a problem that I have encountered many times (at work and in Kaggle competitions). \n\nOne possible solution is to only use features **that are available at the time of the sample**. \n\nFor example, if you know the web activity of a person in 2018-08-10 then you can use all possible features **up to that date** (but nothing in the future) even though future information is available in the training dataset. \n\nAlso, using sklearn [TimeSeriesSplit][1] is a good practice. Also avoid shuffling your train data so that leakage isn't introduced. \n\nFinally, some models are sequential by construction: [RNNs][2] for example and \"classic\" statistical methods ([ARIMA][3] et al).\n\nThe Two Sigma approach is indeed great.\n\nI hope this helps!\n\n\n [1]: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html.\n [2]: https://en.wikipedia.org/wiki/Recurrent_neural_network\n [3]: https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average", "votes": null }, { "id": "396267", "postDate": "09/30/2018 10:59:18", "content": "Thanks for the pointer:)
", "rawMarkdown": "Thanks for the pointer:)", "votes": null }, { "id": "396268", "postDate": "09/30/2018 11:02:15", "content": "@yassine yes, we could constrain ourselves to only use data available up-to-date but the winning solution is most likely to use future information because it is made available to us in the test set.
", "rawMarkdown": "yassine yes, we could constrain ourselves to only use data available up-to-date but the winning solution is most likely to use future information because it is made available to us in the test set.", "votes": null }, { "id": "396270", "postDate": "09/30/2018 11:07:33", "content": "I was on fact think about using RNN. But how do you incorporate future information in RNN for the win?
", "rawMarkdown": "I was on fact think about using RNN. But how do you incorporate future information in RNN for the win?", "votes": null }, { "id": "396277", "postDate": "09/30/2018 11:24:17", "content": "That's for sure. I was thinking about models that will perform better in the \"real world\" and not necessary win.
\n\nFor RNNs, I am a novice and haven't used them a lot but I think you can use a \"two-way\" RNN. One possibility is bi-directional LSTM: this article is a nice explanation.
\n\nGood luck. :)
", "rawMarkdown": "That's for sure. I was thinking about models that will perform better in the \"real world\" and not necessary win. \n\nFor RNNs, I am a novice and haven't used them a lot but I think you can use a \"two-way\" RNN. One possibility is bi-directional LSTM: this [article][1] is a nice explanation. \n\nGood luck. :)\n\n [1]: https://machinelearningmastery.com/develop-bidirectional-lstm-sequence-classification-python-keras/", "votes": null }, { "id": "398382", "postDate": "10/04/2018 03:29:39", "content": "Can not agree more! The best model of this competition is very like to be useless in real world.
", "rawMarkdown": "Can not agree more! The best model of this competition is very like to be useless in real world.", "votes": null } ], "comments": [ { "id": 396108, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/30/2018 02:57:27", "content": "I don't think the competition hosts care if it's unrealistic. Recall the goal of this competition:
\n\n\n\n\nRStudio, the developer of free and open tools for R and enterprise-ready products for teams to scale and share work, has partnered with Google Cloud and Kaggle to demonstrate the business impact that thorough data analysis can have.
\n
They're looking for people to prove that data analysis can improve ML performance; their goal is not to look for the best model and then implement it in real life. I think this is a toy problem to see the upper limits of data analysis. One big reason I believe they're doing this is to prove that R is very useful in data analysis, which can then encourage people to use RStudio, as opposed to using Python Jupyter notebook for example. RStudio can cite this competition when trying to sell their enterprise products.
\n\nTherefore I agree that this competition is unrealistic, but I don't think the competition hosts care about that
", "votes": null, "replies": [ { "id": 396123, "author_name": "louistian", "author_url": "", "post_date": "09/30/2018 03:25:06", "content": "@CoreyLevinson
\n\nThanks for your comments.
\n\nI believe the best way to demonstrate the business impact that thorough data analysis and the machine learning is to use it to solve a real-life problem.
\n\nA flawed toy question does the opposite. I use RStudio's product myself so it is not my intention to offend anyone. So apologies in advance. But at least for me, have a question like this shows a lack of understanding of the fundamental of machine learning.
", "votes": null, "replies": [] }, { "id": 396124, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/30/2018 03:31:07", "content": "I totally agree, but maybe RStudio did not have a problem they wanted to solve, or did not have access to a reliable dataset, and therefore kindly asked Google if they had any random dataset to practice a competition on, to which Google responded with the Merchandise Store dataset.
\n\nMaybe a better competition would be a database of users who visit RStudio website, how many of them download the software, how many of them make an enterprise purchase, etc.
\n\nTo your last sentence: that's up for the customer to decide. I think you can still have intense data analysis in a flawed problem. I don't think you can use this competition to cite LightGBM or xgboost or NN as very strong models, because it's not a real-life problem, but the data analysis is still valid, in my opinion.
", "votes": null, "replies": [] }, { "id": 396126, "author_name": "louistian", "author_url": "", "post_date": "09/30/2018 03:40:49", "content": "I have no particular issue with the dataset. My problem is with the question.
\n\nA much better question would be given only the historical sessions information and ask to predict the users' transactional value in the next n day. In another word, no session information in the test dataset.
\n\nI am not even asking for more or different information. All I am saying is less information should be disclosed in the first place.
", "votes": null, "replies": [] } ] }, { "id": 396120, "author_name": "jpmiller", "author_url": "", "post_date": "09/30/2018 03:19:24", "content": "I'm with you. Also relevant is the fact that so many fields are \"not available in demo dataset\". Any website owner who agrees to share data with Google Analytics can get metrics/features that we don't have here. Landing pages, ad key words, and so on. I don't know how much these things matter, but it does seem that this competition is a spinoff of how we would help a company analyze web purchases in the real world.
", "votes": null, "replies": [] }, { "id": 396200, "author_name": "alexfir", "author_url": "", "post_date": "09/30/2018 08:10:30", "content": "It has been discussed before here
", "votes": null, "replies": [ { "id": 396267, "author_name": "louistian", "author_url": "", "post_date": "09/30/2018 10:59:18", "content": "Thanks for the pointer:)
", "votes": null, "replies": [] } ] }, { "id": 396245, "author_name": "yassinealouini", "author_url": "", "post_date": "09/30/2018 10:13:01", "content": "This is a problem that I have encountered many times (at work and in Kaggle competitions).
\n\nOne possible solution is to only use features that are available at the time of the sample.
\n\nFor example, if you know the web activity of a person in 2018-08-10 then you can use all possible features up to that date (but nothing in the future) even though future information is available in the training dataset.
\n\nAlso, using sklearn TimeSeriesSplit is a good practice. Also avoid shuffling your train data so that leakage isn't introduced.
\n\nFinally, some models are sequential by construction: RNNs for example and \"classic\" statistical methods (ARIMA et al).
\n\nThe Two Sigma approach is indeed great.
\n\nI hope this helps!
", "votes": null, "replies": [ { "id": 396268, "author_name": "louistian", "author_url": "", "post_date": "09/30/2018 11:02:15", "content": "@yassine yes, we could constrain ourselves to only use data available up-to-date but the winning solution is most likely to use future information because it is made available to us in the test set.
", "votes": null, "replies": [] }, { "id": 396270, "author_name": "louistian", "author_url": "", "post_date": "09/30/2018 11:07:33", "content": "I was on fact think about using RNN. But how do you incorporate future information in RNN for the win?
", "votes": null, "replies": [] }, { "id": 396277, "author_name": "yassinealouini", "author_url": "", "post_date": "09/30/2018 11:24:17", "content": "That's for sure. I was thinking about models that will perform better in the \"real world\" and not necessary win.
\n\nFor RNNs, I am a novice and haven't used them a lot but I think you can use a \"two-way\" RNN. One possibility is bi-directional LSTM: this article is a nice explanation.
\n\nGood luck. :)
", "votes": null, "replies": [] } ] }, { "id": 398382, "author_name": "zhangshuhui", "author_url": "", "post_date": "10/04/2018 03:29:39", "content": "Can not agree more! The best model of this competition is very like to be useless in real world.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "396089": "Among many other things, one thing that makes Kaggle great is that we get to using real-life data to solve a real-life problem. \n\nHowever, due to the way this competition is framed, there is a fundamental temporal feature leakage problem embedded in this competition that makes it very unrealistic. If I want to predict a user's spending on day `n`. Knowing whether and when a user will come back in a later time is obviously useful. However, in the real-life situation, you would not know any future web traffic in advance because, well, it's in the future. Competitors are therefore forced to use this **leaked** feature in their solution. \n\nIf we look at this problem at a different angle, the test data includes all of the web traffic data for the next `n` days. If we already know the web traffic, then we would have to know any transactions already in real life. Then what the point of predicting things that are already known?\n\nUPDATE: \nJust saw the Two Sigma competition. This competition should have been done in the same way as a kernel only competition to avoid leakage.", "396108": "I don't think the competition hosts care if it's unrealistic. Recall the goal of this competition:\n\n> RStudio, the developer of free and open tools for R and enterprise-ready products for teams to scale and share work, has partnered with Google Cloud and Kaggle to demonstrate the business impact that thorough data analysis can have.\n\nThey're looking for people to prove that data analysis can improve ML performance; their goal is *not* to look for the best model and then implement it in real life. I think this is a toy problem to see the upper limits of data analysis. One big reason I believe they're doing this is to prove that R is very useful in data analysis, which can then encourage people to use RStudio, as opposed to using Python Jupyter notebook for example. RStudio can cite this competition when trying to sell their enterprise products. \n\nTherefore I agree that this competition is unrealistic, but I don't think the competition hosts care about that", "396120": "I'm with you. Also relevant is the fact that so many fields are \"not available in demo dataset\". Any website owner who agrees to share data with Google Analytics can get metrics/features that we don't have here. Landing pages, ad key words, and so on. I don't know how much these things matter, but it does seem that this competition is a spinoff of how we would help a company analyze web purchases in the real world.", "396123": "CoreyLevinson\n\nThanks for your comments.\n \nI believe the best way to demonstrate the business impact that thorough data analysis and the machine learning is to use it to solve a real-life problem. \n\nA flawed toy question does the opposite. I use RStudio's product myself so it is not my intention to offend anyone. So apologies in advance. But at least for me, have a question like this shows a lack of understanding of the fundamental of machine learning.", "396124": "I totally agree, but maybe RStudio did not have a problem they wanted to solve, or did not have access to a reliable dataset, and therefore kindly asked Google if they had any random dataset to practice a competition on, to which Google responded with the Merchandise Store dataset.\n\nMaybe a better competition would be a database of users who visit RStudio website, how many of them download the software, how many of them make an enterprise purchase, etc.\n\nTo your last sentence: that's up for the customer to decide. I think you can still have intense data analysis in a flawed problem. I don't think you can use this competition to cite LightGBM or xgboost or NN as very strong models, because it's not a real-life problem, but the data analysis is still valid, in my opinion.", "396126": "I have no particular issue with the dataset. My problem is with the question. \n\nA much better question would be given only the historical sessions information and ask to predict the users' transactional value in the next n day. In another word, no session information in the test dataset. \n\nI am not even asking for more or different information. All I am saying is less information should be disclosed in the first place.", "396200": "It has been discussed before [here][1]\n\n\n [1]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66440", "396245": "This is a problem that I have encountered many times (at work and in Kaggle competitions). \n\nOne possible solution is to only use features **that are available at the time of the sample**. \n\nFor example, if you know the web activity of a person in 2018-08-10 then you can use all possible features **up to that date** (but nothing in the future) even though future information is available in the training dataset. \n\nAlso, using sklearn [TimeSeriesSplit][1] is a good practice. Also avoid shuffling your train data so that leakage isn't introduced. \n\nFinally, some models are sequential by construction: [RNNs][2] for example and \"classic\" statistical methods ([ARIMA][3] et al).\n\nThe Two Sigma approach is indeed great.\n\nI hope this helps!\n\n\n [1]: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html.\n [2]: https://en.wikipedia.org/wiki/Recurrent_neural_network\n [3]: https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average", "396267": "Thanks for the pointer:)", "396268": "yassine yes, we could constrain ourselves to only use data available up-to-date but the winning solution is most likely to use future information because it is made available to us in the test set.", "396270": "I was on fact think about using RNN. But how do you incorporate future information in RNN for the win?", "396277": "That's for sure. I was thinking about models that will perform better in the \"real world\" and not necessary win. \n\nFor RNNs, I am a novice and haven't used them a lot but I think you can use a \"two-way\" RNN. One possibility is bi-directional LSTM: this [article][1] is a nice explanation. \n\nGood luck. :)\n\n [1]: https://machinelearningmastery.com/develop-bidirectional-lstm-sequence-classification-python-keras/", "398382": "Can not agree more! The best model of this competition is very like to be useless in real world." }, "source": "meta" }