What I mean here is that with an imbalanced dataset like what we have here, it is very difficult to produce predictions that not only looks good on the metrics but also looks like real data. I can produce a prediction with fairly low RMSE score with various regression techniques, but there are inherently so much uncertainty in the prediction that in the end, what you see as the regression output is really the weighted average of the prediction given revenue = 0 and the prediction given revenue > 0. Since the prior for prediction = 0 is so strong, you typically end up with predictions whose average given revenue > 0 is often 3-4 orders of magnitude away from the true average positive revenues. Yes, we have a prediction, but what use does it have if it does not remotely look like the real data?
\n\nI think in the real world, [probability of revenue > 0] and [hypothetical revenue if it is nonzero] are probably more useful when kept apart than multiplied together. I wonder how well we can do if we actually build a model around that idea. Actually I might try it myself.
\n\nSo what do you guys think? How do you interpret the actual values your models pump out? Does it make sense in your everyday adventures to create a model that produces an artificial statistical construct rather than a prediction that appears real?
", "messages": [ { "id": "394290", "postDate": "09/26/2018 16:10:29", "content": "What I mean here is that with an imbalanced dataset like what we have here, it is very difficult to produce predictions that not only looks good on the metrics but also looks like real data. I can produce a prediction with fairly low RMSE score with various regression techniques, but there are inherently so much uncertainty in the prediction that in the end, what you see as the regression output is really the weighted average of the prediction given revenue = 0 and the prediction given revenue > 0. Since the prior for prediction = 0 is so strong, you typically end up with predictions whose average given revenue > 0 is often 3-4 orders of magnitude away from the true average positive revenues. Yes, we have a prediction, but what use does it have if it does not remotely look like the real data?
\n\nI think in the real world, [probability of revenue > 0] and [hypothetical revenue if it is nonzero] are probably more useful when kept apart than multiplied together. I wonder how well we can do if we actually build a model around that idea. Actually I might try it myself.
\n\nSo what do you guys think? How do you interpret the actual values your models pump out? Does it make sense in your everyday adventures to create a model that produces an artificial statistical construct rather than a prediction that appears real?
", "rawMarkdown": "What I mean here is that with an imbalanced dataset like what we have here, it is very difficult to produce predictions that not only looks good on the metrics but also looks like real data. I can produce a prediction with fairly low RMSE score with various regression techniques, but there are inherently so much uncertainty in the prediction that in the end, what you see as the regression output is really the weighted average of the prediction given revenue = 0 and the prediction given revenue > 0. Since the prior for prediction = 0 is so strong, you typically end up with predictions whose average given revenue > 0 is often 3-4 orders of magnitude away from the true average positive revenues. Yes, we have a prediction, but what use does it have if it does not remotely look like the real data?\n\nI think in the real world, [probability of revenue > 0] and [hypothetical revenue if it is nonzero] are probably more useful when kept apart than multiplied together. I wonder how well we can do if we actually build a model around that idea. Actually I might try it myself.\n\nSo what do you guys think? How do you interpret the actual values your models pump out? Does it make sense in your everyday adventures to create a model that produces an artificial statistical construct rather than a prediction that appears real?", "votes": null }, { "id": "394316", "postDate": "09/26/2018 16:55:28", "content": "I'd like to at least try doing the predictions in 2 steps. First run a classifier to distinguish zero from non-zero revenue. Then run a regression on only the cases where the revenue is non-zero.
\n\nI don't think there is any real world application to the models we are building (other than what we learn by competing). By the time you have all the features in the data set, you also have the revenue amount...
", "rawMarkdown": "I'd like to at least try doing the predictions in 2 steps. First run a classifier to distinguish zero from non-zero revenue. Then run a regression on only the cases where the revenue is non-zero.\n\nI don't think there is any real world application to the models we are building (other than what we learn by competing). By the time you have all the features in the data set, you also have the revenue amount...", "votes": null }, { "id": "394376", "postDate": "09/26/2018 18:45:40", "content": "Yes, the metric for this competition is not a good one, and would lead to disastrously bad models in real life. In the beginning, I've built a couple of simple but useful models that have scored far, far worse on this metric than just a 0 prediction everywhere. I think every data scientist needs to understand that there is way, way more to choosing the metric than just finding a well-behaved transformation of (actual - predicted).
", "rawMarkdown": "Yes, the metric for this competition is not a good one, and would lead to disastrously bad models in real life. In the beginning, I've built a couple of simple but useful models that have scored far, far worse on this metric than just a 0 prediction everywhere. I think every data scientist needs to understand that there is way, way more to choosing the metric than just finding a well-behaved transformation of (actual - predicted).", "votes": null }, { "id": "394388", "postDate": "09/26/2018 19:06:24", "content": "I couldn't agree more. This might lead to bad models in real life case. Also, from the perspective of sums of predictions. Models of the various team can generalize better but have a good gap at leaderboard positions. I guess, the game here is also about a definite and reliable Cross Validation strategy to prevail in private leaderboard
", "rawMarkdown": "I couldn't agree more. This might lead to bad models in real life case. Also, from the perspective of sums of predictions. Models of the various team can generalize better but have a good gap at leaderboard positions. I guess, the game here is also about a definite and reliable Cross Validation strategy to prevail in private leaderboard", "votes": null }, { "id": "394729", "postDate": "09/27/2018 10:40:07", "content": "That's a good approach. Only difficulty being is deciding the threshold for the classification task.
", "rawMarkdown": "That's a good approach. Only difficulty being is deciding the threshold for the classification task.", "votes": null } ], "comments": [ { "id": 394316, "author_name": "afgonczol", "author_url": "", "post_date": "09/26/2018 16:55:28", "content": "I'd like to at least try doing the predictions in 2 steps. First run a classifier to distinguish zero from non-zero revenue. Then run a regression on only the cases where the revenue is non-zero.
\n\nI don't think there is any real world application to the models we are building (other than what we learn by competing). By the time you have all the features in the data set, you also have the revenue amount...
", "votes": null, "replies": [ { "id": 394729, "author_name": "shubhamjain28", "author_url": "", "post_date": "09/27/2018 10:40:07", "content": "That's a good approach. Only difficulty being is deciding the threshold for the classification task.
", "votes": null, "replies": [] } ] }, { "id": 394376, "author_name": "dmitriyguller", "author_url": "", "post_date": "09/26/2018 18:45:40", "content": "Yes, the metric for this competition is not a good one, and would lead to disastrously bad models in real life. In the beginning, I've built a couple of simple but useful models that have scored far, far worse on this metric than just a 0 prediction everywhere. I think every data scientist needs to understand that there is way, way more to choosing the metric than just finding a well-behaved transformation of (actual - predicted).
", "votes": null, "replies": [ { "id": 394388, "author_name": "shaz13", "author_url": "", "post_date": "09/26/2018 19:06:24", "content": "I couldn't agree more. This might lead to bad models in real life case. Also, from the perspective of sums of predictions. Models of the various team can generalize better but have a good gap at leaderboard positions. I guess, the game here is also about a definite and reliable Cross Validation strategy to prevail in private leaderboard
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "394290": "What I mean here is that with an imbalanced dataset like what we have here, it is very difficult to produce predictions that not only looks good on the metrics but also looks like real data. I can produce a prediction with fairly low RMSE score with various regression techniques, but there are inherently so much uncertainty in the prediction that in the end, what you see as the regression output is really the weighted average of the prediction given revenue = 0 and the prediction given revenue > 0. Since the prior for prediction = 0 is so strong, you typically end up with predictions whose average given revenue > 0 is often 3-4 orders of magnitude away from the true average positive revenues. Yes, we have a prediction, but what use does it have if it does not remotely look like the real data?\n\nI think in the real world, [probability of revenue > 0] and [hypothetical revenue if it is nonzero] are probably more useful when kept apart than multiplied together. I wonder how well we can do if we actually build a model around that idea. Actually I might try it myself.\n\nSo what do you guys think? How do you interpret the actual values your models pump out? Does it make sense in your everyday adventures to create a model that produces an artificial statistical construct rather than a prediction that appears real?", "394316": "I'd like to at least try doing the predictions in 2 steps. First run a classifier to distinguish zero from non-zero revenue. Then run a regression on only the cases where the revenue is non-zero.\n\nI don't think there is any real world application to the models we are building (other than what we learn by competing). By the time you have all the features in the data set, you also have the revenue amount...", "394376": "Yes, the metric for this competition is not a good one, and would lead to disastrously bad models in real life. In the beginning, I've built a couple of simple but useful models that have scored far, far worse on this metric than just a 0 prediction everywhere. I think every data scientist needs to understand that there is way, way more to choosing the metric than just finding a well-behaved transformation of (actual - predicted).", "394388": "I couldn't agree more. This might lead to bad models in real life case. Also, from the perspective of sums of predictions. Models of the various team can generalize better but have a good gap at leaderboard positions. I guess, the game here is also about a definite and reliable Cross Validation strategy to prevail in private leaderboard", "394729": "That's a good approach. Only difficulty being is deciding the threshold for the classification task." }, "source": "meta" }