Does Google want to optimise log(revenue) or revenue? Obviously they would want to optimise revenue. This competition is a waste of my time if the \"winner\" is going to be the team with the closest predictions for log(revenue).
", "messages": [ { "id": "388971", "postDate": "09/17/2018 23:10:17", "content": "Does Google want to optimise log(revenue) or revenue? Obviously they would want to optimise revenue. This competition is a waste of my time if the \"winner\" is going to be the team with the closest predictions for log(revenue).
", "rawMarkdown": "Does Google want to optimise log(revenue) or revenue? Obviously they would want to optimise revenue. This competition is a waste of my time if the \"winner\" is going to be the team with the closest predictions for log(revenue).", "votes": null }, { "id": "388976", "postDate": "09/17/2018 23:25:53", "content": "If you can optimize log(revenue) really well, then you can do exp(log(revenue)) and you can optimize revenue very well. The reason they chose log(revenue) is that it is more forgiving if you correctly predict there is a purchase, but were inaccurate with the magnitude of the revenue.
It's wrong to assume that you know better than the business what the correct prediction task should be. Minimizing RMSE on log(revenue) translates into a problem where you are more interested in predicting the ballpark value (read: order of magnitude) of a customer than a hyper-precise value, which is a completely reasonable business goal and probably much more useful than training a model to poorly solve the harder problem of exact value prediction.
\n\nI will point out though that optimizing for log(revenue) prediction can be quite different from optimizing for revenue. Here's an example, using base 10 log-scale for simplicity. Say I have two labels whose ground truth on log-scale are 2 and 3, and my predicted values are respectively 3 and 3.5. On log scale, bringing the predicted 3 to the ground truth 2 would yield a greater reduction in RMSE than bringing 3.5 to 3. But going back to revenue scale, that would mean that improving a prediction by 900 (10^3 - 10^2) is more valuable than improving a prediction by ~2000 points (10^(3.5) - 10^(3)). So clearly optimizing for log(revenue) can have counterintuitive results when it comes to how those predictions translate back into the original scale.
", "rawMarkdown": "It's wrong to assume that you know better than the business what the correct prediction task should be. Minimizing RMSE on log(revenue) translates into a problem where you are more interested in predicting the ballpark value (read: order of magnitude) of a customer than a hyper-precise value, which is a completely reasonable business goal and probably much more useful than training a model to poorly solve the harder problem of exact value prediction.\n\nI will point out though that optimizing for log(revenue) prediction can be quite different from optimizing for revenue. Here's an example, using base 10 log-scale for simplicity. Say I have two labels whose ground truth on log-scale are 2 and 3, and my predicted values are respectively 3 and 3.5. On log scale, bringing the predicted 3 to the ground truth 2 would yield a greater reduction in RMSE than bringing 3.5 to 3. But going back to revenue scale, that would mean that improving a prediction by 900 (10^3 - 10^2) is more valuable than improving a prediction by ~2000 points (10^(3.5) - 10^(3)). So clearly optimizing for log(revenue) can have counterintuitive results when it comes to how those predictions translate back into the original scale.", "votes": null }, { "id": "389086", "postDate": "09/18/2018 06:03:16", "content": "The winner will be the team with the closest predictions for log(revenue) because this is the objective in the competition. The rationale behind the objective (as I understand) is to separate marketing resources to the group of visitors that most probably spend thousands or hundreds or just few dollars. So here is more important to 'guess' the group of visitors and not if somebody spend 1.35 or 1.37$.
", "rawMarkdown": "The winner will be the team with the closest predictions for log(revenue) because this is the objective in the competition. The rationale behind the objective (as I understand) is to separate marketing resources to the group of visitors that most probably spend thousands or hundreds or just few dollars. So here is more important to 'guess' the group of visitors and not if somebody spend 1.35 or 1.37$.", "votes": null }, { "id": "389107", "postDate": "09/18/2018 06:58:54", "content": "So Alexander, you are saying this is a classification problem not a regression problem? If I was Google, I'd want to know what I could do to get more of those top .03% of website visitors who contribute 1/3 of revenue. And those .14% who contribute 1/3 of revenue. And the 1.11% of purchasers who provide the final 1/3 of all revenue. If they can be marketed to, the top 300 spenders are worth an awful lot more each than the bottom 300 spenders (who on a whim might have been $0 spenders, ie unpredictable). I think the \"target\" and the RMSE need a good rewrite.
", "rawMarkdown": "So Alexander, you are saying this is a classification problem not a regression problem? If I was Google, I'd want to know what I could do to get more of those top .03% of website visitors who contribute 1/3 of revenue. And those .14% who contribute 1/3 of revenue. And the 1.11% of purchasers who provide the final 1/3 of all revenue. If they can be marketed to, the top 300 spenders are worth an awful lot more each than the bottom 300 spenders (who on a whim might have been $0 spenders, ie unpredictable). I think the \"target\" and the RMSE need a good rewrite.", "votes": null }, { "id": "389111", "postDate": "09/18/2018 07:15:08", "content": "For a business to have an objective of closest predictions of log(revenue) is just stupid. (Apparently I am wrong to suggest I know better than the business; wrong morally? ethically? legally? I don't know. I do know I would spend as much time as necessary to explain to my boss if I was told to predict log(revenue). Insights aren't insights if they are rubbish. ) 1/3 of revenue is derived from 300 purchases. 1/3 of revenue is derived from another 1300 purchases. Last 1/3 of revenue comes from 10,000 purchases. In stats when data is this skewed, a log transformation could be used (not compulsory) but you transform back because no-one cares less or wants to know about log(revenue). As for \"adding 1\": because dollar values have been multiplied by 1000000 (or whatever it is, all those 0s just swim), \"adding 1\" is equivalent to adding 1/1000000 (or 1/whatever) to the original dollar values. Which is a perfectly ok thing to do, you can add what you like, there is some basis for adding .5 in bioinformatics, if you choose to do an analysis involving log transforms. For me, I have a feeling regression with log transformed data may not be the way to go for analysing this data.
", "rawMarkdown": "For a business to have an objective of closest predictions of log(revenue) is just stupid. (Apparently I am wrong to suggest I know better than the business; wrong morally? ethically? legally? I don't know. I do know I would spend as much time as necessary to explain to my boss if I was told to predict log(revenue). Insights aren't insights if they are rubbish. ) 1/3 of revenue is derived from 300 purchases. 1/3 of revenue is derived from another 1300 purchases. Last 1/3 of revenue comes from 10,000 purchases. In stats when data is this skewed, a log transformation could be used (not compulsory) but you transform back because no-one cares less or wants to know about log(revenue). As for \"adding 1\": because dollar values have been multiplied by 1000000 (or whatever it is, all those 0s just swim), \"adding 1\" is equivalent to adding 1/1000000 (or 1/whatever) to the original dollar values. Which is a perfectly ok thing to do, you can add what you like, there is some basis for adding .5 in bioinformatics, if you choose to do an analysis involving log transforms. For me, I have a feeling regression with log transformed data may not be the way to go for analysing this data.", "votes": null }, { "id": "389143", "postDate": "09/18/2018 08:11:00", "content": "Ingrid, no, I did not want to say this is a classification problem. \nOn intuitive level I understand why log is used. I do not know if somebody can prove using or not using log in some formal way when dealing with revenues, page views, browsers, etc.
", "rawMarkdown": "Ingrid, no, I did not want to say this is a classification problem. \nOn intuitive level I understand why log is used. I do not know if somebody can prove using or not using log in some formal way when dealing with revenues, page views, browsers, etc.", "votes": null }, { "id": "389317", "postDate": "09/18/2018 14:30:36", "content": "I thought it was clear from my post that I meant intellectually, not morally wrong. You're welcome to feel that the competition isn't worth your time, but the reasoning you give to critique the metric is not correct.
\n\nSince you disregarded what I said, I'll reiterate:
\n\n\"No one wants to know about log(revenue)\" is not correct. There are plenty of contexts where log scale (order of magnitude) is more relevant than original scale, like when you care more about differentiating high-revenue customers from low-revenue ones (say 10s vs. 1000s) than differentiating customers within the high-revenue bucket (say 5000 vs. 2000). Order of magnitude buckets may be better aligned with marketing strategies / goals.
It's likely not possible to reach a satisfactory predictive performance threshold when using the original scale. In this case, you either have to ask a simpler question (order of magnitude, not exact value) or give up on the problem. If you think it's possible to do a good job predicting the actual exact revenue with this competition data, why don't you try it and see what happens.
If you can optimize log(revenue) really well, then you can do exp(log(revenue)) and you can optimize revenue very well. The reason they chose log(revenue) is that it is more forgiving if you correctly predict there is a purchase, but were inaccurate with the magnitude of the revenue.
It's wrong to assume that you know better than the business what the correct prediction task should be. Minimizing RMSE on log(revenue) translates into a problem where you are more interested in predicting the ballpark value (read: order of magnitude) of a customer than a hyper-precise value, which is a completely reasonable business goal and probably much more useful than training a model to poorly solve the harder problem of exact value prediction.
\n\nI will point out though that optimizing for log(revenue) prediction can be quite different from optimizing for revenue. Here's an example, using base 10 log-scale for simplicity. Say I have two labels whose ground truth on log-scale are 2 and 3, and my predicted values are respectively 3 and 3.5. On log scale, bringing the predicted 3 to the ground truth 2 would yield a greater reduction in RMSE than bringing 3.5 to 3. But going back to revenue scale, that would mean that improving a prediction by 900 (10^3 - 10^2) is more valuable than improving a prediction by ~2000 points (10^(3.5) - 10^(3)). So clearly optimizing for log(revenue) can have counterintuitive results when it comes to how those predictions translate back into the original scale.
", "votes": null, "replies": [ { "id": 389111, "author_name": "ingridbaade", "author_url": "", "post_date": "09/18/2018 07:15:08", "content": "For a business to have an objective of closest predictions of log(revenue) is just stupid. (Apparently I am wrong to suggest I know better than the business; wrong morally? ethically? legally? I don't know. I do know I would spend as much time as necessary to explain to my boss if I was told to predict log(revenue). Insights aren't insights if they are rubbish. ) 1/3 of revenue is derived from 300 purchases. 1/3 of revenue is derived from another 1300 purchases. Last 1/3 of revenue comes from 10,000 purchases. In stats when data is this skewed, a log transformation could be used (not compulsory) but you transform back because no-one cares less or wants to know about log(revenue). As for \"adding 1\": because dollar values have been multiplied by 1000000 (or whatever it is, all those 0s just swim), \"adding 1\" is equivalent to adding 1/1000000 (or 1/whatever) to the original dollar values. Which is a perfectly ok thing to do, you can add what you like, there is some basis for adding .5 in bioinformatics, if you choose to do an analysis involving log transforms. For me, I have a feeling regression with log transformed data may not be the way to go for analysing this data.
", "votes": null, "replies": [] }, { "id": 389317, "author_name": "aquatic", "author_url": "", "post_date": "09/18/2018 14:30:36", "content": "I thought it was clear from my post that I meant intellectually, not morally wrong. You're welcome to feel that the competition isn't worth your time, but the reasoning you give to critique the metric is not correct.
\n\nSince you disregarded what I said, I'll reiterate:
\n\n\"No one wants to know about log(revenue)\" is not correct. There are plenty of contexts where log scale (order of magnitude) is more relevant than original scale, like when you care more about differentiating high-revenue customers from low-revenue ones (say 10s vs. 1000s) than differentiating customers within the high-revenue bucket (say 5000 vs. 2000). Order of magnitude buckets may be better aligned with marketing strategies / goals.
It's likely not possible to reach a satisfactory predictive performance threshold when using the original scale. In this case, you either have to ask a simpler question (order of magnitude, not exact value) or give up on the problem. If you think it's possible to do a good job predicting the actual exact revenue with this competition data, why don't you try it and see what happens.
The winner will be the team with the closest predictions for log(revenue) because this is the objective in the competition. The rationale behind the objective (as I understand) is to separate marketing resources to the group of visitors that most probably spend thousands or hundreds or just few dollars. So here is more important to 'guess' the group of visitors and not if somebody spend 1.35 or 1.37$.
", "votes": null, "replies": [] }, { "id": 389107, "author_name": "ingridbaade", "author_url": "", "post_date": "09/18/2018 06:58:54", "content": "So Alexander, you are saying this is a classification problem not a regression problem? If I was Google, I'd want to know what I could do to get more of those top .03% of website visitors who contribute 1/3 of revenue. And those .14% who contribute 1/3 of revenue. And the 1.11% of purchasers who provide the final 1/3 of all revenue. If they can be marketed to, the top 300 spenders are worth an awful lot more each than the bottom 300 spenders (who on a whim might have been $0 spenders, ie unpredictable). I think the \"target\" and the RMSE need a good rewrite.
", "votes": null, "replies": [ { "id": 389143, "author_name": "alexfir", "author_url": "", "post_date": "09/18/2018 08:11:00", "content": "Ingrid, no, I did not want to say this is a classification problem. \nOn intuitive level I understand why log is used. I do not know if somebody can prove using or not using log in some formal way when dealing with revenues, page views, browsers, etc.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "388971": "Does Google want to optimise log(revenue) or revenue? Obviously they would want to optimise revenue. This competition is a waste of my time if the \"winner\" is going to be the team with the closest predictions for log(revenue).", "388976": "If you can optimize `log(revenue)` really well, then you can do `exp(log(revenue))` and you can optimize revenue very well. The reason they chose `log(revenue)` is that it is more forgiving if you correctly predict there is a purchase, but were inaccurate with the magnitude of the revenue.", "388988": "It's wrong to assume that you know better than the business what the correct prediction task should be. Minimizing RMSE on log(revenue) translates into a problem where you are more interested in predicting the ballpark value (read: order of magnitude) of a customer than a hyper-precise value, which is a completely reasonable business goal and probably much more useful than training a model to poorly solve the harder problem of exact value prediction.\n\nI will point out though that optimizing for log(revenue) prediction can be quite different from optimizing for revenue. Here's an example, using base 10 log-scale for simplicity. Say I have two labels whose ground truth on log-scale are 2 and 3, and my predicted values are respectively 3 and 3.5. On log scale, bringing the predicted 3 to the ground truth 2 would yield a greater reduction in RMSE than bringing 3.5 to 3. But going back to revenue scale, that would mean that improving a prediction by 900 (10^3 - 10^2) is more valuable than improving a prediction by ~2000 points (10^(3.5) - 10^(3)). So clearly optimizing for log(revenue) can have counterintuitive results when it comes to how those predictions translate back into the original scale.", "389086": "The winner will be the team with the closest predictions for log(revenue) because this is the objective in the competition. The rationale behind the objective (as I understand) is to separate marketing resources to the group of visitors that most probably spend thousands or hundreds or just few dollars. So here is more important to 'guess' the group of visitors and not if somebody spend 1.35 or 1.37$.", "389107": "So Alexander, you are saying this is a classification problem not a regression problem? If I was Google, I'd want to know what I could do to get more of those top .03% of website visitors who contribute 1/3 of revenue. And those .14% who contribute 1/3 of revenue. And the 1.11% of purchasers who provide the final 1/3 of all revenue. If they can be marketed to, the top 300 spenders are worth an awful lot more each than the bottom 300 spenders (who on a whim might have been $0 spenders, ie unpredictable). I think the \"target\" and the RMSE need a good rewrite.", "389111": "For a business to have an objective of closest predictions of log(revenue) is just stupid. (Apparently I am wrong to suggest I know better than the business; wrong morally? ethically? legally? I don't know. I do know I would spend as much time as necessary to explain to my boss if I was told to predict log(revenue). Insights aren't insights if they are rubbish. ) 1/3 of revenue is derived from 300 purchases. 1/3 of revenue is derived from another 1300 purchases. Last 1/3 of revenue comes from 10,000 purchases. In stats when data is this skewed, a log transformation could be used (not compulsory) but you transform back because no-one cares less or wants to know about log(revenue). As for \"adding 1\": because dollar values have been multiplied by 1000000 (or whatever it is, all those 0s just swim), \"adding 1\" is equivalent to adding 1/1000000 (or 1/whatever) to the original dollar values. Which is a perfectly ok thing to do, you can add what you like, there is some basis for adding .5 in bioinformatics, if you choose to do an analysis involving log transforms. For me, I have a feeling regression with log transformed data may not be the way to go for analysing this data.", "389143": "Ingrid, no, I did not want to say this is a classification problem. \nOn intuitive level I understand why log is used. I do not know if somebody can prove using or not using log in some formal way when dealing with revenues, page views, browsers, etc.", "389317": "I thought it was clear from my post that I meant intellectually, not morally wrong. You're welcome to feel that the competition isn't worth your time, but the reasoning you give to critique the metric is not correct.\n\nSince you disregarded what I said, I'll reiterate:\n\n1. \"No one wants to know about log(revenue)\" is not correct. There are plenty of contexts where log scale (order of magnitude) is more relevant than original scale, like when you care more about differentiating high-revenue customers from low-revenue ones (say 10s vs. 1000s) than differentiating customers within the high-revenue bucket (say 5000 vs. 2000). Order of magnitude buckets may be better aligned with marketing strategies / goals. \n\n2. It's likely not possible to reach a satisfactory predictive performance threshold when using the original scale. In this case, you either have to ask a simpler question (order of magnitude, not exact value) or give up on the problem. If you think it's possible to do a good job predicting the actual exact revenue with this competition data, why don't you try it and see what happens." }, "source": "meta" }