{ "id": 67203, "title": "RMSE Calculation Confusion", "url": "/competitions/ga-customer-revenue-prediction/discussion/67203", "author_name": "Neeraj", "post_date": "2018-09-29T23:40:38.588000", "votes": 0, "comment_count": 6, "views": 0, "content": "

Hi Team, @philculliton ; @juliaelliott

\n\n

On the \"Evaluation\" section, you explain how you will evaluate the competition, and I have attached the screenshots - https://ibb.co/ivJumz\nIt says \"where y hat is the natural log of the predicted revenue for a customer and y is the natural log of the actual summed revenue value plus one.\" . \nIt seems you are adding 1 only in case of the actual summed revenue, and not in case of the predicted. Are you sure that's the way you are evaluating ?

\n\n

I see there are 2 ways for us/participants to approach the competition:

\n\n\n\n

I hope I was able to clarify my question without creating more confusion. Please let me know if there's any confusion.

\n\n

Regards,\nNeeraj

", "messages": [ { "id": 396087, "postDate": "2018-09-30T02:21:04.613Z", "content": "

Neeraj, the idea is that you have to come up with the yhat yourself. The actual target is log(1 + sum(revenue)) as stated in the competition...your goal is to get as close as possible to this value. You can choose to add 1 or not in your predictions. There is no requirement that your yhat be generated in a certain way...you could choose, for example the mean of all the revenue predictions on a transaction level, choose log(sum(revenue)) or log(1 + sum(revenue)), it doesn't matter how you come up with it, but just keep in mind that since you want to minimize something of the form log(1 + sum(revenue)), then you probably want your predictions to be of the form log(1 + sum(revenue)) but it's not mandatory.

", "rawMarkdown": "Neeraj, the idea is that *you* have to come up with the yhat yourself. The actual target is `log(1 + sum(revenue))` as stated in the competition...your goal is to get as close as possible to this value. You can choose to add 1 or not in your predictions. There is no requirement that your `yhat` be generated in a certain way...you could choose, for example the mean of all the revenue predictions on a transaction level, choose log(sum(revenue)) or log(1 + sum(revenue)), it doesn't matter how you come up with it, but just keep in mind that since you want to minimize something of the form `log(1 + sum(revenue))`, then you probably want your predictions to be of the form `log(1 + sum(revenue))` but it's not mandatory. ", "replies": [ { "id": 396285, "postDate": "2018-09-30T11:43:54.753Z", "content": "

@returnofsputnik - I see what you mean. But the question specific to RMSE calculation is - It says \"where y hat is the natural log of the predicted revenue for a customer and y is the natural log of the actual summed revenue value plus one.\" . It seems they are adding 1 only in case of the actual summed revenue, and not in case of the predicted.

\n\n

And in the screenshots- in the \"Evaluation\" section also, they mentioned \"predicted log of revenue per customer\". Are they surely not adding 1 in the prediction. We know there's tons of zeros in the actuals. If I predict them accurately as 0 - the log of predicted revenue will become -infinity.

\n\n

That's a concern I am having. What do you think?

", "rawMarkdown": "@returnofsputnik - I see what you mean. But the question specific to RMSE calculation is - It says \"where y hat is the natural log of the predicted revenue for a customer and y is the natural log of the actual summed revenue value plus one.\" . It seems they are adding 1 only in case of the actual summed revenue, and not in case of the predicted.\n\nAnd in the screenshots- in the \"Evaluation\" section also, they mentioned \"predicted log of revenue per customer\". Are they surely not adding 1 in the prediction. We know there's tons of zeros in the actuals. If I predict them accurately as 0 - the log of predicted revenue will become -infinity.\n\nThat's a concern I am having. What do you think?", "votes": 1 }, { "id": 396368, "postDate": "2018-09-30T15:15:06.573Z", "content": "

I don't understand the problem, to be honest.

\n\n
\n

Are they surely not adding 1 in the prediction.

\n
\n\n

\"They\" are not adding 1 in the prediction. They don't modify your predictions at all. Only you provide the prediction. If you predict someone has 0 revenue and then just take the log of that, then your prediction is infinity, but the actual is log1p(0) = 0. That large error was due to your choice of using log() instead of log1p().

", "rawMarkdown": "I don't understand the problem, to be honest.\n\n> Are they surely not adding 1 in the prediction.\n\n\"They\" are not adding 1 in the prediction. They don't modify your predictions at all. Only you provide the prediction. If you predict someone has 0 revenue and then just take the log of that, then your prediction is infinity, but the actual is log1p(0) = 0. That large error was due to your choice of using log() instead of log1p(). ", "votes": -1 }, { "id": 396386, "postDate": "2018-09-30T16:15:16.883Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 396421, "postDate": "2018-09-30T17:57:14.520Z", "content": "

So Neeraj after reading the comments I think you have doubt that we have to predict ln(revenue) or ln(1+revenue).

\n\n

In the evaluation page, they said to predict log of revenue per customer. But in the discussion forum, I think Phil clarified that we have to predict the log 1+revenue (that is log1p).

\n\n

see the links below:

\n\n

https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387628

\n\n

https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387242

", "rawMarkdown": "So Neeraj after reading the comments I think you have doubt that we have to predict ln(revenue) or ln(1+revenue). \n\nIn the evaluation page, they said to predict log of revenue per customer. But in the discussion forum, I think Phil clarified that we have to predict the log 1+revenue (that is log1p).\n\nsee the links below:\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387628\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387242\n" }, { "id": 396068, "postDate": "2018-09-30T01:06:59.780Z", "content": "

@philculliton

", "rawMarkdown": "@philculliton" }, { "id": 396047, "postDate": "2018-09-29T23:40:38.590Z", "content": "

Hi Team, @philculliton ; @juliaelliott

\n\n

On the \"Evaluation\" section, you explain how you will evaluate the competition, and I have attached the screenshots - https://ibb.co/ivJumz\nIt says \"where y hat is the natural log of the predicted revenue for a customer and y is the natural log of the actual summed revenue value plus one.\" . \nIt seems you are adding 1 only in case of the actual summed revenue, and not in case of the predicted. Are you sure that's the way you are evaluating ?

\n\n

I see there are 2 ways for us/participants to approach the competition:

\n\n\n\n

I hope I was able to clarify my question without creating more confusion. Please let me know if there's any confusion.

\n\n

Regards,\nNeeraj

", "rawMarkdown": "Hi Team, @philculliton ; @juliaelliott\n\nOn the \"Evaluation\" section, you explain how you will evaluate the competition, and I have attached the screenshots - https://ibb.co/ivJumz\n**It says \"where y hat is the natural log of the predicted revenue for a customer and y is the natural log of the actual summed revenue value plus one.\"** . \n**It seems you are adding 1 only in case of the actual summed revenue, and not in case of the predicted. Are you sure that's the way you are evaluating ?**\n\n\nI see there are 2 ways for us/participants to approach the competition:\n\n - Participants first aggregate the data on customer level, train it on that level, and predict on that level and make the submissions.\nIn that case, the training a model to predict \"log( 1 + sum(revenue) )\" , so in that case yhat can't be \"the natural log of the predicted revenue for a customer\" since the participants are training and optimizing for something else =yhat = estimate of \"log( 1 + sum(revenue) )\"\nOR\n - Participants train and predict at the same level the data is given. After making the predictions, then they aggregate it to customer level, and submit.\n**The way you are showing the evaluation, it seems you are encouraging this way to training, optimizing and predicting. If that's the case, are you sure \"the 1 is added only in case of actual summed revenue, and not in case of predicted revenue\" ?\nPlease confirm this- since the way the evaluation is done is the way the participants should train the model and optimize the loss.**\n\nI hope I was able to clarify my question without creating more confusion. Please let me know if there's any confusion.\n\n\nRegards,\nNeeraj", "votes": -1 } ], "comments": [ { "id": 396087, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-09-30T02:21:04.613000", "content": "

Neeraj, the idea is that you have to come up with the yhat yourself. The actual target is log(1 + sum(revenue)) as stated in the competition...your goal is to get as close as possible to this value. You can choose to add 1 or not in your predictions. There is no requirement that your yhat be generated in a certain way...you could choose, for example the mean of all the revenue predictions on a transaction level, choose log(sum(revenue)) or log(1 + sum(revenue)), it doesn't matter how you come up with it, but just keep in mind that since you want to minimize something of the form log(1 + sum(revenue)), then you probably want your predictions to be of the form log(1 + sum(revenue)) but it's not mandatory.

", "votes": 0, "replies": [ { "id": 396285, "author_name": "Neeraj", "author_url": "", "post_date": "2018-09-30T11:43:54.753000", "content": "

@returnofsputnik - I see what you mean. But the question specific to RMSE calculation is - It says \"where y hat is the natural log of the predicted revenue for a customer and y is the natural log of the actual summed revenue value plus one.\" . It seems they are adding 1 only in case of the actual summed revenue, and not in case of the predicted.

\n\n

And in the screenshots- in the \"Evaluation\" section also, they mentioned \"predicted log of revenue per customer\". Are they surely not adding 1 in the prediction. We know there's tons of zeros in the actuals. If I predict them accurately as 0 - the log of predicted revenue will become -infinity.

\n\n

That's a concern I am having. What do you think?

", "votes": 1, "replies": [] }, { "id": 396368, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-09-30T15:15:06.573000", "content": "

I don't understand the problem, to be honest.

\n\n
\n

Are they surely not adding 1 in the prediction.

\n
\n\n

\"They\" are not adding 1 in the prediction. They don't modify your predictions at all. Only you provide the prediction. If you predict someone has 0 revenue and then just take the log of that, then your prediction is infinity, but the actual is log1p(0) = 0. That large error was due to your choice of using log() instead of log1p().

", "votes": -1, "replies": [] }, { "id": 396386, "author_name": "", "author_url": "", "post_date": "2018-09-30T16:15:16.883000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 396421, "author_name": "redshellspy", "author_url": "", "post_date": "2018-09-30T17:57:14.520000", "content": "

So Neeraj after reading the comments I think you have doubt that we have to predict ln(revenue) or ln(1+revenue).

\n\n

In the evaluation page, they said to predict log of revenue per customer. But in the discussion forum, I think Phil clarified that we have to predict the log 1+revenue (that is log1p).

\n\n

see the links below:

\n\n

https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387628

\n\n

https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387242

", "votes": 0, "replies": [] }, { "id": 396068, "author_name": "Neeraj", "author_url": "", "post_date": "2018-09-30T01:06:59.780000", "content": "

@philculliton

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "396087": "Neeraj, the idea is that *you* have to come up with the yhat yourself. The actual target is `log(1 + sum(revenue))` as stated in the competition...your goal is to get as close as possible to this value. You can choose to add 1 or not in your predictions. There is no requirement that your `yhat` be generated in a certain way...you could choose, for example the mean of all the revenue predictions on a transaction level, choose log(sum(revenue)) or log(1 + sum(revenue)), it doesn't matter how you come up with it, but just keep in mind that since you want to minimize something of the form `log(1 + sum(revenue))`, then you probably want your predictions to be of the form `log(1 + sum(revenue))` but it's not mandatory. ", "396421": "So Neeraj after reading the comments I think you have doubt that we have to predict ln(revenue) or ln(1+revenue). \n\nIn the evaluation page, they said to predict log of revenue per customer. But in the discussion forum, I think Phil clarified that we have to predict the log 1+revenue (that is log1p).\n\nsee the links below:\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387628\n\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387242\n", "396068": "@philculliton", "396047": "Hi Team, @philculliton ; @juliaelliott\n\nOn the \"Evaluation\" section, you explain how you will evaluate the competition, and I have attached the screenshots - https://ibb.co/ivJumz\n**It says \"where y hat is the natural log of the predicted revenue for a customer and y is the natural log of the actual summed revenue value plus one.\"** . \n**It seems you are adding 1 only in case of the actual summed revenue, and not in case of the predicted. Are you sure that's the way you are evaluating ?**\n\n\nI see there are 2 ways for us/participants to approach the competition:\n\n - Participants first aggregate the data on customer level, train it on that level, and predict on that level and make the submissions.\nIn that case, the training a model to predict \"log( 1 + sum(revenue) )\" , so in that case yhat can't be \"the natural log of the predicted revenue for a customer\" since the participants are training and optimizing for something else =yhat = estimate of \"log( 1 + sum(revenue) )\"\nOR\n - Participants train and predict at the same level the data is given. After making the predictions, then they aggregate it to customer level, and submit.\n**The way you are showing the evaluation, it seems you are encouraging this way to training, optimizing and predicting. If that's the case, are you sure \"the 1 is added only in case of actual summed revenue, and not in case of predicted revenue\" ?\nPlease confirm this- since the way the evaluation is done is the way the participants should train the model and optimize the loss.**\n\nI hope I was able to clarify my question without creating more confusion. Please let me know if there's any confusion.\n\n\nRegards,\nNeeraj" } }