From the data description, targetUser = ln ( ∑ all transactions by user i )
\n\nWhere the evaluation metric is RMSE (where y-hat is the predicted revenue for a customer and y is the natural log of the actual revenue value).
\n\nMy interpretation of the above would be simply adding up all the non-log transactions of a user, then take the natural log. My experience with the LB, however, appears to be otherwise.
\n\nMy naive xgboost model scored 1.69 if I sum the log predicted revenue for each user, but scored only 1.91 when I exp the log predicted revenues back to non-log, sum them up, and take log again.
\n\nUsing fullVisitorId 319401084506866864 as an illustration. There are 34 log predicted revenues in test (with max at 18.32), that sums to 39.925. If I were to convert back to non-log first, the 34 predicted revenues would have a sum of 91246965 and the log1p of that is 18.33.
\n\nAm I missing something here?
", "messages": [ { "id": "387514", "postDate": "09/15/2018 04:54:47", "content": "From the data description, targetUser = ln ( ∑ all transactions by user i )
\n\nWhere the evaluation metric is RMSE (where y-hat is the predicted revenue for a customer and y is the natural log of the actual revenue value).
\n\nMy interpretation of the above would be simply adding up all the non-log transactions of a user, then take the natural log. My experience with the LB, however, appears to be otherwise.
\n\nMy naive xgboost model scored 1.69 if I sum the log predicted revenue for each user, but scored only 1.91 when I exp the log predicted revenues back to non-log, sum them up, and take log again.
\n\nUsing fullVisitorId 319401084506866864 as an illustration. There are 34 log predicted revenues in test (with max at 18.32), that sums to 39.925. If I were to convert back to non-log first, the 34 predicted revenues would have a sum of 91246965 and the log1p of that is 18.33.
\n\nAm I missing something here?
", "rawMarkdown": "From the data description, **targetUser = ln ( ∑ all transactions by user i )**\n\nWhere the evaluation metric is RMSE (where y-hat is the predicted revenue for a customer and y is the natural log of the actual revenue value).\n\nMy interpretation of the above would be simply adding up all the non-log transactions of a user, then take the natural log. My experience with the LB, however, appears to be otherwise.\n\nMy naive xgboost model scored 1.69 if I sum the log predicted revenue for each user, but scored only 1.91 when I exp the log predicted revenues back to non-log, sum them up, and take log again.\n\nUsing fullVisitorId 319401084506866864 as an illustration. There are 34 log predicted revenues in test (with max at 18.32), that sums to 39.925. If I were to convert back to non-log first, the 34 predicted revenues would have a sum of 91246965 and the log1p of that is 18.33.\n\nAm I missing something here?", "votes": null }, { "id": "387554", "postDate": "09/15/2018 06:46:17", "content": "ln(a*b) = ln(a) + ln(b) which means that if you sum up all logs of your original values it is as if you multiplied all values together first and then took the log. Does it make sense to multiply all transactions at all? If not, then you should sum them up first and take the log afterwards.
", "rawMarkdown": "ln(a*b) = ln(a) + ln(b) which means that if you sum up all logs of your original values it is as if you multiplied all values together first and then took the log. Does it make sense to multiply all transactions at all? If not, then you should sum them up first and take the log afterwards.", "votes": null }, { "id": "387564", "postDate": "09/15/2018 07:22:53", "content": "Thanks for the heads up. It is indeed either bug or bad explanation from Kaggle side. I just tested it and converting from log of sums to sum of logs boosted me almost 0.1 points. I hope Kaggle reads it and takes an action.
", "rawMarkdown": "Thanks for the heads up. It is indeed either bug or bad explanation from Kaggle side. I just tested it and converting from log of sums to sum of logs boosted me almost 0.1 points. I hope Kaggle reads it and takes an action.", "votes": null }, { "id": "387762", "postDate": "09/15/2018 16:26:06", "content": "I experienced the same phenomenon that you describe. The correct way, according to Kaggle, would be to exp, sum and then take the log, but summing the logs of transactions gives me a lower RMSE than the former.
", "rawMarkdown": "I experienced the same phenomenon that you describe. The correct way, according to Kaggle, would be to exp, sum and then take the log, but summing the logs of transactions gives me a lower RMSE than the former.", "votes": null }, { "id": "387771", "postDate": "09/15/2018 16:50:31", "content": "The correct - and expected - way is as noted in the Evaluation and Data pages. I know, because that's precisely how I set the targets. I would recommend you optimize for the expected target.
\n\nIt should go without saying, too, that lowering your RMSE on the public test set is NOT the same thing as having a model that is generalizing better.
", "rawMarkdown": "The correct - and expected - way is as noted in the Evaluation and Data pages. I know, because that's precisely how I set the targets. I would recommend you optimize for the expected target.\n\nIt should go without saying, too, that lowering your RMSE on the public test set is NOT the same thing as having a model that is generalizing better.", "votes": null }, { "id": "387786", "postDate": "09/15/2018 17:11:14", "content": "Thanks Phil. That's really good information to know.
", "rawMarkdown": "Thanks Phil. That's really good information to know.", "votes": null }, { "id": "387793", "postDate": "09/15/2018 17:23:10", "content": "You're welcome!
", "rawMarkdown": "You're welcome!", "votes": null }, { "id": "387808", "postDate": "09/15/2018 17:51:17", "content": "Hmm, I haven't submitted yet, but locally the log1p(sum of expm1) way looks much better (about .24 lower RMSE). To be specific, I trained at the visit level (predict log1p) before aggregating to the visitor level to get final RMSE, either by doing RMSE(log1p(sum(expm1(y_true))), log1p(sum(expm1(y_pred)))) or just RMSE(sum(y_true),sum(y_pred)). The former is the way that looked better, which is the opposite of what people have said so far about the public LB result.
\n\nIt definitely seems odd to me that the LB and local results would show the exact opposite pattern.
", "rawMarkdown": "Hmm, I haven't submitted yet, but locally the log1p(sum of expm1) way looks much better (about .24 lower RMSE). To be specific, I trained at the visit level (predict log1p) before aggregating to the visitor level to get final RMSE, either by doing RMSE(log1p(sum(expm1(y_true))), log1p(sum(expm1(y_pred)))) or just RMSE(sum(y_true),sum(y_pred)). The former is the way that looked better, which is the opposite of what people have said so far about the public LB result.\n\nIt definitely seems odd to me that the LB and local results would show the exact opposite pattern.", "votes": null }, { "id": "387828", "postDate": "09/15/2018 18:16:56", "content": "Locally I also get a better result doing RMSE(log1p(sum(expm1(y_true))) than just summing the logs, but in the LB I get opposite results. Weird.
", "rawMarkdown": "Locally I also get a better result doing RMSE(log1p(sum(expm1(y_true))) than just summing the logs, but in the LB I get opposite results. Weird.", "votes": null }, { "id": "388055", "postDate": "09/16/2018 07:50:05", "content": "Edit - I'm less confident of this now I've thought about just how noisy the public leaderboard might be due to the skew of the spending.
\n\nI think that the target is the sum of the logs, and is a possible error; not only does my leaderboard score improve materially when I switch to submitting this way, it reconciles much better with my local cross validation.
\n\nAs Chrisquared says, this is the equivalent of multiplying the spends together, which seems odd.
\n\nMy strategy for the two submissions will be one on each scoring method just in case this is a coincidence.
", "rawMarkdown": "Edit - I'm less confident of this now I've thought about just how noisy the public leaderboard might be due to the skew of the spending.\n\nI think that the target is the sum of the logs, and is a possible error; not only does my leaderboard score improve materially when I switch to submitting this way, it reconciles much better with my local cross validation. \n\nAs Chrisquared says, this is the equivalent of multiplying the spends together, which seems odd.\n\nMy strategy for the two submissions will be one on each scoring method just in case this is a coincidence.", "votes": null }, { "id": "388066", "postDate": "09/16/2018 08:17:45", "content": "Could someone please provide me a link to the source? I think I didnt really get the problem.
", "rawMarkdown": "Could someone please provide me a link to the source? I think I didnt really get the problem.", "votes": null }, { "id": "388071", "postDate": "09/16/2018 08:37:58", "content": "Hello - I think based on your previous comment you're perhaps missing the fact that for the competition it doesn't matter what it makes sense to do, all that matter is what metric is being used to score submissions.
\n\nAs you point out, the sum of the logs is appropriate if you think that values should be combined multiplicatively. This doesn't seem like something that we should want to do, and it isn't the approach being signaled by the organizers, but in practice the sum of the logs is scoring higher than the log of the sums and so there might be an error somewhere in the scoring.
\n\nIf it is an error it's kind of an easy one to make, because most of the values are zero, and log(1+0) = 0 (note that we add 1 to the value to avoid log(0) problems). So summing the logs and taking the log of the sum is equivalent for most of the data.
", "rawMarkdown": "Hello - I think based on your previous comment you're perhaps missing the fact that for the competition it doesn't matter what it makes sense to do, all that matter is what metric is being used to score submissions. \n\nAs you point out, the sum of the logs is appropriate if you think that values should be combined multiplicatively. This doesn't seem like something that we should want to do, and it isn't the approach being signaled by the organizers, but in practice the sum of the logs is scoring higher than the log of the sums and so there might be an error somewhere in the scoring.\n\nIf it is an error it's kind of an easy one to make, because most of the values are zero, and log(1+0) = 0 (note that we add 1 to the value to avoid log(0) problems). So summing the logs and taking the log of the sum is equivalent for most of the data.", "votes": null }, { "id": "388104", "postDate": "09/16/2018 10:30:42", "content": "Hi Phil, I have done several experiments and I don't get why sum of logs worked better. Can you double check if there is a mistake in your implementation. If you had already double checked, sorry in advance.
", "rawMarkdown": "Hi Phil, I have done several experiments and I don't get why sum of logs worked better. Can you double check if there is a mistake in your implementation. If you had already double checked, sorry in advance.", "votes": null }, { "id": "388215", "postDate": "09/16/2018 15:16:48", "content": "Yeah, after carefully reading the other comments I realized that i am missing the context. Now I am curious about it cause I wanna understand if there is a mathematical explanation for the increase of performance. Very interesting.
", "rawMarkdown": "Yeah, after carefully reading the other comments I realized that i am missing the context. Now I am curious about it cause I wanna understand if there is a mathematical explanation for the increase of performance. Very interesting.", "votes": null }, { "id": "388255", "postDate": "09/16/2018 16:36:04", "content": "It's either going to be random noise in the public leaderboard that wont necessarily translate to the final rankings (as implied by Phil), or an error in the scoring. Personally I wouldn't go looking for a mathematical explanation.
", "rawMarkdown": "It's either going to be random noise in the public leaderboard that wont necessarily translate to the final rankings (as implied by Phil), or an error in the scoring. Personally I wouldn't go looking for a mathematical explanation.", "votes": null }, { "id": "388890", "postDate": "09/17/2018 19:15:48", "content": "\"where y hat is the predicted revenue for a customer and y is the natural log of the actual revenue value.\"
\n\nSo if I have one user and one entry for that user and the revenue is 2 ...\nThe best error would be (2-log2)*(2-log2) if I predict the revenue exactly\nThe best error is 0 if I predict the log of the actual revenue exactly
\n\nSo we are looking for a function that approximates the metric for very small values of log (sum of entries)
\n\nlog(very tiny+quite large)?\nAm = max(A1,A2)\nlog(log(exp(A1))+log(exp(A2))) -> log(log(exp(Am))) = log(Am)
\n\nDoes anyone have any thoughts? ;)
", "rawMarkdown": "\"where y hat is the predicted revenue for a customer and y is the natural log of the actual revenue value.\"\n\nSo if I have one user and one entry for that user and the revenue is 2 ...\nThe best error would be (2-log2)*(2-log2) if I predict the revenue exactly\nThe best error is 0 if I predict the **log** of the actual revenue exactly\n\nSo we are looking for a function that approximates the metric for very small values of log (sum of entries)\n\nlog(very tiny+quite large)?\nAm = max(A1,A2)\nlog(log(exp(A1))+log(exp(A2))) -> log(log(exp(Am))) = log(Am)\n\nDoes anyone have any thoughts? ;)", "votes": null }, { "id": "389858", "postDate": "09/19/2018 10:22:10", "content": "Yes my interpretation is that that sentence you quote is very inaccurate. It should not be the natural log but the log1p. Ans similarly the y_hat should be that log1p of (the sum of ) predicted revenue(s) for that customer. The data page explains it a lot better.
", "rawMarkdown": "Yes my interpretation is that that sentence you quote is very inaccurate. It should not be the natural log but the log1p. Ans similarly the y_hat should be that log1p of (the sum of ) predicted revenue(s) for that customer. The data page explains it a lot better.", "votes": null }, { "id": "389869", "postDate": "09/19/2018 10:57:53", "content": "I agree log(x+1) is the only thing that makes sense considering a lot of visitors do not produce any revenue.
", "rawMarkdown": "I agree log(x+1) is the only thing that makes sense considering a lot of visitors do not produce any revenue.", "votes": null }, { "id": "389927", "postDate": "09/19/2018 13:25:08", "content": "\n\n\nMy naive xgboost model scored 1.69 if I sum the log predicted revenue for each user, but scored only 1.91 when I exp the log predicted revenues back to non-log, sum them up, and take log again.
\n
There is a problem with how you're adding up predictions before logging. When you exp the log predicted revenues, you're not getting the correct expected value on the linear scale. What you're getting out of the model is E[log(x)], which is not the same thing as log(E[x]). If you add up the total predictions on the linear scale and compare them against total actuals, you're probably going to find that your total predictions are considerably below total actuals.
", "rawMarkdown": "> My naive xgboost model scored 1.69 if I sum the log predicted revenue for each user, but scored only 1.91 when I exp the log predicted revenues back to non-log, sum them up, and take log again.\n\nThere is a problem with how you're adding up predictions before logging. When you exp the log predicted revenues, you're not getting the correct expected value on the linear scale. What you're getting out of the model is E[log(x)], which is not the same thing as log(E[x]). If you add up the total predictions on the linear scale and compare them against total actuals, you're probably going to find that your total predictions are considerably below total actuals.", "votes": null }, { "id": "390464", "postDate": "09/20/2018 08:57:59", "content": "Thanks for the good ideas to bring it. I know a lot more. :)สมัคร ufabet :J
", "rawMarkdown": "Thanks for the good ideas to bring it. I know a lot more. :)สมัคร ufabet :J", "votes": null }, { "id": "390942", "postDate": "09/21/2018 04:05:00", "content": "Thanks for this insight very helpful
", "rawMarkdown": "Thanks for this insight very helpful", "votes": null }, { "id": "391210", "postDate": "09/21/2018 11:59:38", "content": "I think I understand what you are meaning but, what is the correct way to add the predictions up?
", "rawMarkdown": "I think I understand what you are meaning but, what is the correct way to add the predictions up?", "votes": null }, { "id": "393138", "postDate": "09/24/2018 21:39:00", "content": "Thanks very much to everyone who posted here, your help in finding the problem is greatly appreciated.
", "rawMarkdown": "Thanks very much to everyone who posted here, your help in finding the problem is greatly appreciated.", "votes": null } ], "comments": [ { "id": 387554, "author_name": "chrisquared", "author_url": "", "post_date": "09/15/2018 06:46:17", "content": "ln(a*b) = ln(a) + ln(b) which means that if you sum up all logs of your original values it is as if you multiplied all values together first and then took the log. Does it make sense to multiply all transactions at all? If not, then you should sum them up first and take the log afterwards.
", "votes": null, "replies": [] }, { "id": 387564, "author_name": "aerdem4", "author_url": "", "post_date": "09/15/2018 07:22:53", "content": "Thanks for the heads up. It is indeed either bug or bad explanation from Kaggle side. I just tested it and converting from log of sums to sum of logs boosted me almost 0.1 points. I hope Kaggle reads it and takes an action.
", "votes": null, "replies": [] }, { "id": 387762, "author_name": "axelstram", "author_url": "", "post_date": "09/15/2018 16:26:06", "content": "I experienced the same phenomenon that you describe. The correct way, according to Kaggle, would be to exp, sum and then take the log, but summing the logs of transactions gives me a lower RMSE than the former.
", "votes": null, "replies": [ { "id": 390464, "author_name": "seoemlon06", "author_url": "", "post_date": "09/20/2018 08:57:59", "content": "Thanks for the good ideas to bring it. I know a lot more. :)สมัคร ufabet :J
", "votes": null, "replies": [] } ] }, { "id": 387771, "author_name": "philculliton", "author_url": "", "post_date": "09/15/2018 16:50:31", "content": "The correct - and expected - way is as noted in the Evaluation and Data pages. I know, because that's precisely how I set the targets. I would recommend you optimize for the expected target.
\n\nIt should go without saying, too, that lowering your RMSE on the public test set is NOT the same thing as having a model that is generalizing better.
", "votes": null, "replies": [ { "id": 387786, "author_name": "yilihome", "author_url": "", "post_date": "09/15/2018 17:11:14", "content": "Thanks Phil. That's really good information to know.
", "votes": null, "replies": [] }, { "id": 387793, "author_name": "philculliton", "author_url": "", "post_date": "09/15/2018 17:23:10", "content": "You're welcome!
", "votes": null, "replies": [] }, { "id": 388104, "author_name": "aerdem4", "author_url": "", "post_date": "09/16/2018 10:30:42", "content": "Hi Phil, I have done several experiments and I don't get why sum of logs worked better. Can you double check if there is a mistake in your implementation. If you had already double checked, sorry in advance.
", "votes": null, "replies": [] } ] }, { "id": 387808, "author_name": "aquatic", "author_url": "", "post_date": "09/15/2018 17:51:17", "content": "Hmm, I haven't submitted yet, but locally the log1p(sum of expm1) way looks much better (about .24 lower RMSE). To be specific, I trained at the visit level (predict log1p) before aggregating to the visitor level to get final RMSE, either by doing RMSE(log1p(sum(expm1(y_true))), log1p(sum(expm1(y_pred)))) or just RMSE(sum(y_true),sum(y_pred)). The former is the way that looked better, which is the opposite of what people have said so far about the public LB result.
\n\nIt definitely seems odd to me that the LB and local results would show the exact opposite pattern.
", "votes": null, "replies": [ { "id": 387828, "author_name": "axelstram", "author_url": "", "post_date": "09/15/2018 18:16:56", "content": "Locally I also get a better result doing RMSE(log1p(sum(expm1(y_true))) than just summing the logs, but in the LB I get opposite results. Weird.
", "votes": null, "replies": [] } ] }, { "id": 388055, "author_name": "ca3295", "author_url": "", "post_date": "09/16/2018 07:50:05", "content": "Edit - I'm less confident of this now I've thought about just how noisy the public leaderboard might be due to the skew of the spending.
\n\nI think that the target is the sum of the logs, and is a possible error; not only does my leaderboard score improve materially when I switch to submitting this way, it reconciles much better with my local cross validation.
\n\nAs Chrisquared says, this is the equivalent of multiplying the spends together, which seems odd.
\n\nMy strategy for the two submissions will be one on each scoring method just in case this is a coincidence.
", "votes": null, "replies": [] }, { "id": 388066, "author_name": "chrisquared", "author_url": "", "post_date": "09/16/2018 08:17:45", "content": "Could someone please provide me a link to the source? I think I didnt really get the problem.
", "votes": null, "replies": [ { "id": 388071, "author_name": "ca3295", "author_url": "", "post_date": "09/16/2018 08:37:58", "content": "Hello - I think based on your previous comment you're perhaps missing the fact that for the competition it doesn't matter what it makes sense to do, all that matter is what metric is being used to score submissions.
\n\nAs you point out, the sum of the logs is appropriate if you think that values should be combined multiplicatively. This doesn't seem like something that we should want to do, and it isn't the approach being signaled by the organizers, but in practice the sum of the logs is scoring higher than the log of the sums and so there might be an error somewhere in the scoring.
\n\nIf it is an error it's kind of an easy one to make, because most of the values are zero, and log(1+0) = 0 (note that we add 1 to the value to avoid log(0) problems). So summing the logs and taking the log of the sum is equivalent for most of the data.
", "votes": null, "replies": [] }, { "id": 388215, "author_name": "chrisquared", "author_url": "", "post_date": "09/16/2018 15:16:48", "content": "Yeah, after carefully reading the other comments I realized that i am missing the context. Now I am curious about it cause I wanna understand if there is a mathematical explanation for the increase of performance. Very interesting.
", "votes": null, "replies": [] }, { "id": 388255, "author_name": "ca3295", "author_url": "", "post_date": "09/16/2018 16:36:04", "content": "It's either going to be random noise in the public leaderboard that wont necessarily translate to the final rankings (as implied by Phil), or an error in the scoring. Personally I wouldn't go looking for a mathematical explanation.
", "votes": null, "replies": [] } ] }, { "id": 388890, "author_name": "scirpus", "author_url": "", "post_date": "09/17/2018 19:15:48", "content": "\"where y hat is the predicted revenue for a customer and y is the natural log of the actual revenue value.\"
\n\nSo if I have one user and one entry for that user and the revenue is 2 ...\nThe best error would be (2-log2)*(2-log2) if I predict the revenue exactly\nThe best error is 0 if I predict the log of the actual revenue exactly
\n\nSo we are looking for a function that approximates the metric for very small values of log (sum of entries)
\n\nlog(very tiny+quite large)?\nAm = max(A1,A2)\nlog(log(exp(A1))+log(exp(A2))) -> log(log(exp(Am))) = log(Am)
\n\nDoes anyone have any thoughts? ;)
", "votes": null, "replies": [ { "id": 389858, "author_name": "mrooijer", "author_url": "", "post_date": "09/19/2018 10:22:10", "content": "Yes my interpretation is that that sentence you quote is very inaccurate. It should not be the natural log but the log1p. Ans similarly the y_hat should be that log1p of (the sum of ) predicted revenue(s) for that customer. The data page explains it a lot better.
", "votes": null, "replies": [] }, { "id": 389869, "author_name": "scirpus", "author_url": "", "post_date": "09/19/2018 10:57:53", "content": "I agree log(x+1) is the only thing that makes sense considering a lot of visitors do not produce any revenue.
", "votes": null, "replies": [] } ] }, { "id": 389927, "author_name": "dmitriyguller", "author_url": "", "post_date": "09/19/2018 13:25:08", "content": "\n\n\nMy naive xgboost model scored 1.69 if I sum the log predicted revenue for each user, but scored only 1.91 when I exp the log predicted revenues back to non-log, sum them up, and take log again.
\n
There is a problem with how you're adding up predictions before logging. When you exp the log predicted revenues, you're not getting the correct expected value on the linear scale. What you're getting out of the model is E[log(x)], which is not the same thing as log(E[x]). If you add up the total predictions on the linear scale and compare them against total actuals, you're probably going to find that your total predictions are considerably below total actuals.
", "votes": null, "replies": [ { "id": 391210, "author_name": "agenlu", "author_url": "", "post_date": "09/21/2018 11:59:38", "content": "I think I understand what you are meaning but, what is the correct way to add the predictions up?
", "votes": null, "replies": [] } ] }, { "id": 390942, "author_name": "pavansanagapati", "author_url": "", "post_date": "09/21/2018 04:05:00", "content": "Thanks for this insight very helpful
", "votes": null, "replies": [] }, { "id": 393138, "author_name": "philculliton", "author_url": "", "post_date": "09/24/2018 21:39:00", "content": "Thanks very much to everyone who posted here, your help in finding the problem is greatly appreciated.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "387514": "From the data description, **targetUser = ln ( ∑ all transactions by user i )**\n\nWhere the evaluation metric is RMSE (where y-hat is the predicted revenue for a customer and y is the natural log of the actual revenue value).\n\nMy interpretation of the above would be simply adding up all the non-log transactions of a user, then take the natural log. My experience with the LB, however, appears to be otherwise.\n\nMy naive xgboost model scored 1.69 if I sum the log predicted revenue for each user, but scored only 1.91 when I exp the log predicted revenues back to non-log, sum them up, and take log again.\n\nUsing fullVisitorId 319401084506866864 as an illustration. There are 34 log predicted revenues in test (with max at 18.32), that sums to 39.925. If I were to convert back to non-log first, the 34 predicted revenues would have a sum of 91246965 and the log1p of that is 18.33.\n\nAm I missing something here?", "387554": "ln(a*b) = ln(a) + ln(b) which means that if you sum up all logs of your original values it is as if you multiplied all values together first and then took the log. Does it make sense to multiply all transactions at all? If not, then you should sum them up first and take the log afterwards.", "387564": "Thanks for the heads up. It is indeed either bug or bad explanation from Kaggle side. I just tested it and converting from log of sums to sum of logs boosted me almost 0.1 points. I hope Kaggle reads it and takes an action.", "387762": "I experienced the same phenomenon that you describe. The correct way, according to Kaggle, would be to exp, sum and then take the log, but summing the logs of transactions gives me a lower RMSE than the former.", "387771": "The correct - and expected - way is as noted in the Evaluation and Data pages. I know, because that's precisely how I set the targets. I would recommend you optimize for the expected target.\n\nIt should go without saying, too, that lowering your RMSE on the public test set is NOT the same thing as having a model that is generalizing better.", "387786": "Thanks Phil. That's really good information to know.", "387793": "You're welcome!", "387808": "Hmm, I haven't submitted yet, but locally the log1p(sum of expm1) way looks much better (about .24 lower RMSE). To be specific, I trained at the visit level (predict log1p) before aggregating to the visitor level to get final RMSE, either by doing RMSE(log1p(sum(expm1(y_true))), log1p(sum(expm1(y_pred)))) or just RMSE(sum(y_true),sum(y_pred)). The former is the way that looked better, which is the opposite of what people have said so far about the public LB result.\n\nIt definitely seems odd to me that the LB and local results would show the exact opposite pattern.", "387828": "Locally I also get a better result doing RMSE(log1p(sum(expm1(y_true))) than just summing the logs, but in the LB I get opposite results. Weird.", "388055": "Edit - I'm less confident of this now I've thought about just how noisy the public leaderboard might be due to the skew of the spending.\n\nI think that the target is the sum of the logs, and is a possible error; not only does my leaderboard score improve materially when I switch to submitting this way, it reconciles much better with my local cross validation. \n\nAs Chrisquared says, this is the equivalent of multiplying the spends together, which seems odd.\n\nMy strategy for the two submissions will be one on each scoring method just in case this is a coincidence.", "388066": "Could someone please provide me a link to the source? I think I didnt really get the problem.", "388071": "Hello - I think based on your previous comment you're perhaps missing the fact that for the competition it doesn't matter what it makes sense to do, all that matter is what metric is being used to score submissions. \n\nAs you point out, the sum of the logs is appropriate if you think that values should be combined multiplicatively. This doesn't seem like something that we should want to do, and it isn't the approach being signaled by the organizers, but in practice the sum of the logs is scoring higher than the log of the sums and so there might be an error somewhere in the scoring.\n\nIf it is an error it's kind of an easy one to make, because most of the values are zero, and log(1+0) = 0 (note that we add 1 to the value to avoid log(0) problems). So summing the logs and taking the log of the sum is equivalent for most of the data.", "388104": "Hi Phil, I have done several experiments and I don't get why sum of logs worked better. Can you double check if there is a mistake in your implementation. If you had already double checked, sorry in advance.", "388215": "Yeah, after carefully reading the other comments I realized that i am missing the context. Now I am curious about it cause I wanna understand if there is a mathematical explanation for the increase of performance. Very interesting.", "388255": "It's either going to be random noise in the public leaderboard that wont necessarily translate to the final rankings (as implied by Phil), or an error in the scoring. Personally I wouldn't go looking for a mathematical explanation.", "388890": "\"where y hat is the predicted revenue for a customer and y is the natural log of the actual revenue value.\"\n\nSo if I have one user and one entry for that user and the revenue is 2 ...\nThe best error would be (2-log2)*(2-log2) if I predict the revenue exactly\nThe best error is 0 if I predict the **log** of the actual revenue exactly\n\nSo we are looking for a function that approximates the metric for very small values of log (sum of entries)\n\nlog(very tiny+quite large)?\nAm = max(A1,A2)\nlog(log(exp(A1))+log(exp(A2))) -> log(log(exp(Am))) = log(Am)\n\nDoes anyone have any thoughts? ;)", "389858": "Yes my interpretation is that that sentence you quote is very inaccurate. It should not be the natural log but the log1p. Ans similarly the y_hat should be that log1p of (the sum of ) predicted revenue(s) for that customer. The data page explains it a lot better.", "389869": "I agree log(x+1) is the only thing that makes sense considering a lot of visitors do not produce any revenue.", "389927": "> My naive xgboost model scored 1.69 if I sum the log predicted revenue for each user, but scored only 1.91 when I exp the log predicted revenues back to non-log, sum them up, and take log again.\n\nThere is a problem with how you're adding up predictions before logging. When you exp the log predicted revenues, you're not getting the correct expected value on the linear scale. What you're getting out of the model is E[log(x)], which is not the same thing as log(E[x]). If you add up the total predictions on the linear scale and compare them against total actuals, you're probably going to find that your total predictions are considerably below total actuals.", "390464": "Thanks for the good ideas to bring it. I know a lot more. :)สมัคร ufabet :J", "390942": "Thanks for this insight very helpful", "391210": "I think I understand what you are meaning but, what is the correct way to add the predictions up?", "393138": "Thanks very much to everyone who posted here, your help in finding the problem is greatly appreciated." }, "source": "meta" }