Hi all,
\n\nI noticed that a lot of kernels were transforming the target to log1p(y) to fit a model, then transforming back using expm1. We are estimating E[log1p(y)|X] which is different than log1p(E[y|X]). I tried to take into account this difference by factoring my predictions by the arithmetic mean of the exponentiate residuals of the log model (also known as Smearing estimate). It does something like this:\n$$\\widehat{y} = exp(\\widehat{log1p(y)}) \\times (\\frac{1}{N}\\sum_{i=1}^{N} exp(e_{i})) - 1 \\\\ \\text{with} \\ e_{i} = log1p(y_{i}) - \\widehat{log1p(y_{i})}, \\ i=1...len(train) $$
\n\nI am missing something or not doing it right because my estimate (\\(\\frac{1}{N}\\sum_{i=1}^{N} exp(e_{i})\\)) seems too high and blows off my predictions and my rmse CV. \nThat would be interesting if you had any thoughts on this, maybe it is just not adapted to non-linear models. Or maybe I am overthinking it... Thanks!
\n\nIdeas:
\n\nhttps://stats.stackexchange.com/questions/55692/back-transformation-of-an-mlr-model\nhttps://stats.stackexchange.com/questions/260942/transforming-back-after-a-log-transformation
", "messages": [ { "id": "396079", "postDate": "09/30/2018 01:51:24", "content": "Hi all,
\n\nI noticed that a lot of kernels were transforming the target to log1p(y) to fit a model, then transforming back using expm1. We are estimating E[log1p(y)|X] which is different than log1p(E[y|X]). I tried to take into account this difference by factoring my predictions by the arithmetic mean of the exponentiate residuals of the log model (also known as Smearing estimate). It does something like this:\n$$\\widehat{y} = exp(\\widehat{log1p(y)}) \\times (\\frac{1}{N}\\sum_{i=1}^{N} exp(e_{i})) - 1 \\\\ \\text{with} \\ e_{i} = log1p(y_{i}) - \\widehat{log1p(y_{i})}, \\ i=1...len(train) $$
\n\nI am missing something or not doing it right because my estimate (\\(\\frac{1}{N}\\sum_{i=1}^{N} exp(e_{i})\\)) seems too high and blows off my predictions and my rmse CV. \nThat would be interesting if you had any thoughts on this, maybe it is just not adapted to non-linear models. Or maybe I am overthinking it... Thanks!
\n\nIdeas:
\n\nhttps://stats.stackexchange.com/questions/55692/back-transformation-of-an-mlr-model\nhttps://stats.stackexchange.com/questions/260942/transforming-back-after-a-log-transformation
", "rawMarkdown": "Hi all,\n\nI noticed that a lot of kernels were transforming the target to log1p(y) to fit a model, then transforming back using expm1. We are estimating E[log1p(y)|X] which is different than log1p(E[y|X]). I tried to take into account this difference by factoring my predictions by the arithmetic mean of the exponentiate residuals of the log model (also known as Smearing estimate). It does something like this:\n$$\\widehat{y} = exp(\\widehat{log1p(y)}) \\times (\\frac{1}{N}\\sum_{i=1}^{N} exp(e_{i})) - 1 \\\\\\ \\text{with} \\ e_{i} = log1p(y_{i}) - \\widehat{log1p(y_{i})}, \\ i=1...len(train) $$\n\n\nI am missing something or not doing it right because my estimate (\\\\(\\frac{1}{N}\\sum_{i=1}^{N} exp(e_{i})\\\\)) seems too high and blows off my predictions and my rmse CV. \nThat would be interesting if you had any thoughts on this, maybe it is just not adapted to non-linear models. Or maybe I am overthinking it... Thanks!\n\nIdeas:\n\nhttps://stats.stackexchange.com/questions/55692/back-transformation-of-an-mlr-model\nhttps://stats.stackexchange.com/questions/260942/transforming-back-after-a-log-transformation", "votes": null }, { "id": "396095", "postDate": "09/30/2018 02:35:45", "content": "Two things:
\n\nlog1p(y) is normally distributed, but this is not the case, the data is severely right skewed.exp(1/2 * sigma^2) = exp(1/(2*(N - 1)) * \\Sigma (yhat - mean(yhat))^2) = exp(1/2 * var(yhat)) but this is not what you are multiplying by. What if you change to$$\\exp{(\\mathrm{log1p}(y))} * \\exp{(\\frac{1}{2} \\mathrm{var}(y))} - 1$$
\n\nPlease reply back letting me know if predictions worsened, improved, or stayed the same..
", "rawMarkdown": "Two things:\n\n1. According to https://en.wikipedia.org/wiki/Smearing_retransformation , this retransformation is only valid if `log1p(y)` is normally distributed, but this is not the case, the data is severely right skewed.\n2. Even if it were distributed normally, you aren't using the formula given in that Wikipedia article. it says to use `exp(1/2 * sigma^2) = exp(1/(2*(N - 1)) * \\Sigma (yhat - mean(yhat))^2) = exp(1/2 * var(yhat))` but this is not what you are multiplying by. What if you change to\n\n$$\\exp{(\\mathrm{log1p}(y))} * \\exp{(\\frac{1}{2} \\mathrm{var}(y))} - 1$$\n\nPlease reply back letting me know if predictions worsened, improved, or stayed the same..", "votes": null }, { "id": "396099", "postDate": "09/30/2018 02:46:36", "content": "anyway i dont think this is the way the competition should be attempted, because it implies that you are log transforming everything at the transaction level, and then summing those predictions to an answer, as opposed to making 1 prediction for the aggregate revenue. I'm not trying to deter you, it's just that if your transaction prediction has an error epsilon, then if a user has n visits to the website, your total error in the sum is bounded by n * epsilon, but if you predict at an aggregate level your error can be tau, and it is likely that there exists a threshold amount of visits N such that for all n > N, we have tau < n * epsilon. Not trying to steer you away from a transaction-level model, but you need to keep this in mind, which in my mind makes that model unfavorable
Hey Corey, very good points thanks. I am not exactly using the transformation stated on the wiki page but a non-parametric estimate (please refer to 1st link, I did not find an open source version but the paper is Smearing Estimate: A Nonparametric Retransformation Method ). I totally agree that the log1p(y) may be too skewed to satisfy the assumption of normality. Anyway, I just tried with \\(exp(\\frac{1}{2}var(y)\\)) and it worsen my cv by 0.1. I agree that training at transaction level may not be optimal but I am still trying both approaches!
", "rawMarkdown": "Hey Corey, very good points thanks. I am not exactly using the transformation stated on the wiki page but a non-parametric estimate (please refer to 1st link, I did not find an open source version but the paper is Smearing Estimate: A Nonparametric Retransformation Method ). I totally agree that the log1p(y) may be too skewed to satisfy the assumption of normality. Anyway, I just tried with \\\\(exp(\\frac{1}{2}var(y)\\\\)) and it worsen my cv by 0.1. I agree that training at transaction level may not be optimal but I am still trying both approaches!", "votes": null }, { "id": "396131", "postDate": "09/30/2018 04:05:19", "content": "Maybe run an optimizer function to find out what value c minimizes RMSE(exp(yhat)*c - 1), and from there it's possible to discover the formula for c...but this could be overfit to trainng set at this point. I dont know, but I think your Discussion Topic is extremely important. Let me know if you make progress on it.
Great point, looking forward to further discussion on this. Would such a transformation apply to decision trees ?
", "rawMarkdown": "Great point, looking forward to further discussion on this. Would such a transformation apply to decision trees ?", "votes": null }, { "id": "396571", "postDate": "10/01/2018 02:40:27", "content": "I have same question, because for a tree based algorithms we can't write \nYhat = (theta)*(input) + error.
\n\nThis is the assumption only for linear regression. So I don't think it is applicable to tree based methods. Even the adri says that 1/N * sum (exp(error)) is very high. I think this will not happen in case of linear regression as we assume that error terms have mean zero.
", "rawMarkdown": "I have same question, because for a tree based algorithms we can't write \nYhat = (theta)*(input) + error.\n\nThis is the assumption only for linear regression. So I don't think it is applicable to tree based methods. Even the adri says that 1/N * sum (exp(error)) is very high. I think this will not happen in case of linear regression as we assume that error terms have mean zero.", "votes": null }, { "id": "396625", "postDate": "10/01/2018 05:13:42", "content": "Yes, I have the same concerns
", "rawMarkdown": "Yes, I have the same concerns", "votes": null }, { "id": "396640", "postDate": "10/01/2018 05:40:49", "content": "So adri I saw for some examples the actual revenue is 0 but the predicted value was near 20. So when you take the exp(20) it will be huge. So I think it's better for tree based models, you directly take the exp of predicted value and ignore that error term there.
", "rawMarkdown": "So adri I saw for some examples the actual revenue is 0 but the predicted value was near 20. So when you take the exp(20) it will be huge. So I think it's better for tree based models, you directly take the exp of predicted value and ignore that error term there.", "votes": null } ], "comments": [ { "id": 396095, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/30/2018 02:35:45", "content": "Two things:
\n\nlog1p(y) is normally distributed, but this is not the case, the data is severely right skewed.exp(1/2 * sigma^2) = exp(1/(2*(N - 1)) * \\Sigma (yhat - mean(yhat))^2) = exp(1/2 * var(yhat)) but this is not what you are multiplying by. What if you change to$$\\exp{(\\mathrm{log1p}(y))} * \\exp{(\\frac{1}{2} \\mathrm{var}(y))} - 1$$
\n\nPlease reply back letting me know if predictions worsened, improved, or stayed the same..
", "votes": null, "replies": [ { "id": 396099, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/30/2018 02:46:36", "content": "anyway i dont think this is the way the competition should be attempted, because it implies that you are log transforming everything at the transaction level, and then summing those predictions to an answer, as opposed to making 1 prediction for the aggregate revenue. I'm not trying to deter you, it's just that if your transaction prediction has an error epsilon, then if a user has n visits to the website, your total error in the sum is bounded by n * epsilon, but if you predict at an aggregate level your error can be tau, and it is likely that there exists a threshold amount of visits N such that for all n > N, we have tau < n * epsilon. Not trying to steer you away from a transaction-level model, but you need to keep this in mind, which in my mind makes that model unfavorable
Hey Corey, very good points thanks. I am not exactly using the transformation stated on the wiki page but a non-parametric estimate (please refer to 1st link, I did not find an open source version but the paper is Smearing Estimate: A Nonparametric Retransformation Method ). I totally agree that the log1p(y) may be too skewed to satisfy the assumption of normality. Anyway, I just tried with \\(exp(\\frac{1}{2}var(y)\\)) and it worsen my cv by 0.1. I agree that training at transaction level may not be optimal but I am still trying both approaches!
", "votes": null, "replies": [] }, { "id": 396131, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/30/2018 04:05:19", "content": "Maybe run an optimizer function to find out what value c minimizes RMSE(exp(yhat)*c - 1), and from there it's possible to discover the formula for c...but this could be overfit to trainng set at this point. I dont know, but I think your Discussion Topic is extremely important. Let me know if you make progress on it.
Great point, looking forward to further discussion on this. Would such a transformation apply to decision trees ?
", "votes": null, "replies": [ { "id": 396571, "author_name": "suchith0312", "author_url": "", "post_date": "10/01/2018 02:40:27", "content": "I have same question, because for a tree based algorithms we can't write \nYhat = (theta)*(input) + error.
\n\nThis is the assumption only for linear regression. So I don't think it is applicable to tree based methods. Even the adri says that 1/N * sum (exp(error)) is very high. I think this will not happen in case of linear regression as we assume that error terms have mean zero.
", "votes": null, "replies": [] }, { "id": 396625, "author_name": "tita1708", "author_url": "", "post_date": "10/01/2018 05:13:42", "content": "Yes, I have the same concerns
", "votes": null, "replies": [] }, { "id": 396640, "author_name": "suchith0312", "author_url": "", "post_date": "10/01/2018 05:40:49", "content": "So adri I saw for some examples the actual revenue is 0 but the predicted value was near 20. So when you take the exp(20) it will be huge. So I think it's better for tree based models, you directly take the exp of predicted value and ignore that error term there.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "396079": "Hi all,\n\nI noticed that a lot of kernels were transforming the target to log1p(y) to fit a model, then transforming back using expm1. We are estimating E[log1p(y)|X] which is different than log1p(E[y|X]). I tried to take into account this difference by factoring my predictions by the arithmetic mean of the exponentiate residuals of the log model (also known as Smearing estimate). It does something like this:\n$$\\widehat{y} = exp(\\widehat{log1p(y)}) \\times (\\frac{1}{N}\\sum_{i=1}^{N} exp(e_{i})) - 1 \\\\\\ \\text{with} \\ e_{i} = log1p(y_{i}) - \\widehat{log1p(y_{i})}, \\ i=1...len(train) $$\n\n\nI am missing something or not doing it right because my estimate (\\\\(\\frac{1}{N}\\sum_{i=1}^{N} exp(e_{i})\\\\)) seems too high and blows off my predictions and my rmse CV. \nThat would be interesting if you had any thoughts on this, maybe it is just not adapted to non-linear models. Or maybe I am overthinking it... Thanks!\n\nIdeas:\n\nhttps://stats.stackexchange.com/questions/55692/back-transformation-of-an-mlr-model\nhttps://stats.stackexchange.com/questions/260942/transforming-back-after-a-log-transformation", "396095": "Two things:\n\n1. According to https://en.wikipedia.org/wiki/Smearing_retransformation , this retransformation is only valid if `log1p(y)` is normally distributed, but this is not the case, the data is severely right skewed.\n2. Even if it were distributed normally, you aren't using the formula given in that Wikipedia article. it says to use `exp(1/2 * sigma^2) = exp(1/(2*(N - 1)) * \\Sigma (yhat - mean(yhat))^2) = exp(1/2 * var(yhat))` but this is not what you are multiplying by. What if you change to\n\n$$\\exp{(\\mathrm{log1p}(y))} * \\exp{(\\frac{1}{2} \\mathrm{var}(y))} - 1$$\n\nPlease reply back letting me know if predictions worsened, improved, or stayed the same..", "396099": "anyway i dont think this is the way the competition should be attempted, because it implies that you are log transforming everything at the transaction level, and then summing those predictions to an answer, as opposed to making 1 prediction for the aggregate revenue. I'm not trying to deter you, it's just that if your transaction prediction has an error `epsilon`, then if a user has `n` visits to the website, your total error in the sum is bounded by `n * epsilon`, but if you predict at an aggregate level your error can be `tau`, and it is likely that there exists a threshold amount of visits `N` such that for all `n > N`, we have `tau < n * epsilon`. Not trying to steer you away from a transaction-level model, but you need to keep this in mind, which in my mind makes that model unfavorable", "396127": "Hey Corey, very good points thanks. I am not exactly using the transformation stated on the wiki page but a non-parametric estimate (please refer to 1st link, I did not find an open source version but the paper is Smearing Estimate: A Nonparametric Retransformation Method ). I totally agree that the log1p(y) may be too skewed to satisfy the assumption of normality. Anyway, I just tried with \\\\(exp(\\frac{1}{2}var(y)\\\\)) and it worsen my cv by 0.1. I agree that training at transaction level may not be optimal but I am still trying both approaches!", "396131": "Maybe run an optimizer function to find out what value `c` minimizes RMSE(exp(yhat)*c - 1), and from there it's possible to discover the formula for `c`...but this could be overfit to trainng set at this point. I dont know, but I think your Discussion Topic is extremely important. Let me know if you make progress on it.", "396536": "Great point, looking forward to further discussion on this. Would such a transformation apply to decision trees ?", "396571": "I have same question, because for a tree based algorithms we can't write \nYhat = (theta)*(input) + error.\n\nThis is the assumption only for linear regression. So I don't think it is applicable to tree based methods. Even the adri says that 1/N * sum (exp(error)) is very high. I think this will not happen in case of linear regression as we assume that error terms have mean zero.", "396625": "Yes, I have the same concerns", "396640": "So adri I saw for some examples the actual revenue is 0 but the predicted value was near 20. So when you take the exp(20) it will be huge. So I think it's better for tree based models, you directly take the exp of predicted value and ignore that error term there." }, "source": "meta" }