Hi,
\n\nI saw people used numpy.log1p to transform their data before modeling. I am wondering why people do that (instead of log) and what the mathematical explanation is? Thanks!
", "messages": [ { "id": "266518", "postDate": "01/09/2018 02:08:20", "content": "Hi,
\n\nI saw people used numpy.log1p to transform their data before modeling. I am wondering why people do that (instead of log) and what the mathematical explanation is? Thanks!
", "rawMarkdown": "Hi,\n\nI saw people used numpy.log1p to transform their data before modeling. I am wondering why people do that (instead of log) and what the mathematical explanation is? Thanks!", "votes": null }, { "id": "266531", "postDate": "01/09/2018 02:51:56", "content": "The short answer is logarithm of zero is undefined.
", "rawMarkdown": "The short answer is logarithm of zero is undefined.", "votes": null }, { "id": "266551", "postDate": "01/09/2018 03:52:17", "content": "If you have data that is not normally distributed, you're limited in how you can approach/analyze/model it. Normal distributions are convenient and the tools we have for dealing with them are powerful. Thus, many techniques involve making things look like normal distributions.
\n\nFor example, if the data is bi-modal, then we can ask if it looks like the sum of two different normal distributions. If the data is skewed, then we can ask if it looks log-normal. Etc., etc. In our case, a lot of the data appears to be log-normal.
\n\nSee: https://en.wikipedia.org/wiki/Log-normal_distribution
", "rawMarkdown": "If you have data that is not normally distributed, you're limited in how you can approach/analyze/model it. Normal distributions are convenient and the tools we have for dealing with them are powerful. Thus, many techniques involve making things look like normal distributions.\n\nFor example, if the data is bi-modal, then we can ask if it looks like the sum of two different normal distributions. If the data is skewed, then we can ask if it looks log-normal. Etc., etc. In our case, a lot of the data appears to be log-normal.\n\nSee: https://en.wikipedia.org/wiki/Log-normal_distribution", "votes": null }, { "id": "266553", "postDate": "01/09/2018 03:59:28", "content": "I should add, there are many techniques for transforming data. It's more of an art, than a science. Statisticians always have pretty fancy arsenals. But you can get by with a few simple tricks when you know how to use them correctly. But remember the goal is to get something that is easier to model, i.e. gets a better fit. So you can always test something out and see how it performs if you're unsure about whether a certain transformation is appropriate or not.
", "rawMarkdown": "I should add, there are many techniques for transforming data. It's more of an art, than a science. Statisticians always have pretty fancy arsenals. But you can get by with a few simple tricks when you know how to use them correctly. But remember the goal is to get something that is easier to model, i.e. gets a better fit. So you can always test something out and see how it performs if you're unsure about whether a certain transformation is appropriate or not.", "votes": null }, { "id": "266862", "postDate": "01/10/2018 01:37:25", "content": "0
", "rawMarkdown": "0", "votes": null }, { "id": "266910", "postDate": "01/10/2018 05:49:09", "content": "It is because of the evaluation metric. If you transform the unit sales with log1p then the evaluation metric is the same as rmse with weights, and you can use built-in objective (loss) functions from your favorite algorithm.
\n\nThe next question is why did they select log1p? 2 reasons:
\n\nlog transform makes large sales look similar to small sales: a x % variation will cost the same, whatever the baseline is.
As Ceshine Lee wrote, log transform cannot be applied directly given there can be 0 unit sales. Log1p is one way to mitigate this, but it is not keeping the good property of variation cost being independent from baseline. I would have used something like log(1 + a x) where a is significantly larger than 1 to get best of both. That's what i used in the Web Traffic Forecasting competition, and it led to much better results than log1p. But here we have to use log1p to match the evaluation metrics anyway...
Thanks for the elegant explanation.
\n\nDirectly using log(1 + x) does have a side effect of de-emphasizing the variation for smaller unit sales. I guess whether it is intentional or not can only be answered by Kaggle and the sponsor.
", "rawMarkdown": "Thanks for the elegant explanation.\n\nDirectly using log(1 + x) does have a side effect of de-emphasizing the variation for smaller unit sales. I guess whether it is intentional or not can only be answered by Kaggle and the sponsor.", "votes": null }, { "id": "267002", "postDate": "01/10/2018 10:47:18", "content": "Thanks.
\n\n> Directly using log(1 + x) does have a side effect of de-emphasizing the variation for smaller unit sales
\n\nExactly, small volume series do not matter much here.
", "rawMarkdown": "Thanks.\n\n> Directly using log(1 + x) does have a side effect of de-emphasizing the variation for smaller unit sales\n\nExactly, small volume series do not matter much here.", "votes": null }, { "id": "1210512", "postDate": "02/19/2021 13:46:03", "content": "applied log1p which make my predictions 1 and zeros but i am doing a regresson task can please help me
", "rawMarkdown": "applied log1p which make my predictions 1 and zeros but i am doing a regresson task can please help me", "votes": null } ], "comments": [ { "id": 1210512, "author_name": "subbugalam", "author_url": "", "post_date": "02/19/2021 13:46:03", "content": "applied log1p which make my predictions 1 and zeros but i am doing a regresson task can please help me
", "votes": null, "replies": [] }, { "id": 266531, "author_name": "ceshine", "author_url": "", "post_date": "01/09/2018 02:51:56", "content": "The short answer is logarithm of zero is undefined.
", "votes": null, "replies": [] }, { "id": 266551, "author_name": "puremath86", "author_url": "", "post_date": "01/09/2018 03:52:17", "content": "If you have data that is not normally distributed, you're limited in how you can approach/analyze/model it. Normal distributions are convenient and the tools we have for dealing with them are powerful. Thus, many techniques involve making things look like normal distributions.
\n\nFor example, if the data is bi-modal, then we can ask if it looks like the sum of two different normal distributions. If the data is skewed, then we can ask if it looks log-normal. Etc., etc. In our case, a lot of the data appears to be log-normal.
\n\nSee: https://en.wikipedia.org/wiki/Log-normal_distribution
", "votes": null, "replies": [ { "id": 266553, "author_name": "puremath86", "author_url": "", "post_date": "01/09/2018 03:59:28", "content": "I should add, there are many techniques for transforming data. It's more of an art, than a science. Statisticians always have pretty fancy arsenals. But you can get by with a few simple tricks when you know how to use them correctly. But remember the goal is to get something that is easier to model, i.e. gets a better fit. So you can always test something out and see how it performs if you're unsure about whether a certain transformation is appropriate or not.
", "votes": null, "replies": [] } ] }, { "id": 266862, "author_name": "zhoukj", "author_url": "", "post_date": "01/10/2018 01:37:25", "content": "0
", "votes": null, "replies": [] }, { "id": 266910, "author_name": "cpmpml", "author_url": "", "post_date": "01/10/2018 05:49:09", "content": "It is because of the evaluation metric. If you transform the unit sales with log1p then the evaluation metric is the same as rmse with weights, and you can use built-in objective (loss) functions from your favorite algorithm.
\n\nThe next question is why did they select log1p? 2 reasons:
\n\nlog transform makes large sales look similar to small sales: a x % variation will cost the same, whatever the baseline is.
As Ceshine Lee wrote, log transform cannot be applied directly given there can be 0 unit sales. Log1p is one way to mitigate this, but it is not keeping the good property of variation cost being independent from baseline. I would have used something like log(1 + a x) where a is significantly larger than 1 to get best of both. That's what i used in the Web Traffic Forecasting competition, and it led to much better results than log1p. But here we have to use log1p to match the evaluation metrics anyway...
Thanks for the elegant explanation.
\n\nDirectly using log(1 + x) does have a side effect of de-emphasizing the variation for smaller unit sales. I guess whether it is intentional or not can only be answered by Kaggle and the sponsor.
", "votes": null, "replies": [] }, { "id": 267002, "author_name": "cpmpml", "author_url": "", "post_date": "01/10/2018 10:47:18", "content": "Thanks.
\n\n> Directly using log(1 + x) does have a side effect of de-emphasizing the variation for smaller unit sales
\n\nExactly, small volume series do not matter much here.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "266518": "Hi,\n\nI saw people used numpy.log1p to transform their data before modeling. I am wondering why people do that (instead of log) and what the mathematical explanation is? Thanks!", "266531": "The short answer is logarithm of zero is undefined.", "266551": "If you have data that is not normally distributed, you're limited in how you can approach/analyze/model it. Normal distributions are convenient and the tools we have for dealing with them are powerful. Thus, many techniques involve making things look like normal distributions.\n\nFor example, if the data is bi-modal, then we can ask if it looks like the sum of two different normal distributions. If the data is skewed, then we can ask if it looks log-normal. Etc., etc. In our case, a lot of the data appears to be log-normal.\n\nSee: https://en.wikipedia.org/wiki/Log-normal_distribution", "266553": "I should add, there are many techniques for transforming data. It's more of an art, than a science. Statisticians always have pretty fancy arsenals. But you can get by with a few simple tricks when you know how to use them correctly. But remember the goal is to get something that is easier to model, i.e. gets a better fit. So you can always test something out and see how it performs if you're unsure about whether a certain transformation is appropriate or not.", "266862": "0", "266910": "It is because of the evaluation metric. If you transform the unit sales with log1p then the evaluation metric is the same as rmse with weights, and you can use built-in objective (loss) functions from your favorite algorithm.\n\nThe next question is why did they select log1p? 2 reasons:\n\n 1. log transform makes large sales look similar to small sales: a x % variation will cost the same, whatever the baseline is.\n\n 2. As Ceshine Lee wrote, log transform cannot be applied directly given there can be 0 unit sales. Log1p is one way to mitigate this, but it is not keeping the good property of variation cost being independent from baseline. I would have used something like log(1 + a x) where a is significantly larger than 1 to get best of both. That's what i used in the Web Traffic Forecasting competition, and it led to much better results than log1p. But here we have to use log1p to match the evaluation metrics anyway...", "266938": "Thanks for the elegant explanation.\n\nDirectly using log(1 + x) does have a side effect of de-emphasizing the variation for smaller unit sales. I guess whether it is intentional or not can only be answered by Kaggle and the sponsor.", "267002": "Thanks.\n\n> Directly using log(1 + x) does have a side effect of de-emphasizing the variation for smaller unit sales\n\nExactly, small volume series do not matter much here.", "1210512": "applied log1p which make my predictions 1 and zeros but i am doing a regresson task can please help me" }, "source": "meta" }