Probably a minor point, but given so many users have 0 transaction value, the natural log of that is infinity? Is it better to log(1+x) or manually change the infinity to 0? This depends on how the organizer have prepared the test/actual (i.e. log or log1p).
", "messages": [ { "id": "387493", "postDate": "09/15/2018 04:18:13", "content": "Probably a minor point, but given so many users have 0 transaction value, the natural log of that is infinity? Is it better to log(1+x) or manually change the infinity to 0? This depends on how the organizer have prepared the test/actual (i.e. log or log1p).
", "rawMarkdown": "Probably a minor point, but given so many users have 0 transaction value, the natural log of that is infinity? Is it better to log(1+x) or manually change the infinity to 0? This depends on how the organizer have prepared the test/actual (i.e. log or log1p).", "votes": null }, { "id": "387500", "postDate": "09/15/2018 04:34:28", "content": "i think it log(1+x) rather than log(x)
\n\nhttps://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65691#387188
", "rawMarkdown": "i think it log(1+x) rather than log(x)\n\nhttps://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65691#387188", "votes": null }, { "id": "387503", "postDate": "09/15/2018 04:35:52", "content": "Thanks. This should be clear once the data description page is updated.
", "rawMarkdown": "Thanks. This should be clear once the data description page is updated.", "votes": null }, { "id": "387523", "postDate": "09/15/2018 05:17:38", "content": "When you transform a function f(x) to f(x-a) then the graph of it shifts to the right for a units. In your case you want to transform ln(x) to ln(x+1) which means a=-1. So you shift your graph left for one unit. This also means that you shift the range of valid values for x by one unit to the left. So far so good. \nWhether this is a good idea or not depends on the application.
\n\nAs long as you dont interpret the x values as relations or factors, like you would do in finance e.g. (where a factor of 1.05 means plus 5% and a factor of 1 means plus 0%) you will be fine. \nSince your feature is named \"transaction value\" I suggest that it is an absolute value and not a relative one.\nHence you should be fine with shifting your ln(x) to ln(x+1).
\n\nps. some more general information about this topic:
\n\nI would also like to give you a scenario with relative values and an explanation why the ln() works so good:\nConsider you have two factos like 2 and 0.5 . \nMultiplying by 2 menas you get the double result where multiplying with 0.5 means you get the half result. (dividing by two instead of multiplying by two)\nSo even if they are somehow the \"opposite\" of each other, there is no symmetry in the numbers since 0.5 is 0.5 units away from the factor 1 and 2 is 1 unit away from the factor 1. (multiplying by 1 means you change nothing at all, so this is why you compare your factors to the factor 1) \nThe situation is more clear when you consider the pair of factors 1/10 and 10. Both factors do the same but in the opposite way but the distances from the factor 1 are (very) unequal. (0.1-1=0.9 not equal to 9=10-1)
\n\nNow this is where the log comes into play:\nln(1) = 0 which is good, because multiplying by 1 does nothing.\nWhen you use your calculator and compute ln(0.5) you will get the negative value of ln(2). This means that the logarithm understands that in a relative way 0.5 is as far away from 1 as 2 is. (but in the opposite direction)\nSame holds for ln(1/10) = -ln(10) and all possible pairs.
\n\nSo after taking the logarithm of a relative value we have a symmetric scale around the value 1.(edit: which in fact is 0 but before the log it was 1)\nThis is also the reason why we use the log of odd ratios in logistic regression (called logits).
\n\nIf you see this connection you will never struggle again with your question.
", "rawMarkdown": "When you transform a function f(x) to f(x-a) then the graph of it shifts to the right for a units. In your case you want to transform ln(x) to ln(x+1) which means a=-1. So you shift your graph left for one unit. This also means that you shift the range of valid values for x by one unit to the left. So far so good. \nWhether this is a good idea or not depends on the application.\n\nAs long as you dont interpret the x values as relations or factors, like you would do in finance e.g. (where a factor of 1.05 means plus 5% and a factor of 1 means plus 0%) you will be fine. \nSince your feature is named \"transaction value\" I suggest that it is an absolute value and not a relative one.\nHence you should be fine with shifting your ln(x) to ln(x+1).\n\n\nps. some more general information about this topic:\n\nI would also like to give you a scenario with relative values and an explanation why the ln() works so good:\nConsider you have two factos like 2 and 0.5 . \nMultiplying by 2 menas you get the double result where multiplying with 0.5 means you get the half result. (dividing by two instead of multiplying by two)\nSo even if they are somehow the \"opposite\" of each other, there is no symmetry in the numbers since 0.5 is 0.5 units away from the factor 1 and 2 is 1 unit away from the factor 1. (multiplying by 1 means you change nothing at all, so this is why you compare your factors to the factor 1) \nThe situation is more clear when you consider the pair of factors 1/10 and 10. Both factors do the same but in the opposite way but the distances from the factor 1 are (very) unequal. (0.1-1=0.9 not equal to 9=10-1)\n\nNow this is where the log comes into play:\nln(1) = 0 which is good, because multiplying by 1 does nothing.\nWhen you use your calculator and compute ln(0.5) you will get the negative value of ln(2). This means that the logarithm understands that in a relative way 0.5 is as far away from 1 as 2 is. (but in the opposite direction)\nSame holds for ln(1/10) = -ln(10) and all possible pairs.\n\nSo after taking the logarithm of a relative value we have a symmetric scale around the value 1.(edit: which in fact is 0 but before the log it was 1)\nThis is also the reason why we use the log of odd ratios in logistic regression (called logits).\n\nIf you see this connection you will never struggle again with your question.", "votes": null }, { "id": "387625", "postDate": "09/15/2018 09:43:04", "content": "Thanks for the explanation!
", "rawMarkdown": "Thanks for the explanation!", "votes": null }, { "id": "389690", "postDate": "09/19/2018 04:54:04", "content": "Thanks a lot for the explanation.
", "rawMarkdown": "Thanks a lot for the explanation.", "votes": null }, { "id": "389723", "postDate": "09/19/2018 05:58:17", "content": "Can you please comment on my kernel here on my conclusion of RMSLE?
", "rawMarkdown": "Can you please comment on my kernel here on my [conclusion of RMSLE][1]? \n\n\n [1]: https://www.kaggle.com/krisho007/rmsle-analysis", "votes": null }, { "id": "389725", "postDate": "09/19/2018 06:03:22", "content": "Okay, I will take my time to think about it and comment it.
", "rawMarkdown": "Okay, I will take my time to think about it and comment it.", "votes": null } ], "comments": [ { "id": 387500, "author_name": "suchith0312", "author_url": "", "post_date": "09/15/2018 04:34:28", "content": "i think it log(1+x) rather than log(x)
\n\nhttps://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65691#387188
", "votes": null, "replies": [ { "id": 387503, "author_name": "yilihome", "author_url": "", "post_date": "09/15/2018 04:35:52", "content": "Thanks. This should be clear once the data description page is updated.
", "votes": null, "replies": [] } ] }, { "id": 387523, "author_name": "chrisquared", "author_url": "", "post_date": "09/15/2018 05:17:38", "content": "When you transform a function f(x) to f(x-a) then the graph of it shifts to the right for a units. In your case you want to transform ln(x) to ln(x+1) which means a=-1. So you shift your graph left for one unit. This also means that you shift the range of valid values for x by one unit to the left. So far so good. \nWhether this is a good idea or not depends on the application.
\n\nAs long as you dont interpret the x values as relations or factors, like you would do in finance e.g. (where a factor of 1.05 means plus 5% and a factor of 1 means plus 0%) you will be fine. \nSince your feature is named \"transaction value\" I suggest that it is an absolute value and not a relative one.\nHence you should be fine with shifting your ln(x) to ln(x+1).
\n\nps. some more general information about this topic:
\n\nI would also like to give you a scenario with relative values and an explanation why the ln() works so good:\nConsider you have two factos like 2 and 0.5 . \nMultiplying by 2 menas you get the double result where multiplying with 0.5 means you get the half result. (dividing by two instead of multiplying by two)\nSo even if they are somehow the \"opposite\" of each other, there is no symmetry in the numbers since 0.5 is 0.5 units away from the factor 1 and 2 is 1 unit away from the factor 1. (multiplying by 1 means you change nothing at all, so this is why you compare your factors to the factor 1) \nThe situation is more clear when you consider the pair of factors 1/10 and 10. Both factors do the same but in the opposite way but the distances from the factor 1 are (very) unequal. (0.1-1=0.9 not equal to 9=10-1)
\n\nNow this is where the log comes into play:\nln(1) = 0 which is good, because multiplying by 1 does nothing.\nWhen you use your calculator and compute ln(0.5) you will get the negative value of ln(2). This means that the logarithm understands that in a relative way 0.5 is as far away from 1 as 2 is. (but in the opposite direction)\nSame holds for ln(1/10) = -ln(10) and all possible pairs.
\n\nSo after taking the logarithm of a relative value we have a symmetric scale around the value 1.(edit: which in fact is 0 but before the log it was 1)\nThis is also the reason why we use the log of odd ratios in logistic regression (called logits).
\n\nIf you see this connection you will never struggle again with your question.
", "votes": null, "replies": [ { "id": 387625, "author_name": "yilihome", "author_url": "", "post_date": "09/15/2018 09:43:04", "content": "Thanks for the explanation!
", "votes": null, "replies": [] }, { "id": 389690, "author_name": "krisho007", "author_url": "", "post_date": "09/19/2018 04:54:04", "content": "Thanks a lot for the explanation.
", "votes": null, "replies": [] }, { "id": 389723, "author_name": "krisho007", "author_url": "", "post_date": "09/19/2018 05:58:17", "content": "Can you please comment on my kernel here on my conclusion of RMSLE?
", "votes": null, "replies": [] }, { "id": 389725, "author_name": "chrisquared", "author_url": "", "post_date": "09/19/2018 06:03:22", "content": "Okay, I will take my time to think about it and comment it.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "387493": "Probably a minor point, but given so many users have 0 transaction value, the natural log of that is infinity? Is it better to log(1+x) or manually change the infinity to 0? This depends on how the organizer have prepared the test/actual (i.e. log or log1p).", "387500": "i think it log(1+x) rather than log(x)\n\nhttps://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65691#387188", "387503": "Thanks. This should be clear once the data description page is updated.", "387523": "When you transform a function f(x) to f(x-a) then the graph of it shifts to the right for a units. In your case you want to transform ln(x) to ln(x+1) which means a=-1. So you shift your graph left for one unit. This also means that you shift the range of valid values for x by one unit to the left. So far so good. \nWhether this is a good idea or not depends on the application.\n\nAs long as you dont interpret the x values as relations or factors, like you would do in finance e.g. (where a factor of 1.05 means plus 5% and a factor of 1 means plus 0%) you will be fine. \nSince your feature is named \"transaction value\" I suggest that it is an absolute value and not a relative one.\nHence you should be fine with shifting your ln(x) to ln(x+1).\n\n\nps. some more general information about this topic:\n\nI would also like to give you a scenario with relative values and an explanation why the ln() works so good:\nConsider you have two factos like 2 and 0.5 . \nMultiplying by 2 menas you get the double result where multiplying with 0.5 means you get the half result. (dividing by two instead of multiplying by two)\nSo even if they are somehow the \"opposite\" of each other, there is no symmetry in the numbers since 0.5 is 0.5 units away from the factor 1 and 2 is 1 unit away from the factor 1. (multiplying by 1 means you change nothing at all, so this is why you compare your factors to the factor 1) \nThe situation is more clear when you consider the pair of factors 1/10 and 10. Both factors do the same but in the opposite way but the distances from the factor 1 are (very) unequal. (0.1-1=0.9 not equal to 9=10-1)\n\nNow this is where the log comes into play:\nln(1) = 0 which is good, because multiplying by 1 does nothing.\nWhen you use your calculator and compute ln(0.5) you will get the negative value of ln(2). This means that the logarithm understands that in a relative way 0.5 is as far away from 1 as 2 is. (but in the opposite direction)\nSame holds for ln(1/10) = -ln(10) and all possible pairs.\n\nSo after taking the logarithm of a relative value we have a symmetric scale around the value 1.(edit: which in fact is 0 but before the log it was 1)\nThis is also the reason why we use the log of odd ratios in logistic regression (called logits).\n\nIf you see this connection you will never struggle again with your question.", "387625": "Thanks for the explanation!", "389690": "Thanks a lot for the explanation.", "389723": "Can you please comment on my kernel here on my [conclusion of RMSLE][1]? \n\n\n [1]: https://www.kaggle.com/krisho007/rmsle-analysis", "389725": "Okay, I will take my time to think about it and comment it." }, "source": "meta" }