![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1594747%2F62214f5324f1c4b01376116127c2d7e6%2FSum_Log.JPG?generation=1573373981113759&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1594747%2Fb68ab76ea5d314129ccaaed14ba0814b%2FLog_Sum.JPG?generation=1573373982570514&alt=media\")
Does anyone know the correct way of interpreting this problem?\nSo I've done a log transform of the target variable as it's highly skewed. This improves the skewness drastically and is much easier to view/interpret (happy days).
\n\nNow I wanted to create a daily time series chart of 3 meter_types (electric, steam etc). In order to do this i need to group_by (date, meter) and take the sum of the meter_reading column. However does it make sense to take the sum of the log transformed data? or should i take the sum of the meter_reading column then log transform the sum?
\n\nCharts of the time series below:
\n\n\n
Does anyone know the correct way of interpreting this problem?\nSo I've done a log transform of the target variable as it's highly skewed. This improves the skewness drastically and is much easier to view/interpret (happy days).
\n\nNow I wanted to create a daily time series chart of 3 meter_types (electric, steam etc). In order to do this i need to group_by (date, meter) and take the sum of the meter_reading column. However does it make sense to take the sum of the log transformed data? or should i take the sum of the meter_reading column then log transform the sum?
\n\nCharts of the time series below:
\n\n\n
If you consider mean (sum divided by a constant) which is an estimator of expected value , it's true that:
\n\nDefining F(x) = log (x)\nE(f(x)) != f(E(x)) --> this hold only for linear function
\n\nThat's because of the Law of the unconscious statistician (Lotus). https://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician
\n\nThanks to the law of large numbers you can estimate it by using the mean.
\n\nSo if you want to create some kind of aggregation as a sum or mean on a function i suggest you on applying the function first and aggregate after.
", "rawMarkdown": "If you consider mean (sum divided by a constant) which is an estimator of expected value , it's true that:\n\nDefining F(x) = log (x)\nE(f(x)) != f(E(x)) --> this hold only for linear function\n\nThat's because of the Law of the unconscious statistician (Lotus). [https://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician](LOTUS)\n\nThanks to the law of large numbers you can estimate it by using the mean.\n\nSo if you want to create some kind of aggregation as a sum or mean on a function i suggest you on applying the function first and aggregate after.", "votes": null }, { "id": "669680", "postDate": "11/10/2019 10:17:10", "content": "Hmm, interesting point. Just seems a bit odd that with the charts above using sum(logs) shows that electricity usage is the highest, whereas using the log(sum) shows that steam uses the most.
\n\nthanks for the help
", "rawMarkdown": "Hmm, interesting point. Just seems a bit odd that with the charts above using sum(logs) shows that electricity usage is the highest, whereas using the log(sum) shows that steam uses the most. \n\nthanks for the help", "votes": null } ], "comments": [ { "id": 669644, "author_name": "stenford23", "author_url": "", "post_date": "11/10/2019 09:33:58", "content": "If you consider mean (sum divided by a constant) which is an estimator of expected value , it's true that:
\n\nDefining F(x) = log (x)\nE(f(x)) != f(E(x)) --> this hold only for linear function
\n\nThat's because of the Law of the unconscious statistician (Lotus). https://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician
\n\nThanks to the law of large numbers you can estimate it by using the mean.
\n\nSo if you want to create some kind of aggregation as a sum or mean on a function i suggest you on applying the function first and aggregate after.
", "votes": null, "replies": [ { "id": 669680, "author_name": "bhavek", "author_url": "", "post_date": "11/10/2019 10:17:10", "content": "Hmm, interesting point. Just seems a bit odd that with the charts above using sum(logs) shows that electricity usage is the highest, whereas using the log(sum) shows that steam uses the most.
\n\nthanks for the help
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "669603": "Does anyone know the correct way of interpreting this problem?\nSo I've done a log transform of the target variable as it's highly skewed. This improves the skewness drastically and is much easier to view/interpret (happy days).\n\nNow I wanted to create a daily time series chart of 3 meter_types (electric, steam etc). In order to do this i need to group_by (date, meter) and take the sum of the meter_reading column. However does it make sense to take the sum of the log transformed data? or should i take the sum of the meter_reading column then log transform the sum?\n\nCharts of the time series below:\n\n\n", "669644": "If you consider mean (sum divided by a constant) which is an estimator of expected value , it's true that:\n\nDefining F(x) = log (x)\nE(f(x)) != f(E(x)) --> this hold only for linear function\n\nThat's because of the Law of the unconscious statistician (Lotus). [https://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician](LOTUS)\n\nThanks to the law of large numbers you can estimate it by using the mean.\n\nSo if you want to create some kind of aggregation as a sum or mean on a function i suggest you on applying the function first and aggregate after.", "669680": "Hmm, interesting point. Just seems a bit odd that with the charts above using sum(logs) shows that electricity usage is the highest, whereas using the log(sum) shows that steam uses the most. \n\nthanks for the help" }, "source": "meta" }