I have posted a question about the chosen metric elsewhere too. The log transformation coupled with squared error term in the evaluation metric is causing models to under-forecast the volumes in the original scale by a good 15%+. Has anyone noticed that? Isn't this a concern for anyone?
\n\nOr is the plan to use the top-performing model on the original scale when it's put in production? But then you wouldn't know if some other models may have performed better in the original scale to begin with.
", "messages": [ { "id": "265762", "postDate": "01/06/2018 16:18:54", "content": "I have posted a question about the chosen metric elsewhere too. The log transformation coupled with squared error term in the evaluation metric is causing models to under-forecast the volumes in the original scale by a good 15%+. Has anyone noticed that? Isn't this a concern for anyone?
\n\nOr is the plan to use the top-performing model on the original scale when it's put in production? But then you wouldn't know if some other models may have performed better in the original scale to begin with.
", "rawMarkdown": "I have posted a question about the chosen metric elsewhere too. The log transformation coupled with squared error term in the evaluation metric is causing models to under-forecast the volumes in the original scale by a good 15%+. Has anyone noticed that? Isn't this a concern for anyone? \n\nOr is the plan to use the top-performing model on the original scale when it's put in production? But then you wouldn't know if some other models may have performed better in the original scale to begin with.", "votes": null }, { "id": "266066", "postDate": "01/07/2018 18:00:09", "content": "Can you elaborate why it is \"causing models to under-forecast the volumes in the original scale by a good 15%+\"?
\n\nI may be missing something, but for me it looks unbiased. (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c\nso the result should be unbiased in the sense that the same error over and under the observed value gives the same result.
\n\nMaybe, your bias comes from the fact that the train set does not include sales == 0, not the metric. That's an astronomic source of bias if you allow the exaggeration.
", "rawMarkdown": "Can you elaborate why it is \"causing models to under-forecast the volumes in the original scale by a good 15%+\"?\n\nI may be missing something, but for me it looks unbiased. (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c\nso the result should be unbiased in the sense that the same error over and under the observed value gives the same result.\n\nMaybe, your bias comes from the fact that the train set does not include sales == 0, not the metric. That's an astronomic source of bias if you allow the exaggeration.", "votes": null }, { "id": "266089", "postDate": "01/07/2018 19:22:09", "content": "The bias isn't on the log scale. It's when you go back to the original scale. In your example, I understand that \" (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c \", but all that's happening with the ln transformed data. When you transform back to the original scale (with exp(x)-1 calculation), you see the problem. Remember that our goal is to forecast volumes in the original scale. The ln-transformation is a means to an end to get a good forecast.
\n\nHere's a simple example to illustrate what's going on.
\n\nLet's first consider volumes on the original scale: \nConsider that the actual (original scale) volume for a day (given store, given item) is 5 units. This is the 'truth'. Let's say this day was held-out, and we are forecasting the volume for this day.
\n\nScenario A: forecasted volume = 3 (our model under-forecasted by 2 units)
\n\nScenario B: forecasted volume = 7 (we over-forecasted by 2 units)
\n\nYou can see here that if we were looking at squared error, we would get 4 in both scenarios.
\n\nNow consider ln-transformed data: \nConvert all three numbers with the ln transformation:
\n\nActual volume = ln(1+5) = 1.79
\n\nScenario A forecast: ln(1+3) = 1.39
\n\nScenario B forecast: ln(1+7) = 2.08
\n\nSquared error for first scenario: (1.79 - 1.39)^2 = 0.16
\n\nSquared error for second scenario: (1.79 - 2.08)^2 = 0.08
\n\nYou can see that we have different squared errors in the ln transformed data even though the deviation was the same in the original scale. We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.
\n\nI assume you have your own model created. In the original scale, do a daily volume plot of the total organization level historical volumes through August 15th, and then your forecasted volumes from Aug 16th through Aug 31st, and then you will see the big drop-off.
\n\nAgain, check the volumes in the original scale, not log scale. Others keep telling me the benefits of ln-transformations but that's all irrelevant if forecasts in the original scale have huge errors.
", "rawMarkdown": "The bias isn't on the log scale. It's when you go back to the original scale. In your example, I understand that \" (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c \", but all that's happening with the ln transformed data. When you transform back to the original scale (with exp(x)-1 calculation), you see the problem. Remember that our goal is to forecast volumes in the original scale. The ln-transformation is a means to an end to get a good forecast. \n\nHere's a simple example to illustrate what's going on. \n\nLet's first consider volumes on the original scale: \nConsider that the actual (original scale) volume for a day (given store, given item) is 5 units. This is the 'truth'. Let's say this day was held-out, and we are forecasting the volume for this day. \n\nScenario A: forecasted volume = 3 (our model under-forecasted by 2 units)\n\nScenario B: forecasted volume = 7 (we over-forecasted by 2 units)\n\nYou can see here that if we were looking at squared error, we would get 4 in both scenarios. \n\nNow consider ln-transformed data: \nConvert all three numbers with the ln transformation: \n\nActual volume = ln(1+5) = 1.79\n\nScenario A forecast: ln(1+3) = 1.39\n\nScenario B forecast: ln(1+7) = 2.08\n\nSquared error for first scenario: (1.79 - 1.39)^2 = 0.16\n\nSquared error for second scenario: (1.79 - 2.08)^2 = 0.08\n\nYou can see that we have different squared errors in the ln transformed data even though the deviation was the same in the original scale. We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias. \n\nI assume you have your own model created. In the original scale, do a daily volume plot of the total organization level historical volumes through August 15th, and then your forecasted volumes from Aug 16th through Aug 31st, and then you will see the big drop-off. \n\nAgain, check the volumes in the original scale, not log scale. Others keep telling me the benefits of ln-transformations but that's all irrelevant if forecasts in the original scale have huge errors.", "votes": null } ], "comments": [ { "id": 266066, "author_name": "batangas", "author_url": "", "post_date": "01/07/2018 18:00:09", "content": "Can you elaborate why it is \"causing models to under-forecast the volumes in the original scale by a good 15%+\"?
\n\nI may be missing something, but for me it looks unbiased. (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c\nso the result should be unbiased in the sense that the same error over and under the observed value gives the same result.
\n\nMaybe, your bias comes from the fact that the train set does not include sales == 0, not the metric. That's an astronomic source of bias if you allow the exaggeration.
", "votes": null, "replies": [ { "id": 266089, "author_name": "abudesai", "author_url": "", "post_date": "01/07/2018 19:22:09", "content": "The bias isn't on the log scale. It's when you go back to the original scale. In your example, I understand that \" (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c \", but all that's happening with the ln transformed data. When you transform back to the original scale (with exp(x)-1 calculation), you see the problem. Remember that our goal is to forecast volumes in the original scale. The ln-transformation is a means to an end to get a good forecast.
\n\nHere's a simple example to illustrate what's going on.
\n\nLet's first consider volumes on the original scale: \nConsider that the actual (original scale) volume for a day (given store, given item) is 5 units. This is the 'truth'. Let's say this day was held-out, and we are forecasting the volume for this day.
\n\nScenario A: forecasted volume = 3 (our model under-forecasted by 2 units)
\n\nScenario B: forecasted volume = 7 (we over-forecasted by 2 units)
\n\nYou can see here that if we were looking at squared error, we would get 4 in both scenarios.
\n\nNow consider ln-transformed data: \nConvert all three numbers with the ln transformation:
\n\nActual volume = ln(1+5) = 1.79
\n\nScenario A forecast: ln(1+3) = 1.39
\n\nScenario B forecast: ln(1+7) = 2.08
\n\nSquared error for first scenario: (1.79 - 1.39)^2 = 0.16
\n\nSquared error for second scenario: (1.79 - 2.08)^2 = 0.08
\n\nYou can see that we have different squared errors in the ln transformed data even though the deviation was the same in the original scale. We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.
\n\nI assume you have your own model created. In the original scale, do a daily volume plot of the total organization level historical volumes through August 15th, and then your forecasted volumes from Aug 16th through Aug 31st, and then you will see the big drop-off.
\n\nAgain, check the volumes in the original scale, not log scale. Others keep telling me the benefits of ln-transformations but that's all irrelevant if forecasts in the original scale have huge errors.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "265762": "I have posted a question about the chosen metric elsewhere too. The log transformation coupled with squared error term in the evaluation metric is causing models to under-forecast the volumes in the original scale by a good 15%+. Has anyone noticed that? Isn't this a concern for anyone? \n\nOr is the plan to use the top-performing model on the original scale when it's put in production? But then you wouldn't know if some other models may have performed better in the original scale to begin with.", "266066": "Can you elaborate why it is \"causing models to under-forecast the volumes in the original scale by a good 15%+\"?\n\nI may be missing something, but for me it looks unbiased. (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c\nso the result should be unbiased in the sense that the same error over and under the observed value gives the same result.\n\nMaybe, your bias comes from the fact that the train set does not include sales == 0, not the metric. That's an astronomic source of bias if you allow the exaggeration.", "266089": "The bias isn't on the log scale. It's when you go back to the original scale. In your example, I understand that \" (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c \", but all that's happening with the ln transformed data. When you transform back to the original scale (with exp(x)-1 calculation), you see the problem. Remember that our goal is to forecast volumes in the original scale. The ln-transformation is a means to an end to get a good forecast. \n\nHere's a simple example to illustrate what's going on. \n\nLet's first consider volumes on the original scale: \nConsider that the actual (original scale) volume for a day (given store, given item) is 5 units. This is the 'truth'. Let's say this day was held-out, and we are forecasting the volume for this day. \n\nScenario A: forecasted volume = 3 (our model under-forecasted by 2 units)\n\nScenario B: forecasted volume = 7 (we over-forecasted by 2 units)\n\nYou can see here that if we were looking at squared error, we would get 4 in both scenarios. \n\nNow consider ln-transformed data: \nConvert all three numbers with the ln transformation: \n\nActual volume = ln(1+5) = 1.79\n\nScenario A forecast: ln(1+3) = 1.39\n\nScenario B forecast: ln(1+7) = 2.08\n\nSquared error for first scenario: (1.79 - 1.39)^2 = 0.16\n\nSquared error for second scenario: (1.79 - 2.08)^2 = 0.08\n\nYou can see that we have different squared errors in the ln transformed data even though the deviation was the same in the original scale. We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias. \n\nI assume you have your own model created. In the original scale, do a daily volume plot of the total organization level historical volumes through August 15th, and then your forecasted volumes from Aug 16th through Aug 31st, and then you will see the big drop-off. \n\nAgain, check the volumes in the original scale, not log scale. Others keep telling me the benefits of ln-transformations but that's all irrelevant if forecasts in the original scale have huge errors." }, "source": "meta" }