Does anyone else think this is not a good metric to evaluate forecast performance? Based on the results, it looks like it favors under-forecasting. I checked the XGB results that figures towards the top of the leaderboard. The attached file is the view of total daily history vs forecast. It is very unlikely that the actual volumes are as low as the forecasted volumes from 8/16 to 8/31.
\n\nAD
", "messages": [ { "id": "265256", "postDate": "01/05/2018 01:09:59", "content": "Does anyone else think this is not a good metric to evaluate forecast performance? Based on the results, it looks like it favors under-forecasting. I checked the XGB results that figures towards the top of the leaderboard. The attached file is the view of total daily history vs forecast. It is very unlikely that the actual volumes are as low as the forecasted volumes from 8/16 to 8/31.
\n\nAD
", "rawMarkdown": "Does anyone else think this is not a good metric to evaluate forecast performance? Based on the results, it looks like it favors under-forecasting. I checked the XGB results that figures towards the top of the leaderboard. The attached file is the view of total daily history vs forecast. It is very unlikely that the actual volumes are as low as the forecasted volumes from 8/16 to 8/31. \n\nAD", "votes": null }, { "id": "265276", "postDate": "01/05/2018 03:04:18", "content": "I think NWRMSLE does not favor under-forecasting. Over- or under-forecasting by the same amount(in log scale) receives the same penalty. Maybe there's something wrong with your model. And try to visualize the data in log scale.
", "rawMarkdown": "I think NWRMSLE does not favor under-forecasting. Over- or under-forecasting by the same amount(in log scale) receives the same penalty. Maybe there's something wrong with your model. And try to visualize the data in log scale.", "votes": null }, { "id": "265619", "postDate": "01/06/2018 04:13:34", "content": "On second thought, maybe you're talking about the effect of log transformation. I find this thread helpful. (somehow hyperlink doesn't work so here's the url: https://stats.stackexchange.com/questions/107610/what-is-the-reason-the-log-transformation-is-used-with-right-skewed-distribution)
", "rawMarkdown": "On second thought, maybe you're talking about the effect of log transformation. I find this thread helpful. (somehow hyperlink doesn't work so here's the url: https://stats.stackexchange.com/questions/107610/what-is-the-reason-the-log-transformation-is-used-with-right-skewed-distribution)", "votes": null }, { "id": "265636", "postDate": "01/06/2018 05:39:26", "content": "Thank you Ceshine. \nI think you understand my question now but it wasn't very well worded in the first place so let me explain in case others are confused by it too.
\n\nSome participants in this competition have been kind to post their model code. I checked the results of a couple of the models that are closer to the top of the leaderboard. I found that while the results look good in the log-scale, when you go back to the original scale, the forecasted volumes look to be under-estimated when compared to the most recent available data (in the 'train' set).
\n\nI acknowledge that this was based on eye-balling the total volumes at the daily level. But still the average organization level forecasted volumes were a solid 15% under the most recent daily volumes seen in the train dataset. It's unlikely that the actual total volumes from Aug 16th to Aug 31st are that low. So I wondered why these models are still scoring so high that they are near the top of the leaderboard. Hence my question.
\n\nIf I were the inventory/supply chain manager for this grocery chain organization, I would never use a model that under-forecasts actual sales by such margins. I wouldn't care what's happening in the log scale.
", "rawMarkdown": "Thank you Ceshine. \nI think you understand my question now but it wasn't very well worded in the first place so let me explain in case others are confused by it too. \n\nSome participants in this competition have been kind to post their model code. I checked the results of a couple of the models that are closer to the top of the leaderboard. I found that while the results look good in the log-scale, when you go back to the original scale, the forecasted volumes look to be under-estimated when compared to the most recent available data (in the 'train' set). \n\nI acknowledge that this was based on eye-balling the total volumes at the daily level. But still the average organization level forecasted volumes were a solid 15% under the most recent daily volumes seen in the train dataset. It's unlikely that the actual total volumes from Aug 16th to Aug 31st are that low. So I wondered why these models are still scoring so high that they are near the top of the leaderboard. Hence my question. \n\nIf I were the inventory/supply chain manager for this grocery chain organization, I would never use a model that under-forecasts actual sales by such margins. I wouldn't care what's happening in the log scale.", "votes": null }, { "id": "265823", "postDate": "01/06/2018 20:00:34", "content": "Same under forecast here. Just confirmed that it is caused by LN.
", "rawMarkdown": "Same under forecast here. Just confirmed that it is caused by LN.", "votes": null }, { "id": "265825", "postDate": "01/06/2018 20:08:48", "content": "Forecast is very accurate after LN conversion. However, the real value is much under forecast. it is good for the competition purpose to continue data analysis after LN. In real business case, it will be very different.
", "rawMarkdown": "Forecast is very accurate after LN conversion. However, the real value is much under forecast. it is good for the competition purpose to continue data analysis after LN. In real business case, it will be very different.", "votes": null }, { "id": "265828", "postDate": "01/06/2018 20:15:44", "content": "Very good point ! Thank you!
", "rawMarkdown": "Very good point ! Thank you!", "votes": null }, { "id": "265906", "postDate": "01/07/2018 03:49:56", "content": "Good to know that I finally understand your question correctly.
\n\nTaking log transformation also reduces the influence of outliers in MSE calculation. Otherwise, since it is a right-skewed distribution, using MSE risks being too optimistic most of the time. FWIK, using MSE without normally distributed errors is generally a very bad idea. For this dataset, I find the errors are reasonably normal with log transformation.
\n\nIt really depends on what the sponsor want. IMHO, if they just want to do inventory management, predicting total unit sales for the next 7 days for regular items and next 2-3 days for perishables (numbers of days are random guesses) would make more sense. (reference: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/data)
\n\nQuote from https://stats.stackexchange.com/questions/18844/when-and-why-should-you-take-the-log-of-a-distribution-of-numbers :
\n\n\"Log-scale informs on relative changes (multiplicative), while linear-scale informs on absolute changes (additive). When do you use each? When you care about relative changes, use the log-scale; when you care about absolute changes, use linear-scale. This is true for distributions, but also for any quantity or changes in quantities.\"
", "rawMarkdown": "Good to know that I finally understand your question correctly.\n\nTaking log transformation also reduces the influence of outliers in MSE calculation. Otherwise, since it is a right-skewed distribution, using MSE risks being too optimistic most of the time. FWIK, using MSE without normally distributed errors is generally a very bad idea. For this dataset, I find the errors are reasonably normal with log transformation. \n\nIt really depends on what the sponsor want. IMHO, if they just want to do inventory management, predicting **total** unit sales for the next 7 days for regular items and next 2-3 days for perishables (numbers of days are random guesses) would make more sense. (reference: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/data)\n\nQuote from https://stats.stackexchange.com/questions/18844/when-and-why-should-you-take-the-log-of-a-distribution-of-numbers :\n\n\"Log-scale informs on relative changes (multiplicative), while linear-scale informs on absolute changes (additive). When do you use each? When you care about relative changes, use the log-scale; when you care about absolute changes, use linear-scale. This is true for distributions, but also for any quantity or changes in quantities.\"", "votes": null }, { "id": "265908", "postDate": "01/07/2018 04:12:47", "content": "I think I made the mistake of not addressing the question correctly again... Are you suggesting that the evaluation should not use log transformation so the predicted volume would match the actual volume?
\n\nI think it'd be interesting to compare predictions with and without log-transformation. On the store/item level, the metric of interest would be how many times the prediction is higher than the actual unit_sales.
\n\nAnother thing I'm not sure you're aware of is that the absolute values of unit_sales between different items are not comparable. So adding them together probably doesn't make sense.
", "rawMarkdown": "I think I made the mistake of not addressing the question correctly again... Are you suggesting that the evaluation should not use log transformation so the predicted volume would match the actual volume?\n\nI think it'd be interesting to compare predictions with and without log-transformation. On the store/item level, the metric of interest would be how many times the prediction is higher than the actual unit_sales.\n\nAnother thing I'm not sure you're aware of is that the absolute values of unit_sales between different items are not comparable. So adding them together probably doesn't make sense.", "votes": null }, { "id": "265913", "postDate": "01/07/2018 04:47:10", "content": "Yes, I am suggesting that the evaluation should not use log transformation if it makes models under-predict the actual sales. What good are these forecasting models if they under-estimate the volumes consistently, and by a significant margin?\nI did build a model on the original scale, and when back-tested on training data (which was held out), results showed no bias. So, yes, if you build a model on the original scale (and do it right), you wouldn't see this kind of bias. \nNot claiming my model is better than others', I am just describing what I have seen happen when comparing two models with and without the log-transformation on this particular problem.
", "rawMarkdown": "Yes, I am suggesting that the evaluation should not use log transformation if it makes models under-predict the actual sales. What good are these forecasting models if they under-estimate the volumes consistently, and by a significant margin?\nI did build a model on the original scale, and when back-tested on training data (which was held out), results showed no bias. So, yes, if you build a model on the original scale (and do it right), you wouldn't see this kind of bias. \nNot claiming my model is better than others', I am just describing what I have seen happen when comparing two models with and without the log-transformation on this particular problem.", "votes": null }, { "id": "265922", "postDate": "01/07/2018 05:25:17", "content": "I'm agree with you. Maybe metric like MSE is good for real business case. But metric like RMSLE and MAPE reduce uncertainty in the time series competition.
", "rawMarkdown": "I'm agree with you. Maybe metric like MSE is good for real business case. But metric like RMSLE and MAPE reduce uncertainty in the time series competition.", "votes": null }, { "id": "266067", "postDate": "01/07/2018 18:05:44", "content": "I paste my answer from here again (sorry that the question is asked twice).
\n\nCan you elaborate why it is \"causing models to under-forecast the volumes in the original scale by a good 15%+\"?
\n\nI may be missing something, but for me it looks unbiased. (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c so the result should be unbiased in the sense that the same error over and under the observed value gives the same result.
\n\nMaybe, your bias comes from the fact that the train set does not include sales == 0, not the metric. That's an astronomic source of bias if you allow the exaggeration.
", "rawMarkdown": "I paste my answer from [here](https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47000) again (sorry that the question is asked twice).\n\nCan you elaborate why it is \"causing models to under-forecast the volumes in the original scale by a good 15%+\"?\n\nI may be missing something, but for me it looks unbiased. (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c so the result should be unbiased in the sense that the same error over and under the observed value gives the same result.\n\nMaybe, your bias comes from the fact that the train set does not include sales == 0, not the metric. That's an astronomic source of bias if you allow the exaggeration.", "votes": null }, { "id": "266090", "postDate": "01/07/2018 19:32:42", "content": "Sorry for creating two threads on the same topic. It's my bad.
\n\nPasting the exact same response I gave to your question in the other thread.
\n\nThe bias isn't on the log scale. It's when you go back to the original scale. In your example, I understand that \" (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c \", but all that's happening with the ln transformed data. When you transform back to the original scale (with exp(x)-1 calculation), you see the problem. Remember that our goal is to forecast volumes in the original scale. The ln-transformation is a means to an end to get a good forecast.
\n\nHere's a simple example to illustrate what's going on.
\n\nLet's first consider volumes on the original scale: Consider that the actual (original scale) volume for a day (given store, given item) is 5 units. This is the 'truth'. Let's say this day was held-out, and we are forecasting the volume for this day.
\n\nScenario A: forecasted volume = 3 (our model under-forecasted by 2 units)
\n\nScenario B: forecasted volume = 7 (we over-forecasted by 2 units)
\n\nYou can see here that if we were looking at squared error, we would get 4 in both scenarios.
\n\nNow consider ln-transformed data: Convert all three numbers with the ln transformation:
\n\nActual volume = ln(1+5) = 1.79
\n\nScenario A forecast: ln(1+3) = 1.39
\n\nScenario B forecast: ln(1+7) = 2.08
\n\nSquared error for first scenario: (1.79 - 1.39)^2 = 0.16
\n\nSquared error for second scenario: (1.79 - 2.08)^2 = 0.08
\n\nYou can see that we have different squared errors in the ln transformed data even though the deviation was the same in the original scale. We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.
\n\nI assume you have your own model created. In the original scale, do a daily volume plot of the total organization level historical volumes through August 15th, and then your forecasted volumes from Aug 16th through Aug 31st, and then you will see the big drop-off.
\n\nAgain, check the volumes in the original scale, not log scale. Others keep telling me the benefits of ln-transformations but that's all irrelevant if forecasts in the original scale have huge errors.
", "rawMarkdown": "Sorry for creating two threads on the same topic. It's my bad. \n\nPasting the exact same response I gave to your question in the other thread. \n\nThe bias isn't on the log scale. It's when you go back to the original scale. In your example, I understand that \" (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c \", but all that's happening with the ln transformed data. When you transform back to the original scale (with exp(x)-1 calculation), you see the problem. Remember that our goal is to forecast volumes in the original scale. The ln-transformation is a means to an end to get a good forecast.\n\nHere's a simple example to illustrate what's going on.\n\nLet's first consider volumes on the original scale: Consider that the actual (original scale) volume for a day (given store, given item) is 5 units. This is the 'truth'. Let's say this day was held-out, and we are forecasting the volume for this day.\n\nScenario A: forecasted volume = 3 (our model under-forecasted by 2 units)\n\nScenario B: forecasted volume = 7 (we over-forecasted by 2 units)\n\nYou can see here that if we were looking at squared error, we would get 4 in both scenarios.\n\nNow consider ln-transformed data: Convert all three numbers with the ln transformation:\n\nActual volume = ln(1+5) = 1.79\n\nScenario A forecast: ln(1+3) = 1.39\n\nScenario B forecast: ln(1+7) = 2.08\n\nSquared error for first scenario: (1.79 - 1.39)^2 = 0.16\n\nSquared error for second scenario: (1.79 - 2.08)^2 = 0.08\n\nYou can see that we have different squared errors in the ln transformed data even though the deviation was the same in the original scale. We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.\n\nI assume you have your own model created. In the original scale, do a daily volume plot of the total organization level historical volumes through August 15th, and then your forecasted volumes from Aug 16th through Aug 31st, and then you will see the big drop-off.\n\nAgain, check the volumes in the original scale, not log scale. Others keep telling me the benefits of ln-transformations but that's all irrelevant if forecasts in the original scale have huge errors.", "votes": null }, { "id": "266837", "postDate": "01/09/2018 23:47:55", "content": "What error metric would you use if you worked in S&OP at this retailer? When you build and test your model on that original data... how do you estimate its accuracy?
\n\nI would love to hear what others have to say on this. Is there an adjustment that can be made after fitting the model to adjust values back to original scale that are reasonable to stores?
", "rawMarkdown": "What error metric would you use if you worked in S&OP at this retailer? When you build and test your model on that original data... how do you estimate its accuracy?\n\nI would love to hear what others have to say on this. Is there an adjustment that can be made after fitting the model to adjust values back to original scale that are reasonable to stores?", "votes": null }, { "id": "267159", "postDate": "01/10/2018 17:53:44", "content": "Based on the overview provided on the main page of this competition, it looks like the goal is inventory management to minimize over-stocking as well as out-of-stock situations. In the real business case, I would use multiple metrics to evaluate different aspects of the forecast. I can easily envision a dashboard to report on various metrics and trends in the data.
\n\nTypically, you do want one overall metric for a quick pulse check. If you want to normalize across scales, then MAPE is a simple but useful metric. Of course you would see higher absolute % errors on items with smaller magnitudes. Also, you might care more about the forecast performance on core products, or more expensive items. Then you might want to weight the error with some item-based attributes such as volumes, perishable vs non-perishable, item price, replenishment lead time, etc. This obviously means you need to have quite a bit of domain knowledge to come up with the best weights that are meaningful for your business.
\n\nThere would be other supplementary metrics. For example, # of out-of-stock occurrences per month across all products or some core products might be useful to know.
\n\nBy the way, choosing the right metric is obviously specific to the goal of the problem. If the goal were something else (for example, workforce planning in stores based on forecasted customer traffic), forecast evaluation metrics could be something different altogether.
\n\nRegarding your second question about adjusting to the original scales, I think most peoples models would work well with a little bit of fine-tuning if they used original volumes instead of ln scale volumes. You would obviously then just do regular rmse (not the ln-based rmse) to tune your algorithm.
", "rawMarkdown": "Based on the overview provided on the main page of this competition, it looks like the goal is inventory management to minimize over-stocking as well as out-of-stock situations. In the real business case, I would use multiple metrics to evaluate different aspects of the forecast. I can easily envision a dashboard to report on various metrics and trends in the data.\n\nTypically, you do want one overall metric for a quick pulse check. If you want to normalize across scales, then MAPE is a simple but useful metric. Of course you would see higher absolute % errors on items with smaller magnitudes. Also, you might care more about the forecast performance on core products, or more expensive items. Then you might want to weight the error with some item-based attributes such as volumes, perishable vs non-perishable, item price, replenishment lead time, etc. This obviously means you need to have quite a bit of domain knowledge to come up with the best weights that are meaningful for your business. \n\nThere would be other supplementary metrics. For example, # of out-of-stock occurrences per month across all products or some core products might be useful to know. \n\nBy the way, choosing the right metric is obviously specific to the goal of the problem. If the goal were something else (for example, workforce planning in stores based on forecasted customer traffic), forecast evaluation metrics could be something different altogether. \n\nRegarding your second question about adjusting to the original scales, I think most peoples models would work well with a little bit of fine-tuning if they used original volumes instead of ln scale volumes. You would obviously then just do regular rmse (not the ln-based rmse) to tune your algorithm.", "votes": null }, { "id": "267385", "postDate": "01/11/2018 08:14:52", "content": "\n\n\nWe got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.
\n
Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. So using log will cause over-forecast in the original scale, right?
", "rawMarkdown": "> We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.\n\nModels are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. So using log will cause over-forecast in the original scale, right?", "votes": null } ], "comments": [ { "id": 265276, "author_name": "ceshine", "author_url": "", "post_date": "01/05/2018 03:04:18", "content": "I think NWRMSLE does not favor under-forecasting. Over- or under-forecasting by the same amount(in log scale) receives the same penalty. Maybe there's something wrong with your model. And try to visualize the data in log scale.
", "votes": null, "replies": [] }, { "id": 265619, "author_name": "ceshine", "author_url": "", "post_date": "01/06/2018 04:13:34", "content": "On second thought, maybe you're talking about the effect of log transformation. I find this thread helpful. (somehow hyperlink doesn't work so here's the url: https://stats.stackexchange.com/questions/107610/what-is-the-reason-the-log-transformation-is-used-with-right-skewed-distribution)
", "votes": null, "replies": [ { "id": 265636, "author_name": "abudesai", "author_url": "", "post_date": "01/06/2018 05:39:26", "content": "Thank you Ceshine. \nI think you understand my question now but it wasn't very well worded in the first place so let me explain in case others are confused by it too.
\n\nSome participants in this competition have been kind to post their model code. I checked the results of a couple of the models that are closer to the top of the leaderboard. I found that while the results look good in the log-scale, when you go back to the original scale, the forecasted volumes look to be under-estimated when compared to the most recent available data (in the 'train' set).
\n\nI acknowledge that this was based on eye-balling the total volumes at the daily level. But still the average organization level forecasted volumes were a solid 15% under the most recent daily volumes seen in the train dataset. It's unlikely that the actual total volumes from Aug 16th to Aug 31st are that low. So I wondered why these models are still scoring so high that they are near the top of the leaderboard. Hence my question.
\n\nIf I were the inventory/supply chain manager for this grocery chain organization, I would never use a model that under-forecasts actual sales by such margins. I wouldn't care what's happening in the log scale.
", "votes": null, "replies": [] }, { "id": 265823, "author_name": "paulz11", "author_url": "", "post_date": "01/06/2018 20:00:34", "content": "Same under forecast here. Just confirmed that it is caused by LN.
", "votes": null, "replies": [] }, { "id": 265828, "author_name": "paulz11", "author_url": "", "post_date": "01/06/2018 20:15:44", "content": "Very good point ! Thank you!
", "votes": null, "replies": [] }, { "id": 265906, "author_name": "ceshine", "author_url": "", "post_date": "01/07/2018 03:49:56", "content": "Good to know that I finally understand your question correctly.
\n\nTaking log transformation also reduces the influence of outliers in MSE calculation. Otherwise, since it is a right-skewed distribution, using MSE risks being too optimistic most of the time. FWIK, using MSE without normally distributed errors is generally a very bad idea. For this dataset, I find the errors are reasonably normal with log transformation.
\n\nIt really depends on what the sponsor want. IMHO, if they just want to do inventory management, predicting total unit sales for the next 7 days for regular items and next 2-3 days for perishables (numbers of days are random guesses) would make more sense. (reference: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/data)
\n\nQuote from https://stats.stackexchange.com/questions/18844/when-and-why-should-you-take-the-log-of-a-distribution-of-numbers :
\n\n\"Log-scale informs on relative changes (multiplicative), while linear-scale informs on absolute changes (additive). When do you use each? When you care about relative changes, use the log-scale; when you care about absolute changes, use linear-scale. This is true for distributions, but also for any quantity or changes in quantities.\"
", "votes": null, "replies": [] }, { "id": 265908, "author_name": "ceshine", "author_url": "", "post_date": "01/07/2018 04:12:47", "content": "I think I made the mistake of not addressing the question correctly again... Are you suggesting that the evaluation should not use log transformation so the predicted volume would match the actual volume?
\n\nI think it'd be interesting to compare predictions with and without log-transformation. On the store/item level, the metric of interest would be how many times the prediction is higher than the actual unit_sales.
\n\nAnother thing I'm not sure you're aware of is that the absolute values of unit_sales between different items are not comparable. So adding them together probably doesn't make sense.
", "votes": null, "replies": [] }, { "id": 265913, "author_name": "abudesai", "author_url": "", "post_date": "01/07/2018 04:47:10", "content": "Yes, I am suggesting that the evaluation should not use log transformation if it makes models under-predict the actual sales. What good are these forecasting models if they under-estimate the volumes consistently, and by a significant margin?\nI did build a model on the original scale, and when back-tested on training data (which was held out), results showed no bias. So, yes, if you build a model on the original scale (and do it right), you wouldn't see this kind of bias. \nNot claiming my model is better than others', I am just describing what I have seen happen when comparing two models with and without the log-transformation on this particular problem.
", "votes": null, "replies": [] } ] }, { "id": 265825, "author_name": "paulz11", "author_url": "", "post_date": "01/06/2018 20:08:48", "content": "Forecast is very accurate after LN conversion. However, the real value is much under forecast. it is good for the competition purpose to continue data analysis after LN. In real business case, it will be very different.
", "votes": null, "replies": [ { "id": 265922, "author_name": "h4211819", "author_url": "", "post_date": "01/07/2018 05:25:17", "content": "I'm agree with you. Maybe metric like MSE is good for real business case. But metric like RMSLE and MAPE reduce uncertainty in the time series competition.
", "votes": null, "replies": [] } ] }, { "id": 266067, "author_name": "batangas", "author_url": "", "post_date": "01/07/2018 18:05:44", "content": "I paste my answer from here again (sorry that the question is asked twice).
\n\nCan you elaborate why it is \"causing models to under-forecast the volumes in the original scale by a good 15%+\"?
\n\nI may be missing something, but for me it looks unbiased. (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c so the result should be unbiased in the sense that the same error over and under the observed value gives the same result.
\n\nMaybe, your bias comes from the fact that the train set does not include sales == 0, not the metric. That's an astronomic source of bias if you allow the exaggeration.
", "votes": null, "replies": [ { "id": 266090, "author_name": "abudesai", "author_url": "", "post_date": "01/07/2018 19:32:42", "content": "Sorry for creating two threads on the same topic. It's my bad.
\n\nPasting the exact same response I gave to your question in the other thread.
\n\nThe bias isn't on the log scale. It's when you go back to the original scale. In your example, I understand that \" (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c \", but all that's happening with the ln transformed data. When you transform back to the original scale (with exp(x)-1 calculation), you see the problem. Remember that our goal is to forecast volumes in the original scale. The ln-transformation is a means to an end to get a good forecast.
\n\nHere's a simple example to illustrate what's going on.
\n\nLet's first consider volumes on the original scale: Consider that the actual (original scale) volume for a day (given store, given item) is 5 units. This is the 'truth'. Let's say this day was held-out, and we are forecasting the volume for this day.
\n\nScenario A: forecasted volume = 3 (our model under-forecasted by 2 units)
\n\nScenario B: forecasted volume = 7 (we over-forecasted by 2 units)
\n\nYou can see here that if we were looking at squared error, we would get 4 in both scenarios.
\n\nNow consider ln-transformed data: Convert all three numbers with the ln transformation:
\n\nActual volume = ln(1+5) = 1.79
\n\nScenario A forecast: ln(1+3) = 1.39
\n\nScenario B forecast: ln(1+7) = 2.08
\n\nSquared error for first scenario: (1.79 - 1.39)^2 = 0.16
\n\nSquared error for second scenario: (1.79 - 2.08)^2 = 0.08
\n\nYou can see that we have different squared errors in the ln transformed data even though the deviation was the same in the original scale. We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.
\n\nI assume you have your own model created. In the original scale, do a daily volume plot of the total organization level historical volumes through August 15th, and then your forecasted volumes from Aug 16th through Aug 31st, and then you will see the big drop-off.
\n\nAgain, check the volumes in the original scale, not log scale. Others keep telling me the benefits of ln-transformations but that's all irrelevant if forecasts in the original scale have huge errors.
", "votes": null, "replies": [] }, { "id": 266837, "author_name": "frankcorrigan", "author_url": "", "post_date": "01/09/2018 23:47:55", "content": "What error metric would you use if you worked in S&OP at this retailer? When you build and test your model on that original data... how do you estimate its accuracy?
\n\nI would love to hear what others have to say on this. Is there an adjustment that can be made after fitting the model to adjust values back to original scale that are reasonable to stores?
", "votes": null, "replies": [] }, { "id": 267159, "author_name": "abudesai", "author_url": "", "post_date": "01/10/2018 17:53:44", "content": "Based on the overview provided on the main page of this competition, it looks like the goal is inventory management to minimize over-stocking as well as out-of-stock situations. In the real business case, I would use multiple metrics to evaluate different aspects of the forecast. I can easily envision a dashboard to report on various metrics and trends in the data.
\n\nTypically, you do want one overall metric for a quick pulse check. If you want to normalize across scales, then MAPE is a simple but useful metric. Of course you would see higher absolute % errors on items with smaller magnitudes. Also, you might care more about the forecast performance on core products, or more expensive items. Then you might want to weight the error with some item-based attributes such as volumes, perishable vs non-perishable, item price, replenishment lead time, etc. This obviously means you need to have quite a bit of domain knowledge to come up with the best weights that are meaningful for your business.
\n\nThere would be other supplementary metrics. For example, # of out-of-stock occurrences per month across all products or some core products might be useful to know.
\n\nBy the way, choosing the right metric is obviously specific to the goal of the problem. If the goal were something else (for example, workforce planning in stores based on forecasted customer traffic), forecast evaluation metrics could be something different altogether.
\n\nRegarding your second question about adjusting to the original scales, I think most peoples models would work well with a little bit of fine-tuning if they used original volumes instead of ln scale volumes. You would obviously then just do regular rmse (not the ln-based rmse) to tune your algorithm.
", "votes": null, "replies": [] }, { "id": 267385, "author_name": "clark78118", "author_url": "", "post_date": "01/11/2018 08:14:52", "content": "\n\n\nWe got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.
\n
Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. So using log will cause over-forecast in the original scale, right?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "265256": "Does anyone else think this is not a good metric to evaluate forecast performance? Based on the results, it looks like it favors under-forecasting. I checked the XGB results that figures towards the top of the leaderboard. The attached file is the view of total daily history vs forecast. It is very unlikely that the actual volumes are as low as the forecasted volumes from 8/16 to 8/31. \n\nAD", "265276": "I think NWRMSLE does not favor under-forecasting. Over- or under-forecasting by the same amount(in log scale) receives the same penalty. Maybe there's something wrong with your model. And try to visualize the data in log scale.", "265619": "On second thought, maybe you're talking about the effect of log transformation. I find this thread helpful. (somehow hyperlink doesn't work so here's the url: https://stats.stackexchange.com/questions/107610/what-is-the-reason-the-log-transformation-is-used-with-right-skewed-distribution)", "265636": "Thank you Ceshine. \nI think you understand my question now but it wasn't very well worded in the first place so let me explain in case others are confused by it too. \n\nSome participants in this competition have been kind to post their model code. I checked the results of a couple of the models that are closer to the top of the leaderboard. I found that while the results look good in the log-scale, when you go back to the original scale, the forecasted volumes look to be under-estimated when compared to the most recent available data (in the 'train' set). \n\nI acknowledge that this was based on eye-balling the total volumes at the daily level. But still the average organization level forecasted volumes were a solid 15% under the most recent daily volumes seen in the train dataset. It's unlikely that the actual total volumes from Aug 16th to Aug 31st are that low. So I wondered why these models are still scoring so high that they are near the top of the leaderboard. Hence my question. \n\nIf I were the inventory/supply chain manager for this grocery chain organization, I would never use a model that under-forecasts actual sales by such margins. I wouldn't care what's happening in the log scale.", "265823": "Same under forecast here. Just confirmed that it is caused by LN.", "265825": "Forecast is very accurate after LN conversion. However, the real value is much under forecast. it is good for the competition purpose to continue data analysis after LN. In real business case, it will be very different.", "265828": "Very good point ! Thank you!", "265906": "Good to know that I finally understand your question correctly.\n\nTaking log transformation also reduces the influence of outliers in MSE calculation. Otherwise, since it is a right-skewed distribution, using MSE risks being too optimistic most of the time. FWIK, using MSE without normally distributed errors is generally a very bad idea. For this dataset, I find the errors are reasonably normal with log transformation. \n\nIt really depends on what the sponsor want. IMHO, if they just want to do inventory management, predicting **total** unit sales for the next 7 days for regular items and next 2-3 days for perishables (numbers of days are random guesses) would make more sense. (reference: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/data)\n\nQuote from https://stats.stackexchange.com/questions/18844/when-and-why-should-you-take-the-log-of-a-distribution-of-numbers :\n\n\"Log-scale informs on relative changes (multiplicative), while linear-scale informs on absolute changes (additive). When do you use each? When you care about relative changes, use the log-scale; when you care about absolute changes, use linear-scale. This is true for distributions, but also for any quantity or changes in quantities.\"", "265908": "I think I made the mistake of not addressing the question correctly again... Are you suggesting that the evaluation should not use log transformation so the predicted volume would match the actual volume?\n\nI think it'd be interesting to compare predictions with and without log-transformation. On the store/item level, the metric of interest would be how many times the prediction is higher than the actual unit_sales.\n\nAnother thing I'm not sure you're aware of is that the absolute values of unit_sales between different items are not comparable. So adding them together probably doesn't make sense.", "265913": "Yes, I am suggesting that the evaluation should not use log transformation if it makes models under-predict the actual sales. What good are these forecasting models if they under-estimate the volumes consistently, and by a significant margin?\nI did build a model on the original scale, and when back-tested on training data (which was held out), results showed no bias. So, yes, if you build a model on the original scale (and do it right), you wouldn't see this kind of bias. \nNot claiming my model is better than others', I am just describing what I have seen happen when comparing two models with and without the log-transformation on this particular problem.", "265922": "I'm agree with you. Maybe metric like MSE is good for real business case. But metric like RMSLE and MAPE reduce uncertainty in the time series competition.", "266067": "I paste my answer from [here](https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47000) again (sorry that the question is asked twice).\n\nCan you elaborate why it is \"causing models to under-forecast the volumes in the original scale by a good 15%+\"?\n\nI may be missing something, but for me it looks unbiased. (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c so the result should be unbiased in the sense that the same error over and under the observed value gives the same result.\n\nMaybe, your bias comes from the fact that the train set does not include sales == 0, not the metric. That's an astronomic source of bias if you allow the exaggeration.", "266090": "Sorry for creating two threads on the same topic. It's my bad. \n\nPasting the exact same response I gave to your question in the other thread. \n\nThe bias isn't on the log scale. It's when you go back to the original scale. In your example, I understand that \" (ln(ŷ i+1)−ln(yi+1))2 is the same when ŷ - y = c and ŷ - y = -c \", but all that's happening with the ln transformed data. When you transform back to the original scale (with exp(x)-1 calculation), you see the problem. Remember that our goal is to forecast volumes in the original scale. The ln-transformation is a means to an end to get a good forecast.\n\nHere's a simple example to illustrate what's going on.\n\nLet's first consider volumes on the original scale: Consider that the actual (original scale) volume for a day (given store, given item) is 5 units. This is the 'truth'. Let's say this day was held-out, and we are forecasting the volume for this day.\n\nScenario A: forecasted volume = 3 (our model under-forecasted by 2 units)\n\nScenario B: forecasted volume = 7 (we over-forecasted by 2 units)\n\nYou can see here that if we were looking at squared error, we would get 4 in both scenarios.\n\nNow consider ln-transformed data: Convert all three numbers with the ln transformation:\n\nActual volume = ln(1+5) = 1.79\n\nScenario A forecast: ln(1+3) = 1.39\n\nScenario B forecast: ln(1+7) = 2.08\n\nSquared error for first scenario: (1.79 - 1.39)^2 = 0.16\n\nSquared error for second scenario: (1.79 - 2.08)^2 = 0.08\n\nYou can see that we have different squared errors in the ln transformed data even though the deviation was the same in the original scale. We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.\n\nI assume you have your own model created. In the original scale, do a daily volume plot of the total organization level historical volumes through August 15th, and then your forecasted volumes from Aug 16th through Aug 31st, and then you will see the big drop-off.\n\nAgain, check the volumes in the original scale, not log scale. Others keep telling me the benefits of ln-transformations but that's all irrelevant if forecasts in the original scale have huge errors.", "266837": "What error metric would you use if you worked in S&OP at this retailer? When you build and test your model on that original data... how do you estimate its accuracy?\n\nI would love to hear what others have to say on this. Is there an adjustment that can be made after fitting the model to adjust values back to original scale that are reasonable to stores?", "267159": "Based on the overview provided on the main page of this competition, it looks like the goal is inventory management to minimize over-stocking as well as out-of-stock situations. In the real business case, I would use multiple metrics to evaluate different aspects of the forecast. I can easily envision a dashboard to report on various metrics and trends in the data.\n\nTypically, you do want one overall metric for a quick pulse check. If you want to normalize across scales, then MAPE is a simple but useful metric. Of course you would see higher absolute % errors on items with smaller magnitudes. Also, you might care more about the forecast performance on core products, or more expensive items. Then you might want to weight the error with some item-based attributes such as volumes, perishable vs non-perishable, item price, replenishment lead time, etc. This obviously means you need to have quite a bit of domain knowledge to come up with the best weights that are meaningful for your business. \n\nThere would be other supplementary metrics. For example, # of out-of-stock occurrences per month across all products or some core products might be useful to know. \n\nBy the way, choosing the right metric is obviously specific to the goal of the problem. If the goal were something else (for example, workforce planning in stores based on forecasted customer traffic), forecast evaluation metrics could be something different altogether. \n\nRegarding your second question about adjusting to the original scales, I think most peoples models would work well with a little bit of fine-tuning if they used original volumes instead of ln scale volumes. You would obviously then just do regular rmse (not the ln-based rmse) to tune your algorithm.", "267385": "> We got a bigger error on the under-forecast side than on the over-forecast side. This is the cause of the problem. Models are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. This pulls the predictions to the low side, creating that bias.\n\nModels are inherently being keyed to prevent the under-forecast scenarios in the original scale to avoid the bigger errors in the ln-scale. So using log will cause over-forecast in the original scale, right?" }, "source": "meta" }