The global performance of a forecasting model is evaluated by an error measure. Historically, Mean Absolute Error (MAE) or (Root) Mean Squared Error ((R)MSE) are very popular error measures. However, Mean Square Error is too sensitive to outliers [1] and furthermore,
 both (R)MSE and MAE are scale-dependent measures and hence, it can be hardly used for a comparison across more time series since every single time series has a different impact on the overall results [2]. For example, it has been shown that five of the 1001
 series from the M-competition dominated the RMSE ranking of the forecasting methods and the remaining 996 series had only little impact on the ranking [1].
Symmetric Mean Absolute Percentage Error (SMAPE) [3]:
![](http://www.caos.inf.uc3m.es/%7Ejperalta/ICTSF/images/Evaluation.PNG)
where e
t
= y
t
-F
t
for t = T +1, ..., T +h, it belongs to scale independent error measures, thus can be more easily used to compare methods across different time series. Although the SMAPE was originally proposed in [4] in a different
 form, formula (10) adopts the variant used in [5] since it does not lead to negative values (ranging from 0% to 200%).
For the sake of correct evaluation, we choose
SMAPE
to measure the errors of the competition.
Selected references
J.S. Armstrong, F. Collopy, Error measures for generalizing about forecasting methods: Empirical comparisons, International Journal of Forecasting 8 (1992) 69-80.
J.S. Armstrong, Evaluating methods, in: J.S. Armstrong (Eds.), Principles of Forecasting: A handbook for reasearchers and practitioners, Chap. 14, Kluwer, Boston/Dordrecht/London, 2001, pp. 443-473.
R. J. Hyndman, A. Koehler, Another look at measures of forecast accuracy, International Journal of Forecasting 22(4) (2006) 679-688.
J.S. Armstrong, Long-range forecasting, Wiley New York ETC., 1985.
R. Andrawis, A. Atiya, A new Bayesian formulation for Holt's exponential smoothing, Journal of Forecasting 28(3) (2009) 218-234.