Methodology of forecast errors evaluation methods

From Agri4castWiki
Jump to: navigation, search


Error indicators

The forecasting literature favours three different types of error indicators for use in evaluating forecast performance or for comparing forecasting methods: RMSFE, MAPE and Theil's U coefficient.

The reason behind the preference for RMSFE seems to be its resemblance to mean square error which is a discrepancy measure commonly used in statistics and its correspondence to a quadratic loss function; see for example Deschamps and Mehta (1980). It has however received a lot of criticism because it is affected by the magnitude of the forecasted series (and hence does not allow aggregation over multiple series) and for its lack of reliability. Armstrong and Collopy (1992) found that the rankings of several forecasting methods changed significantly depending on the sample of time series used. Similar comments are made by Makridakis and Hibon (1979). The effect of the series' magnitude on the indicator is eliminated if instead of RMSFE we use RMSPE.

MAPE is the indicator of choice in many studies of forecasting accuracy (Makridakis and Hibon, 1979, Deschamps and Mehta, 1980, Karamouzis, 1985). It does not depend on the series' magnitude or unit of measurement, can be averaged across series and can be used for comparing methods. We manly have used MAPE as an indicator of error size, because, as we already mentioned it is an average absolute error while RMSPE is a proxy to an average absolute error. We have however used RMSPE for specific comparisons.

Besides MAPE we have used MPE which also does not depend on a series magnitude or unit of measurement. It complements MAPE by giving the direction and size of forecasting bias.

Finally, since Theil's U is so widespread we have calculated a similar indicator, and more specifically MRE, where the naïve method uses k=1. We chose MRE because firstly we have a personal preference for indicators measuring absolute instead of squared error and we chose k=1because we wanted to compare the forecasting methods with a slightly more complicated naïve method than the random walk.

In the following section the forecast error indicators chosen are presented and analyzed with particular regards to their statistical properties.

Percentage error

It is the difference between the forecast and the true value of the variable of interest expressed as a proportion of the true value. If Yt is the true value of the variable at time point t and Ýt is a forecast for it, the percentage error (PE) at time point t is given by the formula

Percentage error

The indicator may assume any real number as its value. The further from zero its value is the larger the forecast error. It can be multiplied it by 100, and expressed as a percentage (%). The indicator shows the relative magnitude and the direction of bias in the same way as FE. A disadvantage of the indicator is its lack of symmetry; for example dt=1 means that the forecast is twice as large as the true value but dt=-0.5 means that the true value is twice as large as the forecast. This asymmetry also means that overestimation is penalized more (“looks worse”) than underestimation. The aggregation of percentage error over a given period of time consisting of T points gives the mean percentage error (MPE),

Mean percentage error

and the root mean square percentage error (RMSPE),

Square percentage error

MPE may take any real value while RMSPE takes only positive values. The first aggregate indicates the direction and relative size of the bias of the forecasts while RMSPE gives an average relative size of forecast error over the given period.

Unusually large errors affect MPE and RMSPE. On the other hand, due to the existence of the true values in the denominator of the indicator, the aggregates are not affected by the magnitude or the unit of measurement of the series being forecasted. Alternating large positive and negative errors affect them. This renders them suitable as means for comparing the performance of a forecasting method on several series or the performance of several methods on the same series.

Absolute Percentage error

It is the absolute value of the percentage error. Using the same notation as before, the absolute percentage error (APE) at time point t is given by the formula

Absolute percentage error

The indicator may assume any positive real number as its value. The further from zero its value is the larger the forecast error. It can be multiplied it by 100, and expressed as a percentage (%). A disadvantage of the indicator is its lack of symmetry just like PE.

Aggregating absolute percentage error over a given period of time consisting of T points gives the mean absolute percentage error (MAPE):

Mean absolute percentage error

The aggregate has the same advantages and disadvantages as those of PE and is suitable for the comparison of the performance of a forecasting method on several series or of the performance of several methods on the same series. The fact that it considers absolute error makes it more suitable than PE-based aggregates for evaluating the relative size of error.

Statistical comparisons of crop yield forecasting systems

A statistical comparison has also been carried out where possible; more specifically, Wilcoxon, Friedman and Page tests have been used (Conover, 1998).

Wilcoxon tests have been used to examine the statistical significance of MCYFS' forecasting bias for each crop and at each country. The tests examined whether an observed mean percentage error above or below zero indicated bias or could be due to random fluctuations.

Friedman tests are used in order to compare the forecast error size of MCYFS between countries and also between crops. They indicated countries and crops in which MCYFS has a significantly different performance (in terms of forecast error).

Finally, Page tests have examined whether the average error size of MCYFS reduces as the year advances and also whether it reduces from year to year.