Evolution of MCYFS error over the years

From Agri4castWiki
Jump to: navigation, search


In this section the evolution of the error of MCYFS forecasts over the last seven years is analyzed. The data used are the EU-15 MPE and MAPE calculated as follows: for each combination of crop and year (1996-2002) EU-15 percentage or relative percentage error over all months have been averaged.

Bias

MPE's of forecasts for EU-15 yield for the period 1996-2002 across years and for all crops of interest are the following:

CROP 1996 1997 1998 1999 2000 2001 2002
Soft wheat -8.00 3.06 -3.83 1.70 -0.27 4.82 1.12
Durum wheat 5.28 8.87 -10.04 3.68 -0.17 10.74 -0.73
Barley -6.96 1.52 -1.77 -1.21 -3.11 2.21 1.47
Grain maize -7.65 -6.84 -0.27 -1.31 -0.43 1.20 2.04
Rape seed -0.68 -7.14 -2.17 -6.81 7.99 1.70 7.11
Potato -10.66 -6.23 0.55 -3.97 -2.63 -0.34 -0.43
Sugar beet -4.09 -4.37 -6.08 -5.64 -3.52 6.04 -5.34
Sunflower -1.38 -6.65 4.12 -5.51 -8.52 -1.80

Note: The averages are not based on the same number of months for each Crop, Year combination.

In this table, 12 figures are smaller than –6%, 9 are in the interval [-6%, -3%), 15 figures are in [-3%, 0), 9 are in (0, 3], 5 are in (3%, 6%] and 5 are greater than 6%. More than half of the crop, year combinations have high or unacceptably high errors.

The corresponding line chart of EU-15 annual MPE's of forecasts for the yield of several crops (averaging across months):

Linechart of country MPE's per crop

A negative bias is observed in the first years, followed by a drift of the bias towards zero around 1999 and a subsequent increase towards 2002. Durum wheat is the crop that demonstrates the largest errors on average.

The corresponding boxplots of annual MPE's of forecasts for the yield of several crops (averaging across months):

Linechart of country MPE's per crop

Mostly positive biases are observed for Durum Wheat and negative biases for Grain maize, Potato, Sugar beet and Sunflower.

Error magnitude

MAPE's across years of forecasts for EU-15 yield for the period 1996-2002 for all crops of interest are displayed in the following table.

CROP 1996 1997 1998 1999 2000 2001 2002 AVERAGE RANK
Soft wheat 8.00 3.06 3.83 1.70 0.99 4.82 1.20 3.71
Durum wheat 5.28 8.87 10.04 5.02 2.62 10.74 2.06 5.86
Barley 6.96 1.64 1.77 1.39 3.11 2.21 1.47 3.29
Grain maize 7.65 6.84 2.28 1.31 0.43 1.44 2.04 3.29
Rape seed 2.59 7.14 2.35 6.81 7.99 1.70 7.11 5.29
Potato 10.66 6.23 0.55 3.97 2.63 1.05 1.31 3.43
Sugar beet 4.09 4.37 6.08 5.64 3.52 6.04 5.34 5.43
Sunflower 5.21 6.65 4.92 5.51 8.52 1.80 5.33
Average rank 5.00 4.75 4.00 3.38 3.50 3.88 3.00

Note: The averages are not based on the same number of months for each Crop, Year combination.

The above table, 23 figures are in the interval [0, 3%], 15 are in (3%, 6%] and 17 are greater than 6%. Most crop-year combinations therefore have high errors.

Page's test has been applied to evaluate whether average error size decreases from year to year. The application of the test gives a marginally insignificant result (no decrease) at the 0.05 level (p-value: 0.073).

A linear regression model has been applied, separately for each crop, of MAPE on years. Years have been codified giving a value of –3 to 1996, -2 to 1997, and so on, arriving at 2 for 2001 and 3 for 2002. The following table reports the results of the regressions for estimation of the relationship between MAPE and year:

CROP AVERAGE CHANGE OF MAPE PER YEAR P-VALUE OF ESTIMATE OF CHANGE R2
Soft wheat -0.70 0.144 0.375
Durum wheat -0.48 0.525 0.085
Barley -0.5 0.208 0.294
Grain maize -1.05 0.034 0.626
Rape seed 0.30 0.613 0.055
Potato -1.30 0.041 0.599
Sugar beet 0.16 0.450 0.118
Sunflower -0.31 0.616 0.069

Results are various, depending on the crop, indicating slight reduction or slight increase of MAPE during the last seven years. The strength of the relationship between error size and year as well as its statistical significance also varies. The strongest relationships correspond to the largest average reductions per year and occur in Grain maze (MAPE falls by 1.05 units per year) and in Potato (MAPE falls by 1.3 units per year). Sugar beet and Rape seed demonstrate a slight (statistically insignificant) increase of their MAPE during the last years.

The figure below shows the performance of EU-15 annual MAPE's of forecasts for the yield of several crops (averaging across months), in which there does not seem to be great resemblance in the behaviours of the different crops:

Linechart of country MAPE's per crop

Ranking the crops in the above table with annual MAPE's and comparing with Friedman's test indicates that there is no statistically significant difference in error size between crops (p-value=0.168).

The smaller error of EU-15 forecasts compared to country forecasts is also evident in the following boxplot of annual MAPEs for EU-15, per crop (averaging across months):

Linechart of country MAPE's per crop