Evolution of MCYFS error over the years
In this section the evolution of the error of MCYFS forecasts over the last seven years is analyzed. The data used are the EU-15 MPE and MAPE calculated as follows: for each combination of crop and year (1996-2002) EU-15 percentage or relative percentage error over all months have been averaged.
Bias
MPE's of forecasts for EU-15 yield for the period 1996-2002 across years and for all crops of interest are the following:
| CROP | 1996 | 1997 | 1998 | 1999 | 2000 | 2001 | 2002 |
|---|---|---|---|---|---|---|---|
| Soft wheat | -8.00 | 3.06 | -3.83 | 1.70 | -0.27 | 4.82 | 1.12 |
| Durum wheat | 5.28 | 8.87 | -10.04 | 3.68 | -0.17 | 10.74 | -0.73 |
| Barley | -6.96 | 1.52 | -1.77 | -1.21 | -3.11 | 2.21 | 1.47 |
| Grain maize | -7.65 | -6.84 | -0.27 | -1.31 | -0.43 | 1.20 | 2.04 |
| Rape seed | -0.68 | -7.14 | -2.17 | -6.81 | 7.99 | 1.70 | 7.11 |
| Potato | -10.66 | -6.23 | 0.55 | -3.97 | -2.63 | -0.34 | -0.43 |
| Sugar beet | -4.09 | -4.37 | -6.08 | -5.64 | -3.52 | 6.04 | -5.34 |
| Sunflower | -1.38 | -6.65 | 4.12 | -5.51 | -8.52 | -1.80 |
Note: The averages are not based on the same number of months for each Crop, Year combination.
In this table, 12 figures are smaller than –6%, 9 are in the interval [-6%, -3%), 15 figures are in [-3%, 0), 9 are in (0, 3], 5 are in (3%, 6%] and 5 are greater than 6%. More than half of the crop, year combinations have high or unacceptably high errors.
The corresponding line chart of EU-15 annual MPE's of forecasts for the yield of several crops (averaging across months):
A negative bias is observed in the first years, followed by a drift of the bias towards zero around 1999 and a subsequent increase towards 2002. Durum wheat is the crop that demonstrates the largest errors on average.
The corresponding boxplots of annual MPE's of forecasts for the yield of several crops (averaging across months):
Mostly positive biases are observed for Durum Wheat and negative biases for Grain maize, Potato, Sugar beet and Sunflower.
Error magnitude
MAPE's across years of forecasts for EU-15 yield for the period 1996-2002 for all crops of interest are displayed in the following table.
| CROP | 1996 | 1997 | 1998 | 1999 | 2000 | 2001 | 2002 | AVERAGE RANK |
|---|---|---|---|---|---|---|---|---|
| Soft wheat | 8.00 | 3.06 | 3.83 | 1.70 | 0.99 | 4.82 | 1.20 | 3.71 |
| Durum wheat | 5.28 | 8.87 | 10.04 | 5.02 | 2.62 | 10.74 | 2.06 | 5.86 |
| Barley | 6.96 | 1.64 | 1.77 | 1.39 | 3.11 | 2.21 | 1.47 | 3.29 |
| Grain maize | 7.65 | 6.84 | 2.28 | 1.31 | 0.43 | 1.44 | 2.04 | 3.29 |
| Rape seed | 2.59 | 7.14 | 2.35 | 6.81 | 7.99 | 1.70 | 7.11 | 5.29 |
| Potato | 10.66 | 6.23 | 0.55 | 3.97 | 2.63 | 1.05 | 1.31 | 3.43 |
| Sugar beet | 4.09 | 4.37 | 6.08 | 5.64 | 3.52 | 6.04 | 5.34 | 5.43 |
| Sunflower | 5.21 | 6.65 | 4.92 | 5.51 | 8.52 | 1.80 | 5.33 | |
| Average rank | 5.00 | 4.75 | 4.00 | 3.38 | 3.50 | 3.88 | 3.00 |
Note: The averages are not based on the same number of months for each Crop, Year combination.
The above table, 23 figures are in the interval [0, 3%], 15 are in (3%, 6%] and 17 are greater than 6%. Most crop-year combinations therefore have high errors.
Page's test has been applied to evaluate whether average error size decreases from year to year. The application of the test gives a marginally insignificant result (no decrease) at the 0.05 level (p-value: 0.073).
A linear regression model has been applied, separately for each crop, of MAPE on years. Years have been codified giving a value of –3 to 1996, -2 to 1997, and so on, arriving at 2 for 2001 and 3 for 2002. The following table reports the results of the regressions for estimation of the relationship between MAPE and year:
| CROP | AVERAGE CHANGE OF MAPE PER YEAR | P-VALUE OF ESTIMATE OF CHANGE | R2 |
|---|---|---|---|
| Soft wheat | -0.70 | 0.144 | 0.375 |
| Durum wheat | -0.48 | 0.525 | 0.085 |
| Barley | -0.5 | 0.208 | 0.294 |
| Grain maize | -1.05 | 0.034 | 0.626 |
| Rape seed | 0.30 | 0.613 | 0.055 |
| Potato | -1.30 | 0.041 | 0.599 |
| Sugar beet | 0.16 | 0.450 | 0.118 |
| Sunflower | -0.31 | 0.616 | 0.069 |
Results are various, depending on the crop, indicating slight reduction or slight increase of MAPE during the last seven years. The strength of the relationship between error size and year as well as its statistical significance also varies. The strongest relationships correspond to the largest average reductions per year and occur in Grain maze (MAPE falls by 1.05 units per year) and in Potato (MAPE falls by 1.3 units per year). Sugar beet and Rape seed demonstrate a slight (statistically insignificant) increase of their MAPE during the last years.
The figure below shows the performance of EU-15 annual MAPE's of forecasts for the yield of several crops (averaging across months), in which there does not seem to be great resemblance in the behaviours of the different crops:
Ranking the crops in the above table with annual MAPE's and comparing with Friedman's test indicates that there is no statistically significant difference in error size between crops (p-value=0.168).
The smaller error of EU-15 forecasts compared to country forecasts is also evident in the following boxplot of annual MAPEs for EU-15, per crop (averaging across months):
