Spatial analysis of MCYFS error

From Agri4castWiki
Jump to: navigation, search


In this section the spatial features of the error of MCYFS forecasts are examined. The data used are the MPE and MAPE calculated as follows: for each combination of crop, country the percentage or absolute percentage error have been averaged over all months and years.

Bias

The table of MCYFS MPE for the period 1993-2002 across EU member states and for all crops of interest is presented below:

COUNTRY SOFT WHEAT DURUM WHEAT BARLEY GRAIN MAIZE RAPE SEED POTATO SUGAR BEET SUNFLOWER
AT -0.08 11.03 3.52 -3.56 10.34 -5.57 -3.96 -3.60
BE -1.29 -3.79 -6.75 -6.99 -4.07
DE -1.43 -0.04 -0.91 -2.96 -2.90 -2.07 -3.64 5.07
DK 0.20 -1.55 1.40 -2.65 -1.80
ES -3.05 17.94 -1.03 -4.03 0.45 -9.75 -8.90 7.41
FI 10.86 6.82 18.38 -1.79 -0.04
FR -0.34 4.12 -0.17 -2.36 0.23 -4.47 -0.86 3.30
GR 3.03 4.38 5.65 -0.11 -2.71 -2.66 2.07
IE -2.74 0.29 -3.94 1.96
IT 0.77 1.31 1.59 -1.06 36.54 -0.25 0.57 5.31
LU 0.43
NL 3.70 2.43 -12.10 -0.57 1.14
PT -5.29 -0.71 -5.16 -10.94 1.98 23.29
SE 0.93 -1.72 -3.81 2.53 -2.40
UK 0.16 0.63 0.64 -1.33 -3.15

The marked cells denotes combinations for which the Wilcoxon test indicates significant bias at the 0.05 level (only cases potato and sugar beet, ES), or significance at the 0.15 level (only case Maize, PT) and significance at the 0.5 level (Soft wheat IE); all significance levels have been Bonferonni-adjusted (Note: The averages are not based on the same number of years and months for each Crop, Country combinations.)

In the above table 6 figures are smaller than –6%, 15 are in the interval [-6%, -3%), 29 figures are in [-3%, 0), 20 are in (0, 3], 9 are in (3%, 6%] and 9 are greater than 6%. Therefore a large number of crop - country combinations have errors beyond the 3% limit. Moreover, there is a preponderance of negative biases.

A Wilcoxon test has been performed for each crop, country combination to check the existence of systematic bias, giving a total of 88 tests. The data used for each test were the annual MPEs (e.g. averages of each year's monthly percentage errors) of the respective crop, country combination.

To avoid the effects of multiplicity, the nominal 0.05 level of confidence was adjusted with the Bonferonni adjustment. 6.3 indicates with different colours those combinations which exhibit significant bias at the (Bonferonni adjusted) 0.05, 0.15 and 0.5 level.

Of the 88 tests only 2 found significant biases at the 0.05 level. They refer to Potato and Sugar beet in Spain and they correspond to systematic underestimation. Another feature worth noting is that Grain maize was underestimated on average in all countries. Potato was underestimated in 12 out of the 14 countries, although only in Spain underestimation was systematic. Durum wheat shows a tendency for overestimation, negative biases being small; no bias is significant though. Finally Sunflower yield was overestimated in all countries except Austria , which again, might be explained by the fact that yields in Austria are consistently well above the EU average.

Country wise it is worth noting that the yield in Belgium was underestimated for all crops while in Germany it was underestimated for 7 out of 8 crops. On the contrary, yields in Italy appear to be overestimated on average in 6 out of 8 crops.

Below the boxplots of each crop's MPEs across the fifteen countries and the boxplots of each country's MPEs across the eight crops are presented. These graphs allow at a glance the inspection of the overall bias in each crop and in each country.

Boxplots of country MPE's per crop

The most prominent features in figure are the underestimation of the yield of Grain maize, Potato and Sugar beet and the overestimation of the yield of Sunflower, Durum wheat and Rape seed.

Boxplots of crop MPE's per country

In the above figure the negative biases in Belgium , Germany and Denmark and the positive biases in Finland and Italy can be observed.

From both graphs it is evident that positive biases (overestimations) are relatively higher than negative ones. This is to a certain extent due to the indicator used (the percentage error), which takes larger values for positive than negative differences of the same absolute value. For this reason, the attention to the graphs has not to be so much focused on the distance of the extremes from the zero line but rather on the proportion of each “box” above or below the line.

In order to give a clearer picture of the MCYFS error concerning each crop, Annex 8.4 reports and comments maps of the country MPEs and of the interval the MPEs belong to.

Error magnitude

The table of MCYFS MAPE for the period 1993-2002 across EU member states and for all crops of interest is the following:

COUNTRY SOFT WHEAT DURUM WHEAT BARLEY GRAIN MAIZE RAPE SEED POTATO SUGAR BEET SUNFLOWER AVERAGE RANK
AT 6.59 19.88 9.21 6.93 14.71 7.05 9.46 5.43 6.50
BE 5.61 6.60 11.56 8.66 7.75 7.00
DE 4.42 6.94 5.27 5.83 10.66 7.21 6.86 12.57 4.00
DK 3.87 4.16 12.66 6.83 6.13 3.20
ES 13.20 27.98 15.29 7.84 16.84 9.75 9.05 19.62 8.63
FI 17.18 13.00 22.48 9.98 17.54 11.40
FR 4.76 8.50 4.90 3.61 8.70 5.85 5.44 5.20 2.25
GR 11.05 12.83 14.26 6.08 9.34 6.64 21.35 7.43
IE 7.12 10.12 12.23 15.59 11.75
IT 4.89 6.58 4.96 3.20 50.25 3.25 7.15 8.10 3.75
LU 7.48 9.00
NL 6.94 7.46 20.69 3.74 5.61 6.20
PT 27.01 32.51 27.12 13.03 11.30 35.51 10.67
SE 3.62 7.36 9.90 10.34 5.16 4.60
UK 5.85 5.44 13.40 6.72 9.83 6.40
Average rank 2.21 6.14 3.21 2.44 6.11 3.29 3.62 5.71

Note: The averages are not based on the same number of years and months for each Crop, Country combination.

In this table no figure is in the interval [0, 3%], 23 are in (3%, 6%] and 65 are greater than 6%. All crop, country combinations have therefore high errors. This result is more worrisome than the result reported in the previous because it indicates that small MPEs may be the result of high positive and negative errors. In MAPE figures such averaging does not take place and the quality of forecasts is more accurately reflected in them.

Annex 8.5 reports and comments maps (separately for each crop) which show into which interval belongs the MAPE of each country.

The table also presents one column and one row of average ranks. These are calculated as follows:

  • firstly, the MAPEs within each country have been ranked with rank 1 assigned to the crop with the smallest MAPE. When ranking has been performed in all countries the ranks each crop has received are summed and their sum is divided by the number of countries for which forecasts are available. This indicates how large is the error of each crop compared to the other crops.
  • the column of average ranks has been calculated in a similar way, by ranking countries separately within each crop. The observed averages show, for example, that on average, largest errors occur in Rape seed, while among countries the largest errors on average are observed in Portugal, Ireland and Finland.

The boxplots of each crop’s MAPEs across the fifteen countries and the boxplots of each country’s MAPEs across the eight crops are the following:

Boxplots of country MAPE's per crop


In the above figure it can be observed that besides Rape seed large errors are observed in Durum wheat and Sunflower.

The countries with the largest errors are Portugal, Spain and Finland:

Boxplots of crop MAPE's per country