Evolution of MCYFS error over the months

In this section the temporal evolution of the error of MCYFS forecasts is analyzed. The data used are the MPE and MAPE for EU-15 which have been calculated as average for each combination of crop and month of the EU-15 percentage or relative percentage error over the period 1996-2002.

Bias

MPE's of forecasts for EU-15 yield for the period 1996-2002 across months and for all crops of interest are given in the table below:

CROP	MARCH	APRIL	MAY	JUNE	JULY	AUGUST	SEPT	OCT	NOV
Soft wheat	0.75	-0.13	0.01	-0.24	-2.27	-0.69	-1.40	-0.28	-0.13
Durum wheat	3.76	6.46	2.69	1.50	2.44	0.80	0.61	0.60	1.99
Barley	-0.61	-1.25	-1.27	-1.90	-2.08	-0.95	-1.15	-0.88	0.74
Grain maize		-3.97	-2.03	-1.71	-4.97	-1.90	-3.96	-2.44	2.77
Rape seed	2.27	2.08	0.09	-0.92	-1.86	-0.88	-3.33	-1.93	5.46
Potato		-9.19	-2.92	-4.38	-6.03	-3.56	-4.06	-2.41	-0.81
Sugar beet		-6.19	-1.92	-4.22	-5.51	-4.73	-1.85	-1.61	-2.96
Sunflower		-3.40	-3.79	-4.72	-8.31	-2.72	1.84	-2.72

The bold printed cell denotes combinations for which the Wilcoxon test indicates significant bias at the 0.05 level, all significance levels have been Bonferonni-adjusted. (Note: The averages are not based on the same number of years for each Crop, Month combination)

In the above table figures are smaller than –6%, 13 are in the interval [-6%, -3%), 32 figures are in [-3%, 0), 15 are in (0, 3], 2 are in (3%, 6%] and 1 is greater than 6%. Almost one in three crop-month combinations have high errors. The preponderance of negative biases already observed in national figures is evident here too.

The Wilcoxon test has been performed, for each crop and month combination, to check the existence of systematic bias, giving a total of 67 tests. The data used for each test are the MPEs (e.g. averages of each month's percentage errors across the seven years) of the respective crop, month combination for EU-15.

To avoid the effects of multiplicity, the nominal 0.05 level of confidence was adjusted with the Bonferonni correction. Table 4 indicates with different colours the combinations which exhibits bias significant at the (Bonferonni adjusted) 0.05.

Of the 67 tests only 1 found a significant bias at the 0.05 level. It corresponds to a systematic underestimation of the yield of Grain maize in October. It is worth noting that Potato and Sugar beet were underestimated on average in all months. The yield of Barley, Grain maize and Sunflower were underestimated in all but one months (November, November and September respectively). On the contrary, Durum Wheat yield was overestimated on average in all months.

Line chart of the behaviour of EU-15 MPE's of forecasts for the yield of several crops across months (averaging across years):

The figure shows that bias decreases on average for all crops as the year advances. November however shows a deterioration in Rape seed and slightly less so in Grain maize and Durum wheat. In other words MCYFS predictions are, as expected, more reliable in the last than in the first months of the year. Other important features of the figure are the mostly negative bias of the forecasts for all crops except Durum wheat (positive bias) and the fact that the error for each crop follows roughly the same pattern along the year. Moreover, it can be observed that July stands out from the rest of the months, having a larger bias.

Error magnitude

MAPE's of forecasts for EU-15 yield for the period 1996-2002 across months and for all crops of interest are given in the table below:

CROP	MARCH	APRIL	MAY	JUNE	JULY	AUGUST	SEPT	OCT	NOV
Soft wheat	3.55	3.71	4.16	3.91	5.07	2.68	3.29	2.65	0.13
Durum wheat	14.91	10.20	6.44	5.29	3.84	3.59	5.67	3.86	1.99
Barley	3.15	3.47	2.81	2.74	3.80	2.43	2.38	1.62	0.74
Grain maize		5.90	3.73	3.09	5.36	3.18	3.96	2.44	2.77
Rape seed	5.68	6.68	5.82	4.93	5.50	3.93	3.33	3.67	5.46
Potato		9.19	3.11	4.75	6.77	4.10	4.38	2.68	0.81
Sugar beet		6.19	4.61	6.21	5.51	6.71	1.85	3.12	2.96
Sunflower		3.40	5.86	8.13	8.31	4.10	4.10	4.31
Average rank	7.00	6.75	5.75	5.38	6.63	3.75	3.50	2.50	1.86

Note: The averages are not based on the same number of years for each Crop, Month combination.

In the above table 16 figures are in the interval [0, 3%], 40 are in (3%, 6%] and 11 are greater than 6%. Most crop-month combinations have therefore errors above 3%, but the picture is better than that of country figures in section 2.4.2. This slight improvement is explained by the fact that EU-15 forecasts are weighted averages of country forecasts, and consequently their error is an average of the country errors.

Average ranks for months show a clear reduction of error size as the year advances (with the exception of July). The evident reduction however is not enough to give a significant result in statistical testing according to Page's test (which tests whether error size decreases on average from month to month): the application of the test gives a non-significant result at the 0.05 level (p-value: 0.56). However, since we cannot consider months as independent one from the other, this result must be viewed as a lack of strong indications that the error size of MCYFS forecasts decreases as the year advances, at least at the EU-15 level. A diagrammatic representation of The above table and further evidence for the decrease of error size is given by the following line chart displaying MAPE's of forecasts for the EU-15 yield of several crops (averaging across years):

The same pattern already observed in the previous figure - the deterioration of forecasts in July - can be detected in this figure as well. In fact if July conformed to the pattern of the other months, the result of Page's test would be even more favourable to the hypothesis that MCYFS improve as the year advances.

In an effort to quantify the decrease of MAPE as the year advances a linear regression model has been applied, separately for each crop, of MAPE on month. Months have been codified giving a value of –4 to March, -3 to April, and so on, arriving at 3 for October and 4 for November.

In the following table the regression results for estimation of the relationship between MAPE and month for each crop are reported: R2, the slope of each regression line and the statistical significance of the slope's estimate. The slope represents the average change of MAPE per month.

CROP	AVERAGE CHANGE OF MAPE PER MONTH	P-VALUE OF ESTIMATE OF CHANGE	R2
Soft wheat	-0.33	0.056	0.428
Durum wheat	-1.23	0.004	0.709
Barley	-0.27	0.010	0.633
Grain maize	-0.33	0.075	0.435
Rape seed	-0.27	0.061	0.416
Potato	-0.77	0.039	0.537
Sugar beet	-0.5	0.065	0.459
Sunflower	-0.17	0.694	0.034

For all crops the regressions indicate a reduction in MAPE from month to month. The strength of the relationship between MAPE and month varies across crops, as does the statistical significance, from very strong as in Durum wheat, Barley and Potato to very weak as in Sunflower. Average reduction of MAPE per month is less than 1 MAPE unit (1%) for all crops except Durum wheat whose MAPE decreases by 1.23 units per month. Differences in average reduction must however be considered in relation to the starting value of each crop's MAPE. Durum wheat shows the largest reduction because it starts in March with a MAPE which is almost the triple of the other crops' MAPE and therefore has greater scope for reduction.