Evolution of MCYFS error over the months

From Agri4castWiki
Revision as of 15:31, 14 June 2012 by Raymond (talk | contribs) (Created page with '__NOTOC__ {{Scientific}} In this section the temporal evolution of the error of MCYFS forecasts is analyzed. The data used are the MPE and MAPE for EU-15 which have been calculat…')
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


In this section the temporal evolution of the error of MCYFS forecasts is analyzed. The data used are the MPE and MAPE for EU-15 which have been calculated as average for each combination of crop and month of the EU-15 percentage or relative percentage error over the period 1996-2002.

Bias

The table reporting MCYFS MPE of MCYFS forecasts for EU-15 yield for the period 1996-2002 across months and for all crops of interest is the following:


CROP MARCH APRIL MAY JUNE JULY AUGUST SEPT OCT NOV Soft wheat 0.75 -0.13 0.01 -0.24 -2.27 -0.69 -1.40 -0.28 -0.13 Durum wheat 3.76 6.46 2.69 1.50 2.44 0.80 0.61 0.60 1.99 Barley -0.61 -1.25 -1.27 -1.90 -2.08 -0.95 -1.15 -0.88 0.74 Grain maize -3.97 -2.03 -1.71 -4.97 -1.90 -3.96 -2.44 2.77 Rape seed 2.27 2.08 0.09 -0.92 -1.86 -0.88 -3.33 -1.93 5.46 Potato -9.19 -2.92 -4.38 -6.03 -3.56 -4.06 -2.41 -0.81 Sugar beet -6.19 -1.92 -4.22 -5.51 -4.73 -1.85 -1.61 -2.96 Sunflower -3.40 -3.79 -4.72 -8.31 -2.72 1.84 -2.72


The marked cell denotes combinations for which the Wilcoxon test indicates significant bias at the 0.05 level, all significance levels have been Bonferonni-adjusted. (Note: The averages are not based on the same number of years for each Crop, Month combination)

In the above table figures are smaller than –6%, 13 are in the interval [-6%, -3%), 32 figures are in [-3%, 0), 15 are in (0, 3], 2 are in (3%, 6%] and 1 is greater than 6%. Almost one in three crop-month combinations have high errors. The preponderance of negative biases already observed in national figures is evident here too. The Wilcoxon test has been performed, for each crop and month combination, to check the existence of systematic bias, giving a total of 67 tests. The data used for each test are the MPEs (e.g. averages of each month's percentage errors across the seven years) of the respective crop, month combination for EU-15. To avoid the effects of multiplicity, the nominal 0.05 level of confidence was adjusted with the Bonferonni correction. Table 4 indicates with different colours the combinations which exhibits bias significant at the (Bonferonni adjusted) 0.05. Of the 67 tests only 1 found a significant bias at the 0.05 level. It corresponds to a systematic underestimation of the yield of Grain maize in October. It is worth noting that Potato and Sugar beet were underestimated on average in all months. The yield of Barley, Grain maize and Sunflower were underestimated in all but one months (November, November and September respectively). On the contrary, Durum Wheat yield was overestimated on average in all months. The line chart of the behaviour of MPE (separately for each crop) across months is the following:

Figure 6.5: EU-15 MPEs of MCYFS forecasts for the yield of several crops (averaging across years).

Figure 6.5 shows that bias decreases on average for all crops as the year advances. November however shows a deterioration in Rape seed and slightly less so in Grain maize and Durum wheat. In other words MCYFS predictions are, as expected, more reliable in the last than in the first months of the year. Other important features of Figure 6.5 are the mostly negative bias of the forecasts for all crops except Durum wheat (positive bias) and the fact that the error for each crop follows roughly the same pattern along the year. Moreover, it can be observed that July stands out from the rest of the months, having a larger bias.

==Error magnitude The table reporting MCYFS MAPE across months and for all crops of interest is the following: Table 6.5: MAPE of MCYFS forecasts for EU-15 yield for the period 1996-2002. CROP MAR APR MAY JUNE JULY AUG SEPT OCT NOV Soft wheat 3.55 3.71 4.16 3.91 5.07 2.68 3.29 2.65 0.13 Durum wheat 14.91 10.20 6.44 5.29 3.84 3.59 5.67 3.86 1.99 Barley 3.15 3.47 2.81 2.74 3.80 2.43 2.38 1.62 0.74 Grain maize 5.90 3.73 3.09 5.36 3.18 3.96 2.44 2.77 Rape seed 5.68 6.68 5.82 4.93 5.50 3.93 3.33 3.67 5.46 Potato 9.19 3.11 4.75 6.77 4.10 4.38 2.68 0.81 Sugar beet 6.19 4.61 6.21 5.51 6.71 1.85 3.12 2.96 Sunflower 3.40 5.86 8.13 8.31 4.10 4.10 4.31 Average rank 7.00 6.75 5.75 5.38 6.63 3.75 3.50 2.50 1.86 Note: The averages are not based on the same number of years for each Crop, Month combination.

In Table 6.5, 16 figures are in the interval [0, 3%], 40 are in (3%, 6%] and 11 are greater than 6%. Most crop-month combinations have therefore errors above 3%, but the picture is better than that of country figures in section 2.4.2. This slight improvement is explained by the fact that EU-15 forecasts are weighted averages of country forecasts, and consequently their error is an average of the country errors. Average ranks for months show a clear reduction of error size as the year advances (with the exception of July). The evident reduction however is not enough to give a significant result in statistical testing according to Page's test (which tests whether error size decreases on average from month to month): the application of the test gives a non-significant result at the 0.05 level (p-value: 0.56). However, since we cannot consider months as independent one from the other, this result must be viewed as a lack of strong indications that the error size of MCYFS forecasts decreases as the year advances, at least at the EU-15 level. A diagrammatic representation of Table 6.5 and further evidence for the decrease of error size is given by the following line chart:

Figure 6.6: MAPEs of MCYFS forecasts for the EU-15 yield of several crops (averaging across years).

The same pattern already observed in Figure 5, that is the deterioration of forecasts in July, can be detected in Figure 6 as well; in fact if July conformed to the pattern of the other months, the result of Page's test would be even more favourable to the hypothesis that MCYFS improve as the year advances. In an effort to quantify the decrease of MAPE as the year advances a linear regression model has been applied, separately for each crop, of MAPE on month. Months have been codified giving a value of –4 to March, -3 to April, and so on, arriving at 3 for October and 4 for November. In Table 6 the following results of the regressions for each crop are reported: R2 , the slope of each regression line and the statistical significance of the slope's estimate. The slope represents the average change of MAPE per month. Table 6.6: Regression results for estimation of the relationship between MAPE and month CROP AVERAGE CHANGE OF MAPE PER MONTH P-VALUE OF ESTIMATE OF CHANGE R2 Soft wheat -0.33 0.056 0.428 Durum wheat -1.23 0.004 0.709 Barley -0.27 0.010 0.633 Grain maize -0.33 0.075 0.435 Rape seed -0.27 0.061 0.416 Potato -0.77 0.039 0.537 Sugar beet -0.5 0.065 0.459 Sunflower -0.17 0.694 0.034

For all crops the regressions indicate a reduction in MAPE from month to month. The strength of the relationship between MAPE and month varies across crops, as does the statistical significance, from very strong as in Durum wheat, Barley and Potato to very weak as in Sunflower. Average reduction of MAPE per month is less than 1 MAPE unit (1%) for all crops except Durum wheat whose MAPE decreases by 1.23 units per month. Differences in average reduction must however be considered in relation to the starting value of each crop's MAPE. Durum wheat shows the largest reduction because it starts in March with a MAPE which is almost the triple of the other crops' MAPE and therefore has greater scope for reduction.