Qualitative assessment of mars prediction
The QUAMP study was launched to evaluate the performances of the forecasting system after 10 years of forecast production. The evaluation takes care of data until 2002. A previous study was made in 1998 when the decision to use the system in an operational context was taken.
In the first global evaluation of the system performances made in 1998 (Genovese, 1998) the RMSE error has been used as error indicator by comparing the MCYFS forecasts with ex-post observed results. The average yield forecast error ranged from about 3% to 5% at EU level for the main crops (8.3% for durum wheat). In absolute terms the average error for wheat ranged between about 2 and 4 quintals ha -1 both for European and national levels (Genovese, 2001). In general what has been shown is that the system gives higher errors at the beginning of the season and lower at the end according to a cumulative effect of the climate impact on the crop behaviour.
It is stressed that the assumption of the analysis is that the observed yield (official time series) used as reference are considered without errors and therefore the use of the word MCYFS error maybe not appropriate. More over the project was run in 2003 and at the time of error calculation the official statistics available for 2002 for some crops and countries were not stable.
The table below shows the dynamic of the average RMSE yield error in the course of the year at EU level for the main cereal crops (the % values are calculated over the first 5 years of forecast publications), period 1993-1997. For soft wheat the RMSE error is around 5% in April and decreases to about 3% in September. This analysis was conducted at national level and for the main crops. The error analysis is repeated each year in order to carefully watch the performance of the MCYFS, individuate sources of errors and take decision on which part of the MCYFS has to be improved.
In the following sections the error of MCYFS forecasts is analyzed for the following eight crops of interest:
- Soft wheat
- Durum wheat
- Grain maize
- Rape seed
- Sugar beet
After a short description of the error indicators adopted, the analysis kicks-off with an examination of MCYFS' overall error at the EU-15 level. A more detailed analysis of the MCYFS is then reported in the following sections, where the error is analyzed from three different points of view:
- Spatial distribution: averaged error figures which refer to combinations of crops and countries (EU member states) are examined. For each combination all monthly forecasts produced by MCYFS are considered, percentage and absolute percentage errors are computed and the errors over months and years are averaged. Specific marginal crop, country combinations with very small cultivated areas (below 25000 hectares) were left out of the comparisons;
- Evolution over months: bias and average error size of MCYFS forecasts for EU-15 yields, per month and the way it changes;
- Evolution over years: bias and average error size of MCYFS forecasts for EU-15 yields, per year and the way it changes are analyzed.
Finally, the last section examines whether the performance of the MCYFS in the period 1998-2002 improved compared to its performance in the period 1993-1997.
One aspect of the analysis, from either point of view, concerns the bias of MCYFS predictions. As will be seen in the following sections, averaging the error of MCYFS over months, years or countries, shows that bias exists in the predictions. In some cases it is systematically negative (underestimation of true yield) or systematically positive (overestimation); in other cases the bias is sometimes positive and sometimes negative without a strong pattern.
The main aim of the examination of bias will therefore not be to identify bias (it exists anyway!) but to identify cases of systematic underestimation or overestimation. Therefore, where in the following sections statistical tests for bias are applied or significant bias is mentioned, the concern is on systematic bias.
The other aspect of analysis is error size, i.e. absolute error, the absolute difference between forecast and true yield.