Meteorological data from ECMWF models
The is one of the world's leading numerical modeling centres. It operates a set of global models and of data assimilation systems for the dynamics, thermodynamics and composition of the Earth's fluid envelope and interacting parts of the Earth-system. The data assimilation systems bring observations from ground stations, radiosondes, satellites and many other sources in balance with the meteorological equations to form a physically valid state of the atmosphere. These data is used as initial condition for the various forecast model sets.
In order to extend the period of analysis and to better perform the crop monitoring and yield forecasting, weather forecasts are integrated in the MCYFS. These data permit to have important information on the evolution of the main meteorological phenomena at mesoscale.
Data from ECMWF's Ensemble Prediction System (ENS and ENSextended) and Seasonal forecast model (SEAS) have multiple forecast results. As the atmosphere is a chaotic system where small differences in the initial conditions can lead to in huge differences in the resulting forecasts in 1992 ECMWF introduced an ensemble prediction system, providing information on the uncertainty of a weather forecast. Small perturbations of the initial state are used to produce (nowadays) 50 different initial conditions. Together with the unpertubated control run this results in an ensemble of 51 model results.
Before ECMWF forecasted weather data can be ingested in the MCYFS, the data has to be preprocessed in order to get the appropriate resolutions in time and space.
Data acquisition from ECMWF
Data from six products of the ECMWF model suite is ingested into the MCYFS:
|Model set with ECMWF's abbrevation||Abbreviation within Marsop-4||Number of forecast days used for MCYFS||Number of ensemble members||Original ECMWF grid*||Corresponding original horizontal model resolution*||Acquired resolution in MCYFS**||Delivery of data files and maps|
|ERA-Interim****||ERA||1||1||N128 reduced Gaussian grid||~80 km||0.75° x 0.75°||2nd quarter of new year for the previous year|
|Deterministic model as analysis HRES||OPE||1||1||O1280 octahedral grid***||~9 km ***||0.25° x 0.25°||Daily (10.30 hr)|
|Deterministic forecast HRES||OPE||10||1||O1280 octahedral grid***||~9 km ***||0.25° x 0.25°||Daily (12.00 hr)|
|Ensemble Prediction System ENS||ENS||15||50+1||O640 octahedral grid ***||~18 km ***||0.5° x 0.5°||Daily (14.00 hr)|
|Monthly forecast ENS extended||ENSEXT||32||50+1||O640/O320 octahedral grid ***/******||~18 km / ~36 km ***/******||0.5° x 0.5°||Every Friday (03.00 hr)|
|Seasonal forecast system SEAS||SEAS||183||50+1||N128 reduced Gaussian grid||~80 km||0.75° x 0.75°||Every 8th of the month (14.00 hr)|
* Grid in which the model simulates the weather indicators (state: July 2016). Depending on the model subset, ECMWF uses for surface and pressure levels either a Octahedral grid or a . The octahedral grid names start with ‘O’ followed by the number of latitude lines between the pole and equator. Gaussian grid names start with 'N' followed by number of lines by which latitude is divided.
** Spatial resolution in which the simulated indicators are acquired and loaded into the MCYFS. The simulated indicators are distributed over the earth using a coordinate system.
*** Before March 08th 2016, for surface parameters a was used. The HRES was computed on a N640 grid what corresponds to a horizontal grid size of approximately 16km. ENS and ENSextended was computed during the first 10 days on a N320 grid (~30km horizontal resolution), the remaining days on a N160 grid (~60km horizontal resolution).
**** In more detail: ECWMF runs ERA-Interim on the 2006 release of the integrated forecasting system (IFS) version, Cy31r2.
***** HRES and ENS are run by ECMWF twice daily, based on 00 and 12 UTC observations. The ENSextended is computed by ECMWF twice weekly, basing on Mon 00 and Thu 00 UTC observations. Finally, the SEAS is started by ECMWF each 01st of the month as 00 UTC-run. Depending on the forecast horizon it takes between 5.5 hours (+0 hours HRES) and nearly 9 days (last day SEAS) until the centre disseminates the results.
****** During the first 15 days of the forecast horizo,n ENS and ENSEXT are the same model. After day 15, the ENS is stopped and the ENSEXT is run on a coarser grid. For surface parameters, this is octahedral O320 grid, what translates into a spatial resolution of approximately 36 kilometres.
The short range results of the subsequent, overlapping HRES model are processed as analysis of the previous day and added to the archive (as OPE), assuming this is the best estimator for weather indicators of that day. Details are described below.
The other data of the forecasting suite is replaced when a more recent forecast becomes available (OPE forecast, ENS, ENSEXT and SEAS). As the delivery into the JRC databases needs to take place until 15.00 hours of each day in standard situation the 00 UTC model runs are used. In the rare care that the model dissemination is delayed as fallback the 12 UTC model result of the previous day is taken into account.
ECMWF’s reanalysis data set ERA-Interim is used in the Marsop-projects to build a consistent archive of gridded model results from January 1989 onwards. Below, details are described. Together with the OPE analysis, the ERA-Interim is used within Marsop-4 to calculate climatology.
The ECMWF model computes surface parameters of HRES, ENS and ENSextended on octahedral grids, with different resolutions (since 08 March 2016). Previous model cycles, as well as the current version of the SEAS and the ERA-Interim use a reduced Gaussian grid. The central MCYFS database however requires the initial data in a specific grid resolution with regular latitudes and longitudes, see table XXX. Therefore, conversion is needed.
The deterministic forecast model, within Marsop-4 addressed as OPE, including the short range forecast which is used as analysis, produce forecast weather for grid cells currently on a Octahedral O1280 grid (~9x~9km). This resolution is converted by ECMWF to a reduced Gaussian N640 grid (~16x~16km). As a first step of the Marsop-processing chain, a conversion of the N640 to a regular 0.25 x 0.25 degrees latitude longitude grid (OPE grid) is done. The height model for the OPE is calculated in the same way as the data sets: first the Octahedral grid is converted to a Gaussian N640 reduced grid and next to the regular 0.25° OPE grid (~25x~25km). In addition the height model of a previous version of OPE model (prior to March 2016) is available. The previous OPE version was run on a Gaussian N640 reduced grid and the related height model was directly converted into the OPE grid. For the grid conversion, original software from ECMWF is applied. The grid description is stored in table GRID_<MODEL>.
All surface parameters of the ENS forecast are calculated on a Octahedral O640 grid (~18x~18km). This resolution is converted by ECMWF to a reduced Gaussian N200 grid. As a first step of the Marsop-processing chain, a conversion of the N200 to a regular 0.5 x 0.5 degrees latitude longitude grid (ENS grid) is done. The height model for the ENS is calculated in the same way as the data sets: first the Octahedral grid is converted to a Gaussian N200 reduced grid and next to the regular 0.5° ENS grid. In addition, the height model of a previous version of ENS model (prior to March 2016) is available. For the grid conversion original software from ECMWF is applied. The grid description is stored in table GRID_<MODEL>.
AThe first 15 forecast days of the extended ensemble forecast ENSEXT base on the ENS forecast of the same run. After day 15, the remaining days of the ENSEXT are computed on a coarser grid, what is an octahedral O320 grid for surface parameters. This translates into a spatial resolution of approximately 38 kilometres. To deliver the required regular 0.5° grid for the Marsop-4 processing, ENSEXT is ingested from ECMWF on a reduced Gaussian N128 grid and as the first step of the processing chain converted into the requested regular 0.5° grid. The height model for the OPE is calculated in the same way as the data sets: first the Octahedral O320 grid is converted to a Gaussian N128 reduced grid and next to the regular 0.5° ENSEXT grid. In addition, the height model of a previous version of ENSEXT model (prior to March 2016) is available. For the grid conversion original software from ECMWF is applied. The grid description is stored in table GRID_<MODEL>.
SEASAll forecast days of the Seasonal forecast are calculated for a Gaussian N128 reduced grid (~80x~80km). The results are directly converted into a regular 0.75 x 0.75 degrees latitude longitude grid. The grid description is stored in table GRID_<MODEL>.
ERAThe ERA data are calculated for a Gaussian N128 reduced grid (~80x~80km). The results are directly converted into a regular 0.75 x 0.75 degrees latitude longitude grid. The grid description is stored in table ECMWF_ERA_GRID_GLD (linked to view ECMWF_ERA_GRID).
Applied parameters from ECMWF grib deliveries
In total, analysis and forecast for 35 parameters of the ECMWF re-analysis and forecasting suite is used for the various applications in MCYFS and the production of the static maps.
|List of meteorological indicators from ECMWF as used within Marsop-4|
ECMWF disseminates the model results for the surface layer in WMO FM 92 GRIB format, according WMO specifications, Manual on Codes in WMO Publication Nr 306. To extract the required parameters from the ECMWF data package(s) and to decode the binary GRIB formats the ECMWF GRIB API application program interface for C is used.
As a next step after acquisition and scaling to the regular lat-lon-grids, derived elements and daily indicators as required by JRC are calculated
Aggregation to daily data
First, aggregates of the 3- or 6-hourly data to daily means, extremes or sums are calculated. Total precipitation and global radiation are provided by ECMWF as accumulated values since the begin of the model runtime and therefore differences for the 24-hourly daily sums need to be computed. Algorithms had been developed in the ASEMARS project and differ per ECMWF model set. The box below summarizes the algorithms.
|Algorithms for aggregation to daily data|
|Abbreviations for the elements in the following table refer to the original ECMWF naiming as summarized in section Applied parameters from ECMWF grib deliveries. Subscripted numbers behind the indicator abbreviations indicate the (UTC)-time of the day.
The abbreviations for the model sets refer to the internal naming within Marsop4 as defined in section Data acquisition from ECMWF
To consider the earth's different times zones aggregation rules for 3 different areas (West, Central, East) have been defined. The aggregation rules for the model data align with the general report schedule of ground weather stations (e.g., maximum air temperature in Europe and Africa refers to the period between 06 and 18 UTC of the corresponding day). The following table summarized the deviation rules for the different aggregation zones and data sets. p = previous day, f = following day Temporal resolution of OPE is 3-hourly is 3-hourly for the first 72 hours and 6-hourly afterwards. Thus algorithms for air temperature, dew point and wind speed of the OPE data set change when the aggregation includes forecast time step +72h. Temporal resolution of ENS, ENSEXT and SEAS is 6-hourly. ERA-Interim is available every 3-hours.
* ”X” as representative abbreviation for the ECMWF elements as listed in the first cell of the line
The short range results of the subsequent, overlapping HRES model are processed as analysis of the previous day and added to the archive (as OPE), assuming this is the best estimator for weather indicators of that day. Details are described below. As implemented in prior Marsop-projects, for the analysis of the previous day short range results from subsequent, overlapping runs of the deterministic ECMWF model (HRES) are used. This approach is assumed to be the best estimator for weather indicators of that day. The ECMWF HRES model is initialized twice daily, using the observations of 00 and 12 UTC, respectively, as starting conditions. As in prior Marsop-projects, the first 12 hours of a particular run are not used for the analysis estimate, to avoid possible model spin-up effects. Such effects have been shown in the past e.g. for convective precipitation or wind. For consistency of the weather elements in the analysis, the first 12 hours of the particular runs are skipped for all elements.
This results in a complex computation scheme for the analysis of a certain day:
* ”X” as representative abbreviation for the ECMWF elements as listed in the first cell of the line
Calculation of advanced parameters
Not all indicators can be retrieved directly from the models. These include:
- Transpiration of water surface
- Transpiration of wet bare soil
- Climate water balance
- Vapour pressure
- Snow depth (thickness snow cover)
In general, the evapotranspiration from a reference surface, the so-called reference crop evapotranspiration or reference evapotranspiration can be described by the FAO‑Penman-Monteith (Allen et all., 1998).
Evapotranspiration from a wet bare soil surface (ES0) and from a crop canopy (ET0) is calculated with the well-known Penman formula (Penman, 1948). In general, the evapotranspiration from a water surface (E0) can be described by the Penman formula. Only the albedo and surface roughness differs for these two types of evapotranspiration as explained below:
The net absorbed radiation depends on incoming global radiation, net outgoing long-wave radiation, the latent heat and the reflection coefficient of the considered surface (albedo). For ET0, ES0, and ET0 albedo values of 0.05, 0.15 and 0.20 are used respectively. The evaporative demand is determined by humidity, wind speed and surface roughness. For a free water surface and for the wet bare soil (E0, ES0) a surface roughness value of 0.5 is used. For a more detailed description of the underlying formulae we refer to Supit et al. (1994) and van der Goot (1997).
Climatic water balance
Climatic water balance is calculated based on evapotranspiration calculated through the equation of Penman-Monteith and the total precipitation of a day.
|CWB equals Rain – ET0|
The snow depth (thickness of the snow layer) is derived from snow depth water equivalent and snow density.
|Dsn equals r_water/r_snow * S/c_snow|
|The ECMWF provides snow depth water equivalent SD (m3/m2) for all sets and snow density RSN (kg/m3). According to ECMWF documentation the thickness of the snow covering the ground, Dsn, can be derived with the approach:
Dsn equals r_water/r_snow*S/c_snow
In ECMWF's model documentation snow mass is referred as “snow water equivalent”, and leads to parameter SD, snow depth. Snow fraction is not provided by ECMWF (is not in the catalogue). ECMWF assumes c_snow to be 1 for snow depth > 15 cm (average of the grid box) and <1 for a thinner snow cover.
Calculation of extreme weather events
For the static map production (quicklooks) it is necessary to derive additional parameters out of the raw ECMWF data set. This especially concerns probabilities and aggregated counts of number of days where a special condition is met. To compute the probability for the exceedance of thresholds (e.g. probability of freezing days) first the daily value for each separate ensemble member is computed and then the amount of members which fit the corresponding constraint (p.e. exceed 20mm of daily precipitation sum) is counted. To compute the number of days where a parameter exceed a threshold first the numbers for each separate ensemble member is calculated (from the daily values of each ensemble member). Afterwards the median) is derived for presentation on map. The deterministic run and the ensemble control run are treated like any other ensemble member. Probabilities for anomalies require comparison with ECMWF model climate and are therefore only visualized where available in the ECMWF catalogue.
|Derived probability and other threshold-dependent indicators|
Aggregation to 10-daily, weekly and monthly data
For the production of the maps, as well an aggregation to 10-daily, weekly and monthly aggregates of the daily data takes place. Therefore the average of mean temperature, maximum temperature, minimum temperature, snow depth and the sum of precipitation, ET0, climatic water balance and global radiation is computed.
Extraction of data into files
After processing data are exported as data files and static maps that can be distributed to users and other MCYFS processes.
A simple file naming scheme for the data files was adopted with the general format:
- ROI = region (GLD, EUR, ASI)
- model_code = ECMWF model (ERA, OPE, ENS, ENSEXT or SEAS)
- yyyy = the year (four digits),
- mm = the month number (two digits),
- dd = the day in the month (two digits)
- member = the member number (two digits)
The date in the filename links to the forecast day = 0 (FORECAST_OFFSET = 0). In case of OPE and ERA only member 00 is allowed; in case of the ENS, the ENSEXT and the SEAS the member number runs from 0 to 50.
An example of a file name for each of the 4 models is:
- GLO_OPE_20160715_00.dat OPE data issued July 15, 2016 (only member 00 allowed)
- GLO_ENS_20160704_35.dat ENS data issued July 4, 2016, member 35
- EUR_ENSEXT_20160721_32.dat ENSEXT data basing on ECMWF run July 21, 2016, member 32. *
- EUR_SEAS_20160601_34.dat SEAS data basing on ECMWF run June 1*, 2016, member 34
* ENSEXT is initialized by ECMWF with the observations of Thu 00 and delivered into MCYFS approximately 27 hours later.
Note the OPE and EPS start with member number 0 while the MON and SEA start with member number 1. The date in the filename links to the forecast day = 0 (FORECAST_OFFSET = 0).
An input file basically contains the following structure:
- A header providing geo referencing information
- Blocks of data for the first forecast date (for each variable)
- Blocks of data for the second forecast date (for each variable)
For simplification purposes, below a simple example is given with a detailed explanation.
|Explanation of file format|
| The example contains just rainfall and daily mean temperature for two forecast days for a grid ranging from 20 to 40 degrees longitude and from 50 to 60 degrees latitude, with a grid size of 5 degrees. The forecast is issued on 23 January 2009 and first day in the forecast (FORECAST_OFFSET=0) is linked to this date.
The meaning of each of the lines is given in the following table:
The possible forecast offsets are given in the following table:
* = Day of ECMWF model initialization (“model run”). For instance if the model is initialized on the first October 2014 00 UTC the FORECAST OFFSET = 0 refers to Oct 01 2014. Data is always aggregated to daily values. See MarsWiki for aggregation rules from 3- resp. 6-hourly data to daily values.
The provided grids are summerized in the following table:
All elements refer to the model surface. The temperatures refer to a level 2 meters above (model) ground, wind speed refers to a level 10 meters above (model) ground.
The data files are loaded in the tables WEATHER_<MODEL>_GRID_RAW where <MODEL> is to be replaced by the abbreviation of one of the five ECMWF products (OPE, ENS, ENSEXT or SEAS). In case of ERA data are stored in table ECMWF_ERA_DATA. During loading two actions are executed:
- unit conversion
- plausible range checks
|Unit conversion and range checking|
Extraction of data into maps
The static maps are exported as flat images and animated images with full layout and directly made available to analysts that use them during analysis of weather indicators. The geographic extent of the static maps for Europe is defined by the upper-left corner at 75° North/25° West and the lower-right corner 20° North/70° East, the global maps cover a domain 75° North/180° West to 50° South/180° East. This production line includes GrADS mapping software which is able to create maps directly from GRIB files.
|Overview: Produced maps|
The maps are delivered via ftp to JRC. The abbreviations for the various maps (including the animated) in the following tables are used for the directory structure on JRC's ftp-server and the transmitted zip. The zip-name is extended by the day of the underlying ECMWF model set run.
The following table summarizes the map production as set up for the OPE and the ENS:
The following table summarizes the map production as set up for the ENSEXT and the SEAS:
Scaling ingested grid data to agricultural target grids
As summarized in section Data acquisition from ECMWF forecasts for the subsets ENS, ENSEXT, SEAS and ERA are delivered into the MCYFS on coarser grids than the aspired MCYFS target grids of 0.25° or 25 kilometres. That is why scaling methods towards the target grids are used in the system.
Aim is to have forecasts of the ECMWF subsets ENS, ENSEXT, SEAS and ERA that are as comparable and consistent as possible with the deterministic analysis and forecast OPE. The OPE analysis combines the most advanced assimilation system for observed atmospheric data with highest model grid resolution, involving the most accurate model physics, elevation and land use and soil type models.
Simple interpolation methods, as inverse distance or spline, do not add information to the data. Differences in elevation, land use, land-sea-pattern are not considered. That is why in Marsop-4 for the temperature and humidity elements, wind and radiation statistical relations between the subsets and the OPE analysis are applied. Each land grid point has its location-specific, time-dependant set of equations per subset.
The below pictures visualize the advantage of the applied downscaling method. The downscaling (middle) added information to the raw data (left). Downscaled data and the OPE analysis (right) align.