Difference between revisions of "Meteorological data from ECMWF models"
(→Data acquisition from ECMWF) 
(→Data acquisition from ECMWF) 

Line 20:  Line 20:  
!Model set with ECMWF's abbrevation !! Abbreviation within Marsop4 !!Number of forecast days used for MCYFS !! Number of ensemble members !! Original ECMWF grid* !! Corresponding original horizontal model resolution* !! Acquired resolution in MCYFS** !! Delivery of data files and maps  !Model set with ECMWF's abbrevation !! Abbreviation within Marsop4 !!Number of forecast days used for MCYFS !! Number of ensemble members !! Original ECMWF grid* !! Corresponding original horizontal model resolution* !! Acquired resolution in MCYFS** !! Delivery of data files and maps  
    
−  ERAInterim  +  ERAInterim****  ERA  1  1  N128 reduced Gaussian grid ~80 km  0.75° x 0.75°  2nd quarter of new year for the previous year 
    
−  Deterministic model as analysis HRES  OPE 1  1  O1280 octahedral grid***  ~9 km  0.25° x 0.25°  Daily (10.30 hr)  +  Deterministic model as analysis HRES  OPE 1  1  O1280 octahedral grid***  ~9 km *** 0.25° x 0.25°  Daily (10.30 hr) 
    
−  Deterministic forecast HRES  OPE 10  1  O1280 octahedral grid***  ~9 km  0.25° x 0.25°  Daily (12.00 hr)  +  Deterministic forecast HRES  OPE 10  1  O1280 octahedral grid***  ~9 km *** 0.25° x 0.25°  Daily (12.00 hr) 
    
−  Ensemble Prediction System ENS  ENS 15  50+1  O640 octahedral grid***  ~18 km***  0.5° x 0.5°  Daily (14.00 hr)  +  Ensemble Prediction System ENS  ENS 15  50+1  O640 octahedral grid ***  ~18 km ***  0.5° x 0.5°  Daily (14.00 hr) 
    
−  Monthly forecast ENS extended  ENSEXT 32  50+1  O640/O320 octahedral grid***  ~18 km / ~36 km***  0.5° x 0.5°  Every Friday (03.00 hr)  +  Monthly forecast ENS extended  ENSEXT 32  50+1  O640/O320 octahedral grid ***/******  ~18 km / ~36 km ***/******  0.5° x 0.5°  Every Friday (03.00 hr) 
    
Seasonal forecast system SEAS  SEAS 183  50+1  N128 reduced Gaussian grid  ~80 km  0.75° x 0.75°  Every 8th of the month (14.00 hr)  Seasonal forecast system SEAS  SEAS 183  50+1  N128 reduced Gaussian grid  ~80 km  0.75° x 0.75°  Every 8th of the month (14.00 hr)  
}  }  
−  <nowiki>  +  <nowiki>*</nowiki> Grid in which the model simulates the weather indicators (state: July 2016). Depending on the model subset, ECMWF uses for surface and pressure levels either a [https://software.ecmwf.int/wiki/display/FCST/Introducing+the+octahedral+reduced+Gaussian+grid Octahedral grid] or a {{GloshintReduced Gaussian gridReduced Gaussian grid}}. The octahedral grid names start with ‘O’ followed by the number of latitude lines between the pole and equator. Gaussian grid names start with 'N' followed by number of lines by which latitude is divided.<br> 
−  <nowiki>  +  <nowiki>**</nowiki> Spatial resolution in which the simulated indicators are acquired and loaded into the MCYFS. The simulated indicators are distributed over the earth using a {{GloshintWGS84World Geodetic System, revision 1984WGS84}} coordinate system.<br> 
−  <nowiki>  +  <nowiki>***</nowiki> Before March 08th 2016, for surface parameters a {{GloshintReduced Gaussian gridReduced Gaussian grid}} was used. The HRES was computed on a N640 grid what corresponds to a horizontal grid size of approximately 16km. ENS and ENSextended was computed during the first 10 days on a N320 grid (~30km horizontal resolution), the remaining days on a N160 grid (~60km horizontal resolution).<br> 
−  <nowiki>  +  <nowiki>****</nowiki> In more detail: ECWMF runs ERAInterim on the 2006 release of the integrated forecasting system (IFS) version, Cy31r2.<br> 
−  <nowiki>  +  <nowiki>*****</nowiki> HRES and ENS are run by ECMWF twice daily, based on 00 and 12 UTC observations. The ENSextended is computed by ECMWF twice weekly, basing on Mon 00 and Thu 00 UTC observations. Finally, the SEAS is started by ECMWF each 01st of the month as 00 UTCrun. Depending on the forecast horizon it takes between 5.5 hours (+0 hours HRES) and nearly 9 days (last day SEAS) until the centre disseminates the results.<br> 
−  <nowiki>  +  <nowiki>******</nowiki> During the first 15 days of the forecast horizo,n ENS and ENSEXT are the same model. After day 15, the ENS is stopped and the ENSEXT is run on a coarser grid. For surface parameters, this is octahedral O320 grid, what translates into a spatial resolution of approximately 36 kilometres. <br> 
The short range results of the subsequent, overlapping HRES model are processed as analysis of the previous day and added to the archive (as OPE), assuming this is the best estimator for weather indicators of that day. Details are described below.  The short range results of the subsequent, overlapping HRES model are processed as analysis of the previous day and added to the archive (as OPE), assuming this is the best estimator for weather indicators of that day. Details are described below. 
Revision as of 15:43, 15 July 2016
General description
The ECMWF is one of the world's leading numerical modeling centres. It operates a set of global models and of data assimilation systems for the dynamics, thermodynamics and composition of the Earth's fluid envelope and interacting parts of the Earthsystem. The data assimilation systems bring observations from ground stations, radiosondes, satellites and many other sources in balance with the meteorological equations to form a physically valid state of the atmosphere. These data is used as initial condition for the various forecast model sets.
In order to extend the period of analysis and to better perform the crop monitoring and yield forecasting, weather forecasts are integrated in the MCYFS. These data permit to have important information on the evolution of the main meteorological phenomena at mesoscale.
The ECMWF model results are used to produce meteorological and derived agrometeorological parameters that are visualized in dynamic maps and graphs by the MARS viewer and static map quicklooks.
Data from ECMWF's Ensemble Prediction System (ENS and ENSextended) and Seasonal forecast model (SEAS) have multiple forecast results. As the atmosphere is a chaotic system where small differences in the initial conditions can lead to in huge differences in the resulting forecasts in 1992 ECMWF introduced an ensemble prediction system, providing information on the uncertainty of a weather forecast. Small perturbations of the initial state are used to produce (nowadays) 50 different initial conditions. Together with the unpertubated control run this results in an ensemble of 51 model results.
Before ECMWF forecasted weather data can be ingested in the MCYFS, the data has to be preprocessed in order to get the appropriate resolutions in time and space.
Data acquisition from ECMWF
Model results for surface and pressure levels is provided by ECMWF in FM92 GRIB format which is specified in WMO Publication 306 Manual on Codes.
Data from six products of the ECMWF model suite is ingested into the MCYFS:
Model set with ECMWF's abbrevation  Abbreviation within Marsop4  Number of forecast days used for MCYFS  Number of ensemble members  Original ECMWF grid*  Corresponding original horizontal model resolution*  Acquired resolution in MCYFS**  Delivery of data files and maps 

ERAInterim****  ERA  1  1  N128 reduced Gaussian grid  ~80 km  0.75° x 0.75°  2nd quarter of new year for the previous year 
Deterministic model as analysis HRES  OPE  1  1  O1280 octahedral grid***  ~9 km ***  0.25° x 0.25°  Daily (10.30 hr) 
Deterministic forecast HRES  OPE  10  1  O1280 octahedral grid***  ~9 km ***  0.25° x 0.25°  Daily (12.00 hr) 
Ensemble Prediction System ENS  ENS  15  50+1  O640 octahedral grid ***  ~18 km ***  0.5° x 0.5°  Daily (14.00 hr) 
Monthly forecast ENS extended  ENSEXT  32  50+1  O640/O320 octahedral grid ***/******  ~18 km / ~36 km ***/******  0.5° x 0.5°  Every Friday (03.00 hr) 
Seasonal forecast system SEAS  SEAS  183  50+1  N128 reduced Gaussian grid  ~80 km  0.75° x 0.75°  Every 8th of the month (14.00 hr) 
* Grid in which the model simulates the weather indicators (state: July 2016). Depending on the model subset, ECMWF uses for surface and pressure levels either a Octahedral grid or a Reduced Gaussian grid. The octahedral grid names start with ‘O’ followed by the number of latitude lines between the pole and equator. Gaussian grid names start with 'N' followed by number of lines by which latitude is divided.
** Spatial resolution in which the simulated indicators are acquired and loaded into the MCYFS. The simulated indicators are distributed over the earth using a WGS84 coordinate system.
*** Before March 08th 2016, for surface parameters a Reduced Gaussian grid was used. The HRES was computed on a N640 grid what corresponds to a horizontal grid size of approximately 16km. ENS and ENSextended was computed during the first 10 days on a N320 grid (~30km horizontal resolution), the remaining days on a N160 grid (~60km horizontal resolution).
**** In more detail: ECWMF runs ERAInterim on the 2006 release of the integrated forecasting system (IFS) version, Cy31r2.
***** HRES and ENS are run by ECMWF twice daily, based on 00 and 12 UTC observations. The ENSextended is computed by ECMWF twice weekly, basing on Mon 00 and Thu 00 UTC observations. Finally, the SEAS is started by ECMWF each 01st of the month as 00 UTCrun. Depending on the forecast horizon it takes between 5.5 hours (+0 hours HRES) and nearly 9 days (last day SEAS) until the centre disseminates the results.
****** During the first 15 days of the forecast horizo,n ENS and ENSEXT are the same model. After day 15, the ENS is stopped and the ENSEXT is run on a coarser grid. For surface parameters, this is octahedral O320 grid, what translates into a spatial resolution of approximately 36 kilometres.
The short range results of the subsequent, overlapping HRES model are processed as analysis of the previous day and added to the archive (as OPE), assuming this is the best estimator for weather indicators of that day. Details are described below.
The other data of the forecasting suite is replaced when a more recent forecast becomes available (OPE forecast, ENS, ENSEXT and SEAS). As the delivery into the JRC databases needs to take place until 15.00 hours of each day in standard situation the 00 UTC model runs are used. In the rare care that the model dissemination is delayed as fallback the 12 UTC model result of the previous day is taken into account.
ECMWF’s reanalysis data set ERAInterim is used in the Marsopprojects to build a consistent archive of gridded model results from January 1989 onwards. Below, details are described. Together with the OPE analysis, the ERAInterim is used within Marsop4 to calculate climatology.
Spatial representation
The ECMWF models run on Gaussian grids, for certain parameters and model levels on spectral grids, with different resolutions. The central MCYFS database however requires the initial data in a specific grid resolution with regular latitudes and longitudes. Therefore conversions are needed.
OPE
The Deterministic forecast model and Analysis model (OPE) produce forecast weather for grid cells on a Gaussian N640 reduced grid (~16x~16km). The resolution is converted to a Gaussian N400 reduced grid (~25x~25km) and after this to a regular 0.25 x 0.25 degrees latitude longitude grid (OPE grid). For the OPE grid two height models are kept. First a height model calculated in the same way as the data sets: first aggregation on the Gaussian grid from a Gaussian N640 reduced grid (~16x~16km) to a Gaussian N400 reduced grid (~25x~25km) and next a conversion from the Gaussian grid to the OPE grid. In addition the height model of a previous version of OPE model (prior to January 2010) is available. The previous OPE version was run on a Gaussian N400 reduced grid (~25x~25km) and the related height model was directly converted into the OPE grid. The grid description is stored in table GRID_<MODEL>.
ENS & ENSEXT
The first 10 forecast days (Leg A) of the Ensemble Prediction System and Monthly forecast are modelled for grid cells on a Gaussian N320 reduced grid (~30x~30km). Because the modelling of the remaining days (Leg B and C) is on the Gaussian N160 reduced grid (~60x~60km) it is not possible to switch for the whole forecast depth (EPS: 15 days and MON: 32 days) to a finer resolution. It means that the data of first 10 days must be aggregated. First the resolution is reduced to a Gaussian N200 reduced grid (~50x~50km) and finally converted to a regular 0.5 x 0.5 degrees latitude longitude grid. The height model of the latter grid is calculated in the same way as the data sets: first aggregation on the Gaussian grid from N320 (~30x~30km) to N200 (~50x~50km) and next a conversion from the Gaussian grid N200 to the regular 0.5 x 0.5 degrees latitude longitude grid.
After the first 10 day, the resolution of the models for the remaining forecast days (Leg B and C) is at a Gaussian N160 reduced grid (~60x~60km). The results are directly converted into a regular 0.5 x 0.5 degrees latitude longitude grid.
The grid description is stored in table GRID_<MODEL>.
SEAS
All forecast days of the Seasonal forecast are calculated for a Gaussian N128 reduced grid (~80x~80km). The results are directly converted into a regular 0.75 x 0.75 degrees latitude longitude grid. The grid description is stored in table GRID_<MODEL>.ERA
The ERA data are calculated for a Gaussian N128 reduced grid (~80x~80km). The results are directly converted into a regular 0.75 x 0.75 degrees latitude longitude grid. The grid description is stored in table ECMWF_ERA_GRID_GLD (linked to view ECMWF_ERA_GRID).Decoding and extraction of GRIB data
Data is delivered in GRIB format and hence data is first decoded. In previous years the program ‘wgrib’ has been used which is capable of extracting GRIB content into ASCII files for further processing. Recently ECMWF has released version 1.2.0 of their GRIB API which is the successor of GRIBEX. While GRIBEX was used within FORTRAN programs the new GRIB API is designed for programs written in the C programming language.
Abbreviations used in relation with ECMWF indicators  


During decoding additional indicators required by JRC and partners are calculated. This include aggregation to daily data, calculation of derived indicators and calculation of extreme weather events.
Aggregation to daily data
First of all an aggregation of 3, 6 and 12hourly data to daily data is calculated. Algorithms were developed in the ASEMARS project and differ per ECMWF model. The algorithms are presented in the box below.
Algorithms for aggregation to daily data  

Abbreviations are specified in section Decoding and extraction of GRIB data. Subscript numbers behind the indicator abbreviations indicate the time of the day.
Aggregation areasTo consider the earth's different times zones aggregation rules for 3 different areas (East, Central, West) have been defined. The aggregation rules for the model data refer to the report schedule of synoptical weather stations (e.g. maximum air temperature in Europe and Africa refers to the period between 06 and 18 UTC of the corresponding day). The following table summarized the deviation rules for the different aggregation zones and data sets.

Calculation of advanced parameters
Not all indicators can be retrieved directly from the models. These include:
 Evapotranspiration
 Transpiration of water surface
 Transpiration of wet bare soil
 Climate water balance
 Vapour pressure
 Snow height
Evapotranspiration
In general, the evapotranspiration from a reference surface, the socalled reference crop evapotranspiration or reference evapotranspiration can be described by the FAO‑PenmanMonteith (Allen et all., 1998).
Evapotranspiration from a wet bare soil surface (ES0) and from a crop canopy (ET0) is calculated with the wellknown Penman formula (Penman, 1948). In general, the evapotranspiration from a water surface (E0) can be described by the Penman formula. Only the albedo and surface roughness differs for these two types of evapotranspiration as explained below:
The net absorbed radiation depends on incoming global radiation, net outgoing longwave radiation, the latent heat and the reflection coefficient of the considered surface (albedo). For ET0, ES0, and ET0 albedo values of 0.05, 0.15 and 0.20 are used respectively. The evaporative demand is determined by humidity, wind speed and surface roughness. For a free water surface and for the wet bare soil (E0, ES0) a surface roughness value of 0.5 is used. For a more detailed description of the underlying formulae we refer to Supit et al. (1994) and van der Goot (1997).
Climatic water balance
Climatic water balance is calculated based on evapotranspiration calculated through the equation of PenmanMonteith and the total precipitation of a day.
CWB equals Rain – ET0 

where:

Snow height
The snow height (thickness of the snow layer) is derived from snow depth (water equivalent) and snow density.
Dsn equals r_water/r_snow * S/c_snow 

The ECMWF catalogue lists snow depth SD (water equivalent) for all sets and snow density RSN (kg/m3) which is available for OPE, ENS, MON but not for SEA. According to ECMWF documentation snow height Dsn can be derived with the approach:
Dsn equals r_water/r_snow*S/c_snow with
In ECMWF's model documentation snow mass is (sometimes) referred as “snow water equivalent”, and leads to parameter SD, snow depth. Snow fraction is not in the catalogue. ECMWF assumes c_snow to be 1 for snow height > 15 cm (average of the grid box) and <1 for a thinner snow cover.

Calculation of extreme weather events
For the static map production over Europe it is necessary to derive additional parameters out of the raw data set. This especially concerns probabilities and aggregated counts of number of days where a special condition is met. Some of the probabilities to be mapped are available directly from ECMWF. Other probabilities need to be derived from individual ensemble runs. In this case it is checked for how many of the ensemble members a certain condition applies (e.g. TempMin < 0°C). The probability of the event is the fraction of ensemble members forecasting it against the total number of ensemble members. The operational run and the ensemble control run are treated like any other ensemble member.
Derived probability and other thresholddependent indicators  


Aggregation to 10daily and monthly data
After each 10day period and at the end of each month aggregation for this 10day/month period takes place. Additionally a forecast of the next dekad, basing on aggregated forecasts for the next 10 days (resp. 8/9/11 days for the last dekad) is delivered. The daily data are aggregated from days to dekads by taking the average of mean temperature, maximum temperature, minimum temperature, snow depth and the sum of precipitation, ET0 and global radiation.
Additionally for the map production the number of occurrences of certain events (such as frost, hot or rainy days) is counted.
Extraction of data into files
After processing data are exported as data files and static maps that can be distributed to users and other MCYFS processes.
A simple file naming scheme was adopted with the general format:
<ROI>_<model_code>_<timestep>_<yyyy><mm><dd>_<member>.dat
In which:
 ROI = region (GLD, EUR, ASI)
 model_code = ECMWF model (ERA, OPE, EPS, MON or SEA)
 timestep = temporal resolution of data: day, dekad, month
 yyyy = the year (four digits),
 mm = the month number (two digits),
 dd = the day in the month (two digits)
 member = the member number (two digits)
Note the OPE and EPS start with member number 0 while the MON and SEA start with member number 1. The date in the filename links to the forecast day = 0 (FORECAST_OFFSET = 0).
An example of a file name for each of the 4 models is:
 EUR_OPE_day_20100715_00.dat OPE data for July 15, 2010 (only member 00 allowed)
 EUR_EPS_day_20100704_35.dat EPS data for July 4, 2010, member 35
 EUR_MON_day_20100702_32.dat MON data for Friday July 2, 2010, member 32
 EUR_SEA_day_20100601_34.dat SEA data for June 1*, 2007, member 34
* Model runs the 8th but has a hindcast of 8 days
Note the OPE and EPS start with member number 0 while the MON and SEA start with member number 1. The date in the filename links to the forecast day = 0 (FORECAST_OFFSET = 0).
An input file basically contains the following structure:
 A header providing geo referencing information
 Blocks of data for the first forecast date (for each variable)
 Blocks of data for the second forecast date (for each variable)
 etc.
For simplification purposes, below a simple example is given with a detailed explanation.
Explanation of file format  

The example contains just rainfall and daily mean temperature for two forecast dates for a grid ranging from 20 to 40 degrees longitude and from 50 to 60 degrees latitude, with a grid size of 5 degrees. The forecast is issued on 23 January 2009 and first day in the forecast (FORECAST_OFFSET=0) is linked to this date.
The meaning of each of the lines is given in the following table:
The variable abbreviations and their explanation are given in the following table:

The data files are loaded in the tables WEATHER_<MODEL>_GRID_RAW where <MODEL> is to be replaced by the abbreviation of one of the five ECMWF products (HIS, OPE, EPS, MON or SEA). In case of ERA data are stored in table ECMWF_ERA_DATA. During loading two actions are executed:
 unit conversion
 plausible range checks
Unit conversion and range checking  


In parallel daily, decadal and monthly aggregates of the analysis and deterministic forecast (HIS, OPE) is provided as csv to JRC and Vito.
csv format description and deliverables  


Extraction of data into maps
The static maps are exported as flat images or animated images with full layout and directly made available to analysts that use them during analysis of weather indicators. The geographic extent of the static maps is defined by the upperleft corner at 75° North/25° West and the lowerright corner 20° North/70° East. This production line includes GrADS mapping software which is able to create maps directly from GRIB files. For the weekly and monthly maps the absolute difference to longterm average values are calculated.
Overview: Produced maps  

