Difference between revisions of "Meteorological data from ground stations"

From Agri4castWiki
Jump to: navigation, search
(Acquisition, checks and processing daily station data)
(Data quality check)
 
(507 intermediate revisions by 8 users not shown)
Line 1: Line 1:
==Acquisition and processing daily station data==
+
__NOTOC__
The meteorological station data acquisition and check consists of:
+
{{Scientific}}
* Station information.
+
==General description==
* Raw daily meteorological data.
+
The processing of observed station weather into the MCYFS involves four steps:
* Processed daily meteorological data.
+
[[File:Flowchart_station_weather_preprocessing_steps.jpg|link=|frame|preprocessing of station weather data|none]]
  
 +
==Data acquisition from weather stations==
 +
[[Image:All stations with data.jpg|thumb|200px|Weather stations (black dots) for which data are available for (part of) the period from 1975 until present]]
 +
The selection of stations is limited to those stations that regularly collect data and can supply data in near real time. Relevant meta data of stations includes station number, station name, latitude, longitude and altitude. This data is available in the object STATIONS.
  
The stations are limited to those for which data not only are regularly collected but which can also be received and processed in semi-real time (Burrill and Vossen, 1992). Relevant information of stations includes WMO station number, station name, latitude, longitude and altitude. This data is available in the tables STATIONS and WEATHER_STATION. Fig. 2.1 gives an overview of all stations for which daily data are available for (part of) the period from 1975 until now. The total number balances around 6000. Only around 2000 has a sufficient temporal coverage (enough observations within one year) to be used in the spatial interpolation procedure (see § 3).
+
Currently, data acquisition and processing applies to two regional windows: Europe and China. Mainly examples from Europe are shown in this documentation.
  
 +
Some of the historic meteorological data were purchased directly from National Meteorological Services. Others were acquired via the {{Gloshint|GTS|Global Telecommunication System|GTS}}. As data are obtained from a variety of different sources, considerable pre-processing was necessary to convert them into a standard format. Around 1992 the historic meteorological data represented approximately 380 stations in the EU, Switzerland, Poland and Slovenia with data from 1949 to 1991 ([[References|Burrill and Vossen, 1992]]). Later the historic sets have been extended with stations in Eastern Europe, western Russia, Maghreb and Turkey. The historic data were converted into consistent units and checked on realistic values. The database was also scanned for inconsistencies, such as successive days with the same value for a variable, or minimum temperatures higher than maximum temperatures ([[References|Burrill and Vossen, 1992]]).
  
Some of the historic meteorological data are purchased directly from various National Meteorological Services, others are acquired via the Global Telecommunication System (GTS). As the data are obtained from a variety of different sources, considerable preprocessing is necessary to convert them to a standard format. Two different procedures are applied for distinct subsets of the data set. The historic data came directly from National Meteorological Services. Around 1992 they represented approximately 380 stations in the EU, Switzerland, Poland and Slovenia with data from 1949 to 1991 (Burrill and Vossen, 1992). Later the historic sets have been extended with stations in eastern Europe, western Russia, Maghreb and Turkey. The historic data were converted into consistent units and were checked on realistic values. The database was also scanned for inconsistencies, such as successive days with the same value for a variable, or minimum temperatures higher than maximum temperatures (Burrill and Vossen, 1992).
+
From 1991 to present, meteorological data is received in near-real-time from open data sources and from contracted providers like ECOMET or national or regional meteorological services. Sources include the WMO GTS network, NOAA data access points, regional and national meteorological services, and the access points for non-essential WMO reports. The data arrives either in standardized encoded formats as defined by the WMO or ICAO, or in proprietary formats as used by individual providers. It is first decoded and converted into a generalized structure, including unit and time zone conversions, alignment of reference periods and - where needed - the assignment of station-id. Basic first level data sanity checks are applied. In a next step, the data is converted into the input-format as required by the [[Software Tools#QUACKME|QUACKME]] software package. The temporal resolution of the data ranges from 1-hourly to 24-hourly, depending on the parameter. QUACKME  is applying data quality checks, calculates derived parameters and daily aggregates and writes the data into the file formats as expected by the EFAS and MCYFS downstream processes.
  
 +
In recent years, the earlier archives (1975-2004) of Scandinavia and eastern Europe have been enriched. In 2016 data from around 300 Chinese stations have been acquired starting a new service for this region.
  
[[Image:Laea projection.jpg|thumb|250px|right|The meteorological stations for which data are available for (part of) the period from 1975 until the current day]]
+
====Available stations====
From 1991 to present, meteorological data are received in near real time from the GTS network for different hours within one day. The data are pre-processed and quality checked using the AMDAC software package (MeteoConsult, 1991) (see Appendix 6) which extracts, decodes and processes the GTS data. After decoding, the following data are checked for consistency and errors: air temperature, dew-point temperature (humidity), pressure at sea level, wind speed, amounts of precipitation, clouds, and sunshine duration. This error checking compares each observation with the corresponding values of the surrounding stations and compares that particular observation with observations at other times in the same day at the same station. Obvious errors in the observations are corrected automatically and a message is written to a log file; other errors are flagged for possible correction by an operator (Burrill and Vossen, 1992). Finally, the data are converted into daily values. This comprises the selection of minimum and maximum temperature, the aggregation of the rainfall, cloud cover and sunshine duration, the calculation of mean vapor pressure etc.
+
The stations, stored in object STATIONS holds over 10221 stations distributed over 40 countries in Europe and neighboring countries. Over 5100 of these stations provide weather data in near real time. All weather data is stored in the stations weather database (daily data in object WEATHER_OBS_STATION and 3- and 6-hourly precipitation in object WEATHER_OBS_STATION_RAIN).  
  
==Overview of available stations: collected data and data sources==
+
Raw station data is collected from various sources:
Globally in the MARS DB are present data referred to more than 6000 stations distributed in 48 countries, but of these, only one third present an adequate level reliability and regularity providing data. In the table 2.1 are reported the number of meteorological stations by country used in operational way in the MCYFS.
+
* {{Gloshint|GTS|WMO's Global Telecommunication System|GTS}} (essential data and data licensed by {{Gloshint|ECOMET|Economic interest grouping of the National Meteorological Services of the European Economic Area|ECOMET}} restrictions)
 +
* {{Gloshint|NOAA|National oceanic and atmospheric administartion|NOAA}} (USA)
 +
* European National Meteorological Institutes (NMI) (licensed)
 +
* Various regional networks in Europe
  
 +
For transmission and international exchange, the internationally exchanged station reports are encoded in formats standardized and maintained by {{Gloshint|WMO|World Meteorological Organization|WMO}} and [http://www.icao.int/ International Civil Aviation Organizaton (ICAO)]
  
[[Image:Meteo_station_network_density.jpg‎|thumb|250px|right|Spatial distribution of the meteo stations network density]]
+
* [https://www.wmo.int/pages/prog/www/DPS/Meetings/ET-DRC-KUALA-04/Doc5-1(1).doc SYNOP (WMO-code FM12)]
In general, the meteo stations density in the monitored areas is sufficient for the purpose of the project. In the Fig. 2.2 it is shown which is in average the surface covered by one station. Considering that each cell of the CGMS-grid is 50x50 km (equivalent to 2.500 km2), is evident that the main agricultural areas present at least one station for each grid cell or one station for a group of four cells (equivalent to 10.000 Km2).
+
* [https://library.wmo.int/doc_num.php?explnum_id=10722 BUFR (WMO-code FM94]
 +
* {{Gloshint|METAR|Meteorological Aviation Reports|METAR}}
  
The data are collected from various sources:
+
Observations as provided directly by National Meteorological Institutes or regional authorities come from secondary networks and are provided in proprietary formats.
*GBDS
 
*ECOMET
 
*USA-NOAA (including METAR).
 
  
Observations of maximum and minimum temperatures, precipitation amounts and sunshine duration (when available) are contained in the main hours synoptic. METAR data provide temperature, dew point, visibility and cloud amount. As far as available, they can be used for intermediate or even non-standard (i.e. all but main and intermediate) hours. From most countries outside Europe, 3-hourly synoptic data are exchanged world wide and can be made available through Meteo Consult.
+
Meteorological stations selected in priority are those located in the agricultural zones and equally distributed over the mainland (instead of over islands - for Portugal, Spain or Greece in particular). In particular, for western Russia (western of Urals) the main areas covered are the agricultural districts.
  
 +
In the case of China roughly 300 stations were selected meeting the following criteria:
 +
* Near real time delivery
 +
* A 20-years archive
 +
* Located in the main agricultural areas
 +
* Covering the elements: precipitation, minimum and maximum temperature, humidity and wind speed
  
Data from outside the ECOMET area are transmitted from the Royal Netherlands Meteorological Institute (KNMI) as if WMO essential. A number of countries in Europe, especially in the east, are aiming to become a member of ECOMET. This might lead to a reduction in the amount of data freely available.
+
The raw station data for China is collected from {{Gloshint|GTS|WMO's Global Telecommunication System|GTS}}.
  
{|class="collapsing_table collapsible collapsed"
+
====Basic indicators====
!'''Available number of meteorological stations by country'''
+
The basic indicators that are received from weather stations include:
 +
* Sum of precipitation
 +
* 2m air temperature
 +
* Maximum of 2m air temperature
 +
* Minimum of 2m air temperature
 +
* Downward directed solar radiation measured at earth's surface (global radiation)
 +
* Duration of sunshine
 +
* Total cloud cover
 +
* Water vapour pressure
 +
* Relative humidity
 +
* 10m mean wind speed
 +
* Snow depth
 +
 
 +
For WMO SYNOP FM 12 and BUFR FM 94 bulletins, WMO defines regional regulations to consider time zones and national coding practices. The extent of reported parameters and the report frequency differs per country and is for ECOMET member countries affected as well by license restrictions.
 +
 
 +
The METAR code is standardized through the ICAO. In Europe and China, the WMO-maintained codes SYNOP FM12 and BUFR FM 94 provide higher accuracy for the various parameters and more detail. In these regions, METAR provides only temperature, dew point, visibility, cloud amount and wind speed and is reported in coarser increments for the various parameters. The reporting frequency is determined by the individual flight operation of each airport. Nevertheless, METAR reports are used as well, mostly to fill spatial gaps in areas with less WMO stations.
 +
 
 +
Observation data from several European regional meteorological networks became available after 2015. The data from regional networks are mostly not available in the standard meteorological formats, but have to be collected and converted individually. The quality of the data is determined by the installed sensors and the siting of the stations. The reported parameters and frequency differ by network.
 +
 
 +
The following table summarizes basic information on the availability and reporting regulations from the various observing station data sources:
 +
 
 +
{|class="wikitable"
 +
!Parameter !! Reference periods of reports as defined by WMO !! WMO formats BUFR or SYNOP (*) !! METAR (**) !! Regional networks
 +
|-
 +
|Sum of precipitation || 24-hourly sum, 12-hourly sums, 6-hourly sum, 1-hourly sum reported, depending on region WMO-region and local regulations|| Europe: 06 UTC: past 24 hours / 00 UTC and 12 UTC: past 6 hours / 06 UTC and 18 UTC: past 12 hours / 1-hourly -- China: Reports 00 UTC for past 24 hours, some stations report 21 UTC for previous 24 hours (***) || Not reported in Europe and China || Individual, mostly 1-hourly
 +
|-
 +
|2 m air temperature || Instantaneous value || Reported with 0.1 K accuracy || Reported as full degrees || Mostly 0.1 K accuracy
 +
|-
 +
|Maximum 2 m air temperature || Maximum of continuous measurement during reference period (****) || Europe and China: reported 18 UTC || Not reported in Europe and China || Individual
 +
|-
 +
|Minimum 2 m air temperature || Minimum of continuous measurement during reference period (****) || Europe and China: reported 06 UTC || Not reported in Europe and China || Individual
 +
|-
 +
|Downward directed surface solar radiation (global radiation) || Sum accumulated over past 24 hours, sum past 1 hour || Available for some European countries at 00 UTC, 1-hourly || Not reported in Europe and China || Individual definition, mostly 1-hourly
 +
|-
 +
|Duration of sunshine || Sum accumulated over past 24 hours || Most European countries report at 06 UTC || Not reported in Europe and China || Individual, mostly 1-hourly
 +
|-
 +
|Total cloud cover || Instantaneous value || Octas 0-8 || 5 stages, only clouds up to a height of 5000 feet over ground reported || Not reported
 +
|-
 +
|Measures for the humidity of the air at 2 m above ground: dew point, water vapour pressure and relative humidity || Instantaneous value of dew point temperature reported (*****) || Reported with 0.1 K accuracy || Reported as full degrees || to be derived from other humidity parameters like relative humidity and air temperature
 +
|-
 +
|10 m mean wind speed || Mean over past 10 minutes || Meters per second || Mostly full knots, occasionally less accuracy during low wind situations || Individual definition
 +
|-
 +
|Snow depth || Instantaneous value, increasing automatization of measurement || When a station reports snow depth, it is done in Europe by 06 UTC, in China by 00 UTC || not reported || Not reported
 +
 
 +
|}
 +
<nowiki>(*) </nowiki>Main synoptic hours are 00, 06, 12, 18 UTC. Intermediate synoptic hours are 03, 09, 15, 21 UTC.
 +
For most European countries, 1-hourly data is used as well.<br>
 +
<nowiki>(**) </nowiki>The report frequency is determined by the airport's schedule and can be as often as 20 minutes. The frequency of reports can change over daytime, weekday, and season.<br>
 +
<nowiki>(***) </nowiki>In BUFR, several countries do not provide the reference period during dry conditions in the FM94 code, supposedly by accident. In this case, it is assumed that the WMO definitions for the reference period are applied.<br>
 +
<nowiki>(****) </nowiki>Europe: Covers past 12 hours. China: Covers past 24 hours. <br>
 +
<nowiki>(*****) </nowiki>Other thermodynamical measures for the humidity of air can be calculated from dew point and air temperature.<br>
 +
 
 +
==Data quality check==
 +
The software package [[Software Tools#QUACKME|Quality Checks Meteorological Data (QUACKME)]] as developed by the JRC is the main processing tool for completing and quality evaluation of actual meteorological data which is used as input for agro-meteorological models. The data processing workflow with quality control and aggregation can be described as follows.
 +
 
 +
====Near real-time pre-processing (1-hourly reports with extended information at intermediate and main synoptic hours, irregular reports)====
 +
* Near real-time collection of reports from the various data sources.
 +
* Decoding of the WMO and ICAO standard formats with dedicated decoder software (FMDecode). Reports in other formats from regional, secondary networks are translated into a uniform structure using individual proprietary converters.
 +
* The data is converted into a generalized structure, including the conversion towards UTC, standard units, the alignment of reference periods and the calculation of derived parameters. Basic sanity checks are applied.
 +
 
 +
====Preparation of QUACKME input data====
 +
* Generate a csv with all available observation data for the period of 24 hours (07 UTC - 06 UTC next day) for the European region and for China, respectively. The format of the csv is described in the {{PDFlink|[[media:Quackme_TechGuide_JRC128152_01.pdf|QUACKME Technical Guide]]}}
 +
* When data from a station are found to be erroneous for a longer time, the station can be listed on a so-called blacklist, either by parameter or for the whole report. Observations from blacklisted station-parameter combinations are not written into the csv. The blacklist is manually checked every three months.If the messages are considered trustworthy again, the station-parameter combination is removed from the blacklist. 
 +
* Generate csv with location specific, near real time forecasts for the same stations and period as the data in the observation-csv. The format of this csv is as well described in the QUACKME Technical Guide.
 +
For a number of weather elements, QUACKME compares the observed values with near-real-time forecast values. The forecast is used as reference for the reasonable range of possible values. The forecasts are obtained through a technique called MOS (Model Output Statistics). Meteorological forecast models, e.g. the ECMWF model, compute the physical status of the atmosphere on a grid, and the results represent the expected situation per grid box. The MOS forecast is using statistical relationships between the observations of a particular station and historic model forecasts for surrounding grid points. Each observing location has its own statistics. In this way, the local conditions at the weather station can be modelled much more accurately. QUACKME is using the individual location forecasts to define time- and location-dependent thresholds for the trustworthiness of station reports, for the elements air temperature (including minimum and maximum), dew point (applies to all derived measures for the humidity of the air), precipitation, and wind speed, respectively. That way, the thresholds consider season, climatology and even the actual weather pattern. A welcome side effect is the high spatial consistency of the statistical MOS approach and therefore of the thresholds. Individual MOS forecasts is used for almost all stations (approx. 5000, state January 2021).
 +
 
 +
====Running the QUACKME modules and interactive data quality checks by the meteorologist====
 +
* This does not apply for precipitation, i.e. for consecutive reports of 0 mm. This rather typical reporting bug is not found when quality checks are applied on to the data of the very day. Due to the mostly “patchy” pattern of precipitation events quality checks accept dry stations in between. To find stations that report consecutively 0 mm several weeks of history need to be considered, see [[#Retrospective checks and blacklisting of suspect stations |retrospective checks]].
 +
* Correct automatically obvious errors detected while performing these checks;
 +
* Automatically fill gaps in the database through interpolation based on time consistency criteria;
 +
* Flag dubious observations which cannot be corrected automatically;
 +
* Write all automatic corrections and flagged dubious observations to a log file;
 +
* Have the flagged observations checked and, if necessary, corrected by a trained meteorologist; when a correction is done, the derived parameters are recalculated and the data are written back to the database.
 +
 
 +
Dedicated trained and qualified meteorologists go through the dubious observation values that are flagged as such by the QUACKME automatic pre-checking program. An interactive system for the visualization of meteorological data is used to graphically visualize and analyze additional information such as:
 +
* Station observation data
 +
* Satellite images
 +
* Precipitation Radar data
 +
* Analysis and short range forecasts computed by physical models of the atmosphere
 +
* Short range forecasts for weather station locations
 +
This additional data is used by the analyst to decide on either confirmation or rejection of the observed values.
 +
 
 +
====Conversion to daily values====
 +
Once the database has been filled following the method described above, data are aggregated to daily values. This includes the indicators as summarized in the following table:
 +
 
 +
{|class="wikitable"
 +
!Parameter !! Aggregation !! Reference period Europe !! Reference period China
 +
|-
 +
|Total cloud cover (N) || Daily mean || 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day || 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
 +
|-
 +
|Duration of sunshine (Msun) || 24-hourly sum || 00–24 UTC || Not available
 +
|-
 +
|Downward directed surface solar radiation (global radiation) (Mrad) || 24-hourly sum || 00-24 UTC || Not available
 +
|-
 +
|Minimum 2m air temperature (Tn) || Lowest value of continuous reference period (*) || 18 previous day -06 UTC || 06 UTC previous day – 06 UTC
 +
|-
 +
|Maximum 2m air temperature (Tx) || Highest value of continuous reference period (**) || 06-18 UTC || 18 UTC previous day – 18 UTC
 +
|-
 +
|Water vapour pressure (e) || Daily mean || 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day || 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
 +
|-
 +
|10m mean wind speed (ff10) || Daily mean || 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day || 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
 +
|-
 +
|Sum of precipitation (RRR) || 24-hourly sum || Mostly 06 UTC until 06 UTC next morning  || Mostly 00 UTC – 00 UTC next day (indicator 2).<br>For some stations 21 UTC previous day – 21 UTC (indicator 6)
 +
|-
 +
|2m air temperature (TT) || 03-hourly instantaneous values during daytime || 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC || 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC
 +
|-
 +
|Relative humidity (RH) || 03-hourly instantaneous values during daytime || 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC || 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC
 +
|-
 +
|State of soil || Instantaneous value (***) || 00 UTC following day||
 +
|-
 +
|Water vapour pressure deficit (vpd) || Daily mean || 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day || 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
 +
|-
 +
|Slope of saturation vapour pressure vs. temperature curve slope || Daily mean || 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day || 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
 +
|-
 +
|Total cloud cover (N) || Daytime mean || 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC || 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC
 +
|-
 +
|Low or (when no low clouds) medium clouds (Nh) || Daytime mean || 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC || Not available
 +
|-
 +
|Calculated sunshine duration (Csun) || 24-hourly sum || To be calculated by QUACKME, 0-24 UTC of the day specified || To be calculated by QUACKME, 18 UTC previous day - 18 UTC of the day specified
 +
|-
 +
|Highest possible global radiation at clear sky (Crad) || 24-hourly sum || To be calculated by QUACKME,  0-24 UTC of the day specified || To be calculated by QUACKME, 18 UTC previous day - 18 UTC of the day specified
 +
|-
 +
|Potential evapotranspiration (ETP) || 24-hourly sum || To be calculated by QUACKME, 0-24 UTC of the day specified || To be calculated by QUACKME, 18 UTC previous day - 18 UTC of the day specified
 +
|-
 +
|Visibility (VV) || Daytime mean || 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC || 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC
 
|-
 
|-
|
+
|Snow depth || Instantaneous value || 06 UTC || 00 UTC
{|class="wikitable"
 
!Code !!Country !!JRC !!B6 !!B3 !! R !!T !! E !! M !! P
 
|-
 
|AL||Albania||-||-||-||-||-||-||-||-
 
|-
 
|AM||Armenia||-||3||-||-||1||-||1||-
 
|-
 
|AT||Austria||-||15||15||93||93||21||6||-
 
|-
 
|ETC||Etcetera||-||-||-||-||-||-||-||-
 
|}
 
Where:
 
* JRC = JRC aimed number of stations
 
* B6 =  number of stations in GBDS (main hour observations)
 
* B3 = number of stations in GBDS (main and intermediate hour observations)
 
* R = mean number of stations with precipitation observation available
 
* T = mean number of stations with minimum and maximum temperature observation available
 
* E = maximum number of stations available through ECOMET according to ECOMET catalogue except for Greece where catalogue is wrong (main and intermediate hours, including GBDS stations)
 
* M = number of METAR stations available (partly overlapping GBDS and ECOMET stations)
 
* P = price per year for each additional station not in Global Basic Data Set (based on 6 synops per day)
 
{|class="wikitable"
 
|+Words and expressions in the next sections have the following meanings
 
!Term
 
!Description
 
|-
 
|Main hours || 00, 06, 12 and 18 UTC
 
|-
 
|Intermediate hours || 03, 09, 15 and 21 UTC
 
|-
 
|UTC || Universal Time Co-ordinate (also referred to as GMT, Greenwich Mean Time)
 
|-
 
|SYNOP || Observations in special code, for most observational stations made 24 times per day. For a very limited number of stations, the synops (mostly for the main hours only) are distributed freely over the GTS (Global Telecommunication System).
 
|-
 
|METAR || Observations made especially for airports. The observations can be obtained through ICAO (International Civil Aviation Organization). The advantage is that the information generally is available hourly, but the disadvantage is that it is more global in its elements. E.g. no maximum and minimum temperatures are coded nor precipitation amounts.
 
|-
 
|GBDS || Global Basic Data Set; the set of stations with freely available data (also referred to as WMO essential)
 
|-
 
|ECOMET || Consortium of National Meteorological Services in Europe that facilitates the selling of data to private sector companies
 
|}
 
 
|}
 
|}
 +
<nowiki>(*)</nowiki>When no minimum is reported but hourly instantaneous temperatures QUACKME estimates the minimum from the hourly local early morning values, see {{PDFlink|[[media:Quackme_TechGuide_JRC128152_01.pdf|QUACKME Technical Guide]]}}<br>
 +
<nowiki>(**)</nowiki>When no maximum is reported but hourly instantaneous temperatures QUACKME estimates the maximum from the hourly local afternoon values, see {{PDFlink|[[media:Quackme_TechGuide_JRC128152_01.pdf|QUACKME Technical Guide]]}}<br>
 +
<nowiki>(***)</nowiki>Code, for translation see [https://library.wmo.int/doc_num.php?explnum_id=10722 BUFR documentation.]
 +
 +
Information on the way the daily element values are constructed/defined is stored in the object WEATHER_OBS_STATION_INFO. Currently this is only done for precipitation e.g. period definition of the daily rainfall sum. Codes are:
 +
* 0 = real observation 06 - 06 UTC next day
 +
* 1 = period 06 - 06 UTC next day, short range forecast has been used to cover the complete period
 +
* 2 = real observation 00 UTC - 24 UTC
 +
* 3 = real observation 03 UTC - 03 UTC next day
 +
* 4 = real observation 12 UTC previous day - 12 UTC
 +
* 5 = real observation 18 UTC previous day - 18 UTC
 +
* 6 = real observation 21 UTC previous day - 21 UTC
  
Meteorological stations selected in priority are those located in the agricultural zones and equally distributed over the mainland (instead of over islands - for Portugal, Spain or Greece in particular). In particular, for western Russia (western of Urals) the main areas covered are the agricultural districts. Since 1 March 2004, a renewed station list has become operational. The most important changes are:
+
More information on the 3- and 6-hourly precipitation data are stored in object WEATHER_OBS_STATION_RAIN (column IDFLAG). Codes are:
 +
* 1 = Changed by meteorologist (not applicable)
 +
* 2 = Automaticaly corrected (not applicable)
 +
* 3 = Observation
 +
* 4 = Linear interpolation from observations
 +
* 5 = Interpolated via MOS from observations
 +
* 6 = MOS analyses (not managed yet)
  
* For some countries (Austria, Belgium, Germany, Netherlands, Norway, Switzerland and United Kingdom) a fixed set of stations was used in the past. All available relevant station data from these countries are now sent to JRC. The reason for this is that sometimes stations are closed or the number of observed elements or observation hours is decreased. Increasing the total number of stations should ensure that the total required number is always reached.
+
Finally, meta data of all stations is checked once a year.
* For countries where the aimed number of stations is or was not met, efforts have been undertaken to provide the maximum number available. Successful examples are Estonia, Spain, Turkey and Ukraine (see also below). The number for Morocco has also been increased.
 
* In the case of Spain, there are not enough synoptic stations to meet the aimed number. Data from additional stations were provided to meet the demand. However, these stations only report maximum and minimum temperature and precipitation amounts.
 
* For Turkey, additional station observations have become available through ECOMET. The JRC criteria are met by purchasing these data.
 
* In the case of Ukraine, many more stations have become available as a result of extensive negotiations which took a rather long period of time. The average number of available stations is now about 170. Moreover, observations are now present at 3-hourly time intervals.
 
* A special case is Portugal: the same set of stations is delivered as was usual during the last few years. However, we now only count the continental stations and not the island stations anymore. This results in a slight deficiency of stations for Portugal which cannot be solved, since there are no more stations at all.
 
* Improvements have not yet been achieved for Albania, Belarus, Latvia, Lithuania, Moldavia, Morocco (see above) and Romania. It isn’t possible to expect to meet the required number of stations for these countries on a short term. Either the station networks in these countries have been degraded or the National Weather Services are very difficult to contact or do business with.
 
  
==Quality check of parameters==
+
====Retrospective checks and blacklisting of suspect stations====
For data quality check a specific software named AMDAC has been developed. The software performs the following actions:
+
Some suspicious station reports are only detectable by checking time series of several weeks. Continuous reports of 0 mm precipitation (instead of a "precipitation not observed" flag) do not stick out in daily rainfall sums, but only by investigating the station's reports over a longer period. Global radiation and cloud cover have a high spatial volatility, and continuous observation or encoding errors at a certain station become more explicit when looking into several weeks of station reports.
* Decode intermediate-hour and main-hour SYNOP reports and METAR reports from weather stations circulating on the GTS in the geographic zone of interest (defined above);
 
* Check the quality and correct the obvious errors in the received weather reports;
 
* Perform time consistency checks to compare the values of reported parameters with those previously or subsequently reported for the same station;
 
* Correct automatically obvious errors detected while performing consistency checks;
 
* Fill up automatically gaps in the database through interpolation based on time consistency criteria;
 
* Flag errors and dubious observations which can not automatically be corrected, and write these to a log file;
 
  
A description of AMDAC is available in documents which can be downloaded form the MARS ftp site (see Appendix 6). The availability check for station data will be executed on daily basis. The consistency checks to be performed on the data of a certain day imply that observations from 18 UTC of the previous day until 12 UTC of the next day are available in order to compute an interpolation in time if missing values occurred. The values of the following observation elements are checked: air temperature, dew-point temperature, pressure at sea level, wind speed, amount of precipitation, amount of clouds, duration of sunshine.
+
For all European stations, the QUACKME output of the past 40 days is inspected each week through time series checks. Provided a station reported on more than half of the tested days, the reports are checked, consulting ECMWF model analysis and short range forecasts for model grid points surrounding the station of request.  
  
 +
The checks are set up as follows:
 +
=====Precipitation=====
 +
A station is flagged as suspicious for precipitation when suspicious consecutive zero rainfall reports are detection. Criteria are
 +
* The observed precipitation sum is 0 mm whan aggregated over the whole checked period.
 +
AND
 +
* the ECMWF model near real time forecasts during the checked period included at least 10 wet days. A day is considered being wet when more than 0.5 mm precipitation is forecasted by the model's near real time foreast.
  
For detailed description of the consistency checks to be performed, refer to Appendix 6. Obvious errors in the observations are automatically corrected and a message is written to a log file. The operator who has the possibility of modifying the data can read these latter messages. After the observations of a station are checked (and if necessary corrected) the derived parameters are recalculated and the data are written back to the database.
+
=====Radiation=====
 +
For each day of the investigated period, the station’s maximum possible daily solar radiation sum is calculated, based on its latitude, the time of year, and using a standard atmospheric optical depth.
 +
A station is flagged as being suspicious for radiation when:
 +
* There are at least 10 days with observed solar radiation exceeding 110 % of the maximum possible amount of solar radiation.
 +
OR
 +
* There are at least 10 days on which the observed solar radiation remained below 10 % of the maximum possible daily sum of solar radiation.
 +
OR
 +
* There are at least 10 days with observed radiation of 0 MJ m-2 day-1 whilst the ECMWF short-range forecast analysed solar radiation exceeding 0 MJ m-2 day-1.
 +
OR
 +
* The total sum of observed solar radiation is less than 25 % of the maximum possible radiation sum for the period, whilst the sum of the model's short-range forecasts for the parameter exceeded 25 % of the maximum possible daily sum of solar radiation. Naturally, the maximum possible radiation period's is only summed up from days with observations being available.
  
 +
=====Mean daytime cloudiness=====
 +
A station is flagged as being suspicious for cloudiness when:
 +
* A difference of more than 2.5 octa between the daily mean of observed total cloud cover and the daily mean of ECMWF model analysis and short-range forecast for total cloud is found for all days of the investigated period.
 +
OR
 +
* The reported instantaneous cloudiness was always higher than 4.0 octa whilst the model analysis and short-range forecasted for at least three time steps (hours) in the period a total cloud cover of less then 3.0 octa.
 +
OR
 +
* For all time steps in the period more than 5 octa total cloud cover was reported.
  
Once the database has been filled using the previous module, a final check is performed on the daily file before store in Data Base. This automated quality check consists in verifying the following conditions:
+
=====Duration of sunshine=====
 +
For each day in the investigated period, the maximum day length is calculated based on the day of the year and the station latitude. Dividing the observed sunshine duration for a day by the calculated day length gives the relative sunshine. <br>
 +
A station is flagged as being suspicious for sunshine duration when:
 +
* For more than 10 days in the period, the observed duration of sunshine is more than 110% of the calculated day length.
 +
OR
 +
Depending on the dominant season during the period:
 +
* Summer: highest relative sunshine value is less than 30% (i.e. the station is always cloudy).
 +
* Winter: the lowest relative sunshine value is more than 70% (i.e. the station is always sunny)
 +
* Spring/autumn: highest relative sunshine value is less than 30% (see summer check) OR the lowest relative sunshine value is more than 70% (see winter check).
  
 +
The dominant season is determined as the season with the largest number of days in the investigated period. When 50% of the investigated days are winter/summer days, the dominant season will be winter/summer.
 +
 +
<br>
 +
When the process flags stations as suspect a final manual inspection by a meteorologist follows. If the time series of the station are found to be wrong the following actions are executed:
 +
* The station is added to a blacklist: the station is immediately excluded from the operational station list.
 +
* The erroneous time series are deleted from the objects WEATHER_OBS_STATION_RAIN and WEATHER_OBS_STATION. The erroneous values are flagged (object WEATHER_OBS_STATION_RAIN, column TYPE) or deleted (object WEATHER_OBS_STATION and WEATHER_OBS_STATION_INFO) and deleted values are saved in separate objects (WEATHER_OBS_STATION_ERRORS and WEATHER_OBS_STATION_INFO_ERR).
 +
* All affected grid cells (object WEATHER_OBS_GRID) and regions (object WEATHER_OBS_REGIONCOVER) are reprocessed at regular time intervals. This also includes the crop simulation results.
 +
 +
Every three months, by the end of the quarter, each station on the blacklist is verified. Afterwards it is decided if stations can return to the operational work flow. Falsely blocked data is back-ordered, added and reprocessed.
 +
 +
====Station data availability====
 +
Each month an overview is created showing the delivered number of stations per country. Information is also added on sudden changes and follow-up actions.
 +
[[Media:MonthlyDeliveryReport.PNG|Example monthly overview]]
 +
 +
Every day, the newly produced data files are compared with those of the previous day. If the number of delivered values for individual countries and parameters decreases significantly, an alert is sent by email is sent to the project team. The threshold above which a decline in the number of values delivered is considered critical depends exponentially on the number of values in the country.
 +
 +
The number of values flagged by the weak, heavy and threshold checks of QUACKME are monitored on a daily and on a monthly basis. Stations that are flagged particularly frequently are identified and the cause can be analysed separately. As a result, stations can be blacklisted or an improvement of the QUACKME checks can be suggested.
 +
 +
The following maps illustrate the available stations (red 0-20% - green 80-100%) for the main elements in a recent year 2019. The main elements (maximum temperature, minimum temperature, precipitation, sun shine, cloud cover, wind speed and vapor pressure) have a good spatial spread over Europe with a relative high spatial density in western and central Europe. Availability of measured radiation is mainly limited to western and central Europe.
 +
 +
{|class="wikitable" 
 +
|-
 +
|[[File:Stations tmax 2020.jpg|350px|left]]||[[File:Stations tmin 2020.jpg|350px|left]]||[[File:Stations rain 2020.jpg|350px|left]]
 +
|-
 +
|maximum temperature||minimum temperature||precipitation
 +
|-
 +
|[[File:Stations radiation 2020.jpg|350px|left]]||[[File:Stations sunshine 2020.jpg|350px|left]]||[[File:Stations cloud 2020.jpg|350px|left]]
 +
|-
 +
|global radiation||sunshine||cloud cover
 +
|-
 +
|[[File:Stations wind 2020.jpg|350px|left]]||[[File:Stations vapour 2020.jpg|350px|left]]||[[File:Stations snow 2020.jpg|350px|left]]
 +
|-
 +
|wind speed 10m||vapor pressure||snow depth
 +
|}
 +
 +
The following graph shows the increase of observations for the main elements between 1975 and 2019. Most elements have at least 600,000 annual observations which equals over more than 1600 stations in case they would have a complete temporal coverage. However, most stations have temporal gaps and therefore the number of reporting stations is much higher. Since 2004 the number of observations increased up-till a level of around 1,500,000 reported by more than 4500 stations. During the recent years also observations of radiation related elements increased drastically. This is especially true for cloud cover and sunshine. Prior to 1995 these elements have a relative low number of observations meaning that the global radiation of these years, required in MCYFS, is mainly based on the daily temperature range, see [[#Calculation of advanced parameters |Calculation of advanced parameters]].
 +
 +
[[File:Availability over time 2020.JPG|800px|none]]
 +
 +
 +
In general the station density and available data in the monitored areas is considered sufficiently high for the purpose of the project.
 +
 +
==Ingestion into the database==
 +
After the station weather data passed all checks, daily weather data is exported to a fixed formatted ASCII file (s-file) containing the data of a single day that can be imported in the object WEATHER_OBS_STATION. In the near real time situation a s-file is delivered one day later. For example in the afternoon of day 31 March 2016 the following file is generated: s20160330.dat.
 +
 +
{|class="collapsing_table collapsible collapsed"
 +
!Format ASCII s*.dat file (daily station weather)
 +
|-
 +
|
 
{|class="wikitable"
 
{|class="wikitable"
!Parameter
+
! ELEMENT !! POSITION TEXT FILE !! DESCRIPTION !! UNIT !! AGGREGATION_PERIOD/REFERENCE PERIOD
!Constraint
+
|-
 +
| STATION_NUMBER||position(01:07)||station number||-||-
 +
|-
 +
| DAY||position(08:17)||date as YYYYmmdd||-||-
 +
|-
 +
| CLOUD_24_TOTAL||position(18:23)||daily mean of total cloud cover||octas|| mean over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| SUNSHINE||position(24:29)||sunshine duration||h|| sum of 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| RAD_MEA||position(30:35)||measured global radiation||MJ m-2 day-1|| sum of 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| TEMP_MIN||position(36:41)||minimum temperature||0C|| minimum of 12 hours in Europe, 24 hours in China, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| TEMP_MAX||position(42:47)||maximum temperature||0C|| maxiumum of 12 hours in Europe, 24 hours in China, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| VAP_PRES||position(48:53)||daily mean vapour pressure||hPa|| mean over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 
|-
 
|-
|Daily mean of total cloud cover : N || 0 to 8 octas
+
| WIND_10||position(54:59)||daily mean wind speed at 10 metres||m.s-1|| mean over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 
|-
 
|-
|Measured sunshine duration: Msun || 0 to 24 hours
+
| RAIN||position(60:65)||amount of precipitation||mm.d-1|| sum over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 
|-
 
|-
|Minimum temperature: Tn || -15 to 25°C
+
| TEMP_06||position(66:71)||air temperature morning||degC||06 UTC in Europe, 00 UTC in China
 
|-
 
|-
|Maximum temperature: Tx || 0 to 40°C
+
| HUM_06||position(72:77)||relative humidity morning||%||06 UTC in Europe, 00 UTC in China
 
|-
 
|-
|Maximum temperature - Minimum temperature || 0< Tx-Tn <30°C
+
| TEMP_09||position(78:83)||air temperature late morning||degC||09 UTC in Europe, 03 UTC in China
 
|-
 
|-
|Daily mean vapor pressure: e || 0 to 30 hPa
+
| HUM_09||position(84:89)||relative humidity late morning||%||09 UTC in Europe, 03 UTC in China
 
|-
 
|-
|Daily mean wind speed at 10 metres: ff10 || 0 to 15 m/s
+
| TEMP_12||position(90:95)||air temperature midday||degC||12 UTC in Europe, 06 UTC in China
 
|-
 
|-
|Amount of precipitation from 6 UTC-6 UTC: RRR || 0 to 75 mm
+
| HUM_12||position(96:101)||relative humidity midday||%||12 UTC in Europe, 06 UTC in China
 
|-
 
|-
|Air temperature: TT || -15 to 40°C
+
| TEMP_15||position(102:107)||air temperature afternoon||degC||15 UTC in Europe, 09 UTC in China
 
|-
 
|-
|Relative humidity: RH || 20 to 100%
+
| HUM_15||position(108:113)||relative humidity afternoon||%||15 UTC in Europe, 09 UTC in China
 
|-
 
|-
|Daily mean vapor pressure deficit: vpd || 0 to 40 hPa
+
| TEMP_18||position(114:119)||air temperature evening||degC||18 UTC in Europe, 12 UTC in China
 
|-
 
|-
|Daily mean slope of saturation vapor pressure vs. temperature curve: slope || 0 to 3 hPa/°C
+
| HUM_18||position(120:125)||relative humidity evening||%||18 UTC in Europe, 12 UTC in China
 
|-
 
|-
|Daytime mean of total cloud cover: N || 0 to 8 octas
+
| STATE_SOIL||position(126:131)||state of the soil||code*||instanteneous value, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| VAP_PRES_Def||position(132:137)||daily mean vapour pressure deficit||hPa|| mean over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| SLOPE_VP_VS_T||position(138:143)||daily mean slope saturation vapour pressure vs. temperature curve||hPa/°C|| mean over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| CLOUD_DAYTIME_TOTAL||position(144:149)||daytime mean of total cloud cover||octas|| mean over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| CLOUD_DAYTIME_LOW||position(150:155)||daytime mean amount of CL clouds or, if no CL clouds are present, the daytime mean amount of CM clouds||octas|| mean over daytime hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| CLOUD_SHADOW||position(156:161)||daytime mean amount of shadow clouds||octas|| mean over daytime hours
 +
|-
 +
| no name**||position(162:167)||Calculated sunshine duration Csun||%|| sum over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| no name**||position(168:173)||Highest possible global radiation at clear sky Crad||MJ m-2 day-1|| sum over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| no name**||position(174:179)||Potential evapotranspiration ETP||mm/day|| sum over 24 hours, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| VISIBILITY||position(180:185)||daytime mean visibility||kilometres|| daytime mean, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| SNOW_DEPTH||position(186:191)||snow depth||cm|| instanteneous value, see [[#Conversion to daily values |Conversion to daily values]]
 +
|-
 +
| RAIN_MOS||position(192:197)||indicator for reference period and usage of short range forecast in daily sum of precipitation RAIN||-||0 = real observation 06 - 06 UTC next day, 1 = period 06 - 06 UTC next day, short range forecast has been used to cover the complete period, 2 = real observation 00 UTC - 24 UTC, 3 = real observation 03 UTC - 03 UTC next day, 4 = real observation 12 UTC previous day - 12 UTC, 5 = real observation 18 UTC previous day - 18 UTC, 6 = real observation 21 UTC previous day - 21 UTC)
 
|-
 
|-
|Penman evaporation: ETP || 0 to 10 mm/day
 
 
|}
 
|}
 +
<nowiki>*</nowiki> Codes for state of soil: 0 = surface of ground dry, without cracks or appreciable amount of dust or loose sand, 1 = surface of ground moist<br>3 = flooded, 4 = frozen, 5 = glaze on ground, 6 = loose dry dust or sand not covering the ground completely, 7 = thin cover of loose dry dust or sand covering the ground completely, 8 = moderate or thick cover of loose dry dust or sand covering the ground completely, 9 = extremely dry with cracks. <br>All codes refer to a surface of ground without snow or measurable ice cover.<br>
 +
<nowiki>**</nowiki> Not imported into the MCYFS database
 +
|}
 +
<br>
 +
The 3-hourly rainfall data is exported to a plain ASCII file (rrr3h_*.txt file) containing the data of one 3-hourly time step within one single day. This data can be imported in the object WEATHER_OBS_STATION_RAIN. In the near real time service each day 8 rrr3h_*.txt files are generated at once containing data of 8 3-hourly time steps:
 +
* 09 UTC (06-09 UTC of previous day)
 +
* 12 UTC (09-12 UTC of previous day)
 +
* 15 UTC (12-15 UTC of previous day)
 +
* 18 UTC (15-18 UTC of previous day)
 +
* 21 UTC (18-21 UTC of previous day)
 +
* 00 UTC (21-00 UTC of previous day)
 +
* 03 UTC (00-03 UTC of present day)
 +
* 06 UTC (03-06 UTC of present day)
 +
 +
For example in the afternoon of day 31 March 2016 the following files are generated: rrr3h_2016033009.txt, rrr3h_2016033012.txt, rrr3h_2016033015.txt, rrr3h_2016033018.txt, rrr3h_2016033021.txt, rrr3h_2016033100.txt, rrr3h_2016033103.txt and rrr3h_2016033106.txt.
 +
 +
{|class="collapsing_table collapsible collapsed"
 +
!Format ASCII rrr3h_*.txt file (3-hourly station rainfall)
 +
|-
 +
|
 +
{|class="wikitable"
 +
! Column header
 +
|-
 +
| WMO number
 +
|-
 +
| Date / time (UTC)
 +
|-
 +
| Precipitation (mm/3h)
 +
|-
 +
| Flag (1=Changed by meteorologist/2=Automatically corrected/3=Observation/4=Linear interpolation from observations/5=Interpolated via MOS from observations/6=MOS analyses)
 +
|-
 +
|}
 +
|}
 +
 +
<br>
 +
The 6-hourly rainfall data is exported to a plain ASCII file (rrr_*.txt file) containing the data of one 6-hourly time step within one single day. This data can be imported in the object WEATHER_OBS_STATION_RAIN. In the near real time service each day 4 rrr_*.txt files are generated at once containing data of 4 6-hourly time steps:
 +
* 12 UTC (06-12 UTC of previous day)
 +
* 18 UTC (12-18 UTC of previous day)
 +
* 00 UTC (18-00 UTC of previous day)
 +
* 06 UTC (00-06 UTC of present day)
 +
 +
For example in the afternoon of day 31 March 2016 the following files are generated: rrr_2016033012.txt, rrr_2016033018.txt, rrr_2016033100.txt and rrr_2016033106.txt.
 +
 +
{|class="collapsing_table collapsible collapsed"
 +
!Format ASCII rrr_*.txt file (6-hourly station rainfall)
 +
|-
 +
|
 +
{|class="wikitable"
 +
! Column header
 +
|-
 +
| WMO number
 +
|-
 +
| Date / time (UTC)
 +
|-
 +
| Precipitation (mm/6h)
 +
|-
 +
| Flag (1=Changed by meteorologist/2=Automatically corrected/3=Observation/4=Linear interpolation from observations/5=Interpolated via MOS from observations/6=MOS analyses)
 +
|-
 +
|}
 +
|}
 +
 +
<br>
 +
 +
==Calculation of advanced parameters==
 +
===Global radiation===
 +
Global radiation is the daily sum of incoming solar radiation that reaches the earth surface. It is mainly composed of wavelengths between 0.3 μm and 3 μm. Approximately half of the incoming radiation with wavelengths between 0.4 and 0.7 μm is Photosynthetically Active Radiation (PAR). Global radiation is the driving variable in the growth-determining CO2 assimilation process and thus crop growth models are sensitive to radiation data ([[References|van Diepen, 1992]]).
 +
 +
A major problem is the scarcity of measured global radiation. In cases where no direct observations are available it must be derived from sunshine duration, cloud cover and/or temperature, on the basis of statistical relationships. If measured global radiation is missing, it is based on one of three formulae (Ångström-Prescott, Supit-Van Kappel, and Hargreaves), depending on the availability of meteorological parameters. An important component in these formulae is the amount of Angot radiation which is the extraterrestrial radiation integrated over the day at certain latitude on a certain day. The calculation of the Angot radiation and the three different formulae are described by [[References|Supit et al. (1994)]] and [[References|van der Goot (1998a)]].
 +
 +
===Angot radiation===
 +
The principle component of all three formulae is the extraterrestrial radiation, or Angot radiation. In fact, all of the three formulae estimate the fraction of Angot radiation actually received at the earth surface. The Angot radiation is calculated as:
 +
 +
{{Hidden
 +
|[[File:angot_radiation.jpg|Angot radiation]]
 +
|where:
 +
* ''Ra        : Daily extra-terrestrial radiation, Angot radiation [J*m-2*d-1]''
 +
* ''Sc,d      : Solar constant at the top of the atmosphere for a certain day [J*m-2*s-1]''
 +
* ''∫sinβ dth : integral of solar height over the day [s]''
 +
 +
The solar constant at the top of the atmosphere is calculated as:
 +
 +
[[File:solar_constant.jpg|Solar constant]]
 +
 +
where:
 +
* ''Sc,d : Solar constant at the top of the atmosphere for a certain day [J*m-2*s-1]''
 +
* ''Sc : Average solar radiation at the top of atmosphere [J*m-2*s-1] (1370 J*m-2*s-1; I.E.A., 1978)''
 +
* ''td : number of day (January 1 equal to 1) [-]''
 +
 +
The integral of the solar height over the day is a function of both the latitude of the position being considered as well as the day of the year. The solar declination angle is a function of the day of the year, and is calculated as follows:
 +
 +
[[File:solar_declination.jpg|Solar declination]]
 +
 +
Where:
 +
* ''δ : Solar declination [radians]''
 +
* ''td : number of day (January 1 equal to 1) [-]''
 +
 +
For a given latitude, the necessary calculations now concern the calculation of the astronomical day length, and the integral of the solar height. The astronomical day-length is calculated as follows:
 +
 +
[[File:astronomical_day_length.jpg|Astronomical day-length]]
 +
 +
where:
 +
* ''D   : day-length [h]''
 +
* ''sinLD : sin(δ) * sin(latitude* π/180) [-]''
 +
* ''cosLD : cos(δ) * cos(latitude* π/180) [-]''
 +
 +
The integral of the solar height over the day can be calculated as:
 +
 +
[[File:solar_height.jpg|Integral of solar height over the day]]
 +
 +
Where:
 +
* ''∫sinβ dth : integral of solar height over the day [s]''
 +
* ''D       : daylength [h]''
 +
* ''sinLD    : sin(δ) * sin(latitude* π/180) [-]''
 +
* ''cosLD    : cos(δ) * cos(latitude* π/180) [-]''
 +
 +
For very high latitudes (>67°N), for a certain number of days per year, the day length can be 24 hours. In this case the above formulae no longer apply. The program checks for the value of sinLD/cosLD, and in case this value exceeds 1.0, the day length is set to 24 hours and the integral set to 24*3600 seconds.
 +
}}
 +
 +
The following hierarchical method is used to calculate global radiation for each station ([[References|Supit and van Kappel, 1998]]) in case measured global radiation is missing:
 +
 +
===Ångström-Prescott formula===
 +
If sunshine duration is available, global radiation is calculated using the equation postulated by Ångström (1924) and modified by [[References|Prescott (1940)]]. The two constants in this equation depend on the geographic location.
 +
 +
{{Hidden
 +
|[[File:angstrom_formula.jpg|Ångström-Prescott formula]]
 +
|where:
 +
* ''Rg : global radiation [ J m-2 d-1]''
 +
* ''Ra : Angot radiation [ J m-2 d-1]''
 +
* ''n : bright sunshine hours per day [h]''
 +
* ''L : astronomical day length [h]''
 +
* ''Aa, Ba : regression coefficients (Ångström-Prescott) [-]''
 +
}}
 +
 +
===Supit-Van Kappel formula===
 +
When neither measured radiation nor sunshine duration are available, but minimum and maximum temperature and daytime cloud cover are known, the Supit-Van Kappel formula is used. This is an extension of the Hargreaves formula ([[References|Supit, 1994]]). The regression coefficients depend on the geographic location.
 +
 +
{{Hidden
 +
|[[File:supit_formula.jpg|Supit-Van Kappel formula]]
 +
|where:
 +
* ''Tmin, Tmax : minimum and maximum daily temperature [°C]''
 +
* ''CC : cloud cover in octets [-]''
 +
* ''As, Bs : regression coefficients (Supit) [-]''
 +
* ''Cs : regression coefficient (Supit-Van Kappel) [J m-2 d-1]''
 +
}}
 +
 +
===Hargreaves formula===
 +
When only the minimum and maximum temperatures are known the equation of [[References|Hargreaves et al. (1985)]] is used. The regression coefficients depend on the geographic location.
 +
 +
{{Hidden
 +
|[[File:hargreaves_formula.jpg|Hargreaves formula]]
 +
|where:
 +
* ''Ah : regression coefficient (Hargreaves) [-]''
 +
* ''Bh : regression coefficient (Hargreaves) [J m-2 d-1]''
 +
}}
 +
 +
Any one of the above three methods has an additional upper limit. The maximum calculated global radiation is set to Angot radiation, corrected for atmospheric transmissivity, by multiplying the Angot value with the sum of the Angstrom A and B coefficients.
 +
 +
===Deriving Ångström-Prescott, Supit-Van Kappel, and Hargreaves regression constants===
 +
The main problem with the application of the Ångström-Prescott, Supit-Van Kappel, and Hargreaves formulae is the quality of the regression constants. Studies by [[References|Supit (1994)]], [[References|Supit and van Kappel (1998)]] and [[References|van Kappel and Supit (1998)]] showed no relationship between latitude and the coefficients for Europe, although such a relation is frequently used to estimate these regression constants. Initially in MCYFS regression constants of [[References|Supit and van Kappel (1998)]] and [[References|van Kappel and Supit (1998)]] for Europe were used. They obtained sets of regression constants for the formulae for as many weather stations as possible, with a geographic distribution that corresponds to the area of interest for the MCYFS. As a result, a set of 256 reference stations was identified for which a relevant set of measured radiation data and other parameters in the formulae existed. For these stations regression constants were calculated based on measured radiation data for the three formulae mentioned above.
 +
 +
In 2012 the regression coefficients of these solar radiation models for Europe were updated using a new set of weather station data (temperature, sunshine and cloudcover) and an alternative training data set: 6 years (2005-2010) of the down-welling surface shortwave radiation flux (DSSF) 30-minutes product derived from Meteosat Second Generation satellite data by the Land Surface Analysis Satellite Applications Facility (LSA SAF) ([[References|Bojanowski et al.,2013]]). For each solar radiation model a set of weather stations was selected having sufficient observations of either sunshine duration, or cloud cover/temperature or only temperature (minimum and maximum) to perform a regression analysis. Results are stored in object STATION_REFERENCE_COEFFICIENTS (CGMS14SYS).
 +
 +
Station archive data for China did not include measured radiation nor sunshine. Therefore radiation was derived from other observed elements namely cloud cover and minimum and maximum temperature. The Hargreaves and Supit-VanKappel models have been trained using modelled radiation by [[References|Tang et al., 2013]]. The 50yrRad database of [[References|Tang et al., 2013]] containing ‘modelled’ radiation data for 716 CMA stations, has demonstrated its superior performance over previous estimates of locally calibrated Angstrom-Prescott models. While radiation is based on the Hargreaves or Supit-VanKappel models, coefficients of the Angstrom method are still required to calculate net outgoing long wave radiation within the potential evapotranspiration calculation. For determining Angstrom coefficients only the 50yrRad archive was used. Since no sunshine duration data is available, an alternative was sought. Transmissivity was derived by dividing the measured solar radiation at the ground by the solar radiation at the top of the atmosphere. By selecting only the period between day of year 150 and 200 (during mid-summer) the transmissivity is almost constant and can be linked to the Angstrom coefficients.
 +
 +
The program [[SupitConstants]] uses this set of data (via the view SUPIT_REFERENCE_STATIONS, CGMS14SYS), consisting of latitude, longitude, altitude and calculated regression constants, to derive the regression constants for all stations in the MCYFS. Interpolation of the regression constants of the reference stations to other stations is based on a distance weighted average of the three nearest stations. This process is carried out once, unless the set of reference stations changes or when new stations are added or when meta data of stations change.
 +
 +
{{Hidden|Interpolation of regression constants
 +
| Data of the reference stations, consisting of latitude, longitude, altitude and the regression constants, is being used for the derivation of the regression constants for the set of stations used for the interpolation of the daily meteorological data. This is a process that only has to be carried out once, unless the set of reference stations changes or when new stations are added or when meta data of stations change. Once the regression constants have been established for the operational set of stations, the global radiation estimation can proceed using any one of the formulae.
 +
 +
The interpolation of the regression constants is based on a simple distance weighted average of the three nearest stations. For each of the three sets of constants (Ångström-Prescott, Supit-Van Kappel, and Hargreaves) a subset is created from the complete set of reference stations, by selecting only those stations that have the regression coefficients for the desired method. This subset of stations is then sorted based on distance to the station for which the regression coefficients are being calculated. This sorting process is also subject to an altitude threshold test i.e. if the altitude difference between the target station and a reference station is greater than a set threshold the reference station is rejected in favour of the next nearest reference station. Depending on a distance threshold, the nearest one, two or three stations are then used to calculate the regression constants. If the threshold tests exclude all stations, the nearest station will be used, regardless of the distance. The altitude threshold value is 200 m; the distance threshold is 200 km.
 +
 +
The distance weighted average method used, is based on the relative distance of the reference stations to the station of interest.
 +
 +
<nowiki>Assume the distances d0, d1 and d2 to be the distances to the three nearest reference stations, and w0, w1 and w2 the weights to be used in the calculation. As an example, assume that d1 is 2*d0, then w1 will be w0/2. More general, w1 = w0*d0/d1. Similarly, w2 = w0*d0/d2. Furthermore, the sum of the weights should be 1, so w0+w1+w2 = 1. From the above, the following relation can be established:</nowiki>
 +
 +
* '''w0 equals d1d2 / (d0d1 + d0d2 + d1d2)'''
 +
* '''w1 equals d0d2 / (d0d1 + d0d2 + d1d2)'''
 +
* '''w2 equals d0d1 / (d0d1 + d0d2 + d1d2)'''
 +
}}
 +
 +
Interpolated regression constants are written in the temporary object SUPIT_CONSTANTS (CGMS14SYS) and copied to object STATIONS (CGMS14SYS). After the regression constants have been established for all stations, global radiation can be calculated by using any one of the above formulae. Finally, the derived daily global radiation of each station is written into object WEATHER_OBS_STATION_CALCULATED (see [[Appendix 6: Flow diagrams#flowchart additional weather calculation|flowchart]]).
 +
 +
===Evapotranspiration===
 +
 +
Daily meteorological station data collected from stations does not contain potential evapotranspiration by crop, wet soils and open water. Potential crop evapotranspiration (ET0) is calculated by the Penman-Monteith equation while potential evapotranspiration of wet soils (ES0) and open water (E0) is calculated by the Penman equation.
 +
 +
{{Hidden|Calculation of potential evapotranspiration
 +
|
 +
++ Penman-Monteith ++
 +
 +
Daily meteorological station data collected from stations does not contain potential crop evapotranspiration. This parameter is calculated by the Penman-Monteith equation ([[References|Allen et all., 1998]]). In general, the evapotranspiration from a reference surface, the so-called reference crop evapotranspiration or reference evapotranspiration can be described by the FAO‑Penman-Monteith:
 +
 +
[[File:ET0.jpg|FAO‑Penman-Monteith equation]]
 +
where:
 +
* ''ETo  : reference evapotranspiration [mm*day-1]''
 +
* ''Rn    : net radiation at the crop surface [MJ*m-2* day-1]''
 +
* ''G    : soil heat flux density [MJ* m-2*day-1]''
 +
* ''T    : mean daily air temperature at 2 m height [°C]''
 +
* ''u2    : wind speed at 2 m height [m*s-1]''
 +
* ''es    : saturation vapor pressure [kPa]''
 +
* ''ea    : actual vapor pressure [kPa]''
 +
* ''es-ea : saturation vapor pressure deficit [kPa]''
 +
* ''Δ   : slope vapor pressure curve [kPa* °C-1]''
 +
* ''γ    : psychrometric constant [kPa*°C-1]''
 +
 +
 +
First, some preprocessing is done:
 +
* Vapour presure is converted from hPa to kPa
 +
* Incoming shortwave radiation (Rs) is converted from kJ.m-2.d-1 to MJ.m-2.d-1
 +
* Windspeed is converted from 10 m to 2 m:
 +
[[File:Formula wind-speed.jpg| 150px| ]]
 +
where:
 +
* ''u2    : wind speed at 2 m above ground surface [m s-1]''
 +
* ''uz    : measured wind speed at z m above ground surface [m s-1]''
 +
* ''z    : height of measurement above ground surface [m]''
 +
 +
Next, the different components of this formula are calculated. As the magnitude of the day or ten-day soil heat flux (G) beneath the grass reference surface is relatively small, it is ignored.
 +
 +
The net radiation (Rn) is the difference between the incoming net shortwave radiation (Rns) and the outgoing net longwave radiation (Rnl). The net shortwave radiation (Rns) is calculated as follows:
 +
 +
[[File:Formula net-incoming-radation.jpg| 100px | ]]
 +
where:
 +
* ''Rns  : net shortwave radiation [MJ.m-2.day-1]''
 +
* ''α    : albedo or canopy reflection coefficient, which is 0.23 for the hypothetical grass reference crop [dimensionless]''
 +
* ''Rs    : incoming shortwave radiation [MJ*m-2* day-1]''
 +
 +
The outgoing net longwave radiation (Rnl) is calculated as follows. First clear-sky radiation (Rso) is derived:
 +
 +
[[File:Formula clear-sky-radiation.jpg| 150px | ]]
 +
where:
 +
* ''Rso  : clear-sky radiation [MJ.m-2.day-1]''
 +
* ''z    : station elevation above sea level [m]''
 +
* ''Ra    : extraterrestrial radiation, Angot radiation [MJ.m-2.day-1]''
 +
 +
Then, the outgoing net longwave radiation (Rnl) is calculated:
 +
 +
[[File:Formula net-longwave-radiation.jpg| 300px | ]]
 +
where:
 +
* ''Rnl    : outgoing net longwave radiation [MJ.m-2.day-1]''
 +
* ''σ      : Stefan-Boltzmann constant [4.903 10-9 MJ.K-4.m-2.day-1]''
 +
* ''Ra    : extraterrestrial radiation, Angot radiation [MJ*m-2* day-1]''
 +
* ''Tmax  : maximum absolute temperature during the 24-hour period [<math>K =</math> °C + 273.16]''
 +
* ''Tmin  : minimum absolute temperature during the 24-hour period [<math>K =</math> °C + 273.16]''
 +
* ''ea    : actual vapour pressure [kPa]''
 +
* ''Rs/Rso : relative shortwave radiation (limited to <math><=</math> 1.0) [dimensionless]''
 +
 +
The psychrometric constant is corrected for atmospheric pressure:
 +
 +
[[File:Formula atmospheric-pressure.jpg| 150px | ]]
 +
where:
 +
* ''P      : atmospheric pressure [kPa]''
 +
* ''z      : elevation above sea level [m]''
 +
 +
[[File:Formula psychrometric-constant.jpg| 150px |]]
 +
where:
 +
* ''γ      : psychrometric constant [kPa °C-1]''
 +
* ''λ      : latent heat of vaporization, 2.45 [MJ.kg-1]''
 +
* ''cp    : specific heat at constant pressure, 1.013 10-3 [MJ.kg-1.°C-1]''
 +
* ''Є      : ratio molecular weight of water vapour/dry air <math>=</math> 0.622''
 +
 +
Next, saturated-vapour-pressure is calculated for both the minimum and maximum temperature and averaged afterwards:
 +
 +
[[File:Formula saturated-vapour-pressure.jpg| 150px | ]]
 +
where:
 +
* ''e°(T)  : saturation vapour pressure at the air temperature T [kPa]''
 +
* ''T      : air temperature [°C]''
 +
 +
Finally, the slope of the saturation vapour pressure curve is determined (first the minimum and maximum temperature are averaged to obtain the average temperature):
 +
 +
[[File:Formula slope-vapour-pressure-curve.jpg| 200px |]]
 +
where:
 +
* ''Δ      : slope of saturation vapour pressure curve at air temperature T [kPa °C-1]''
 +
* ''T      : air temperature [°C]''
 +
 +
 +
++ Penman ++
 +
 +
The Penman-Monteith algorithm is valid only for a reference canopy (ET0) and therefore it is not used to calculate the reference values for bare soil and open water (ES0, E0). The background is that the Penman-Monteith model is basically a surface energy balance where the net solar radiation is partitioned over latent and sensible heat fluxes (ignoring the soil heat flux). To estimate this partitioning, the method links between the surface and air temperature. However, the assumptions underlying the model are valid only when the surface where this partitioning takes place is the same for the latent and sensible heat fluxes. For a crop canopy this assumption is valid because the leaves of the canopy form the surface where both latent heat flux (through stomata) and sensible heat flux (through leaf temperature) are partitioned.
 +
 +
For a soil, this principle does not work because when the soil is drying the evaporation front will quickly disappear below the surface and therefore the assumption that the partitioning surface is the same does not hold anymore. For water surfaces, the assumptions underlying Penman-Monteith do not hold because there is no direct relationship between the temperature of the water surface and the net incoming radiation as radiation is absorbed by the water column and the temperature of the water surface is co-determined by other factors (mixing, etc.). Only for a very shallow layer of water (1 cm) the Penman-Monteith methodology could be applied. For bare soil and open water the Penman model is preferred. Although it partially suffers from the same problems, it is calibrated somewhat better for open water and bare soil based on its empirical wind function.
 +
 +
Finally, in crop simulation models the open water evaporation and bare soil evaporation only play a minor role (pre-sowing conditions and flooded rice at early stages), it is not worth investing much effort in improved estimates of the reference values.
 +
 +
Evapotranspiration from a wet bare soil surface (ES0) and from a water surface (E0) is calculated with the Penman formula ([[References|Penman, 1948]]). Only the albedo and surface roughness differs for these two types of evapotranspiration as explained below:
 +
 +
[[File:E0.jpg|Penman equation]]
 +
where:
 +
* ''E0 : evapotranspiration from a water surface [mm*d-1]''
 +
* ''Rna : net absorbed radiation [mm*d-1]''
 +
* ''EA : Evaporative demand [mm*d-1]''
 +
* ''Δ : Slope of the saturation vapor pressure curve [mbar*C-1]''
 +
* ''γ : Psychometric constant (0.67) [mbar*C-1]''
 +
 +
The net absorbed radiation depends on incoming global radiation, net outgoing long-wave radiation, the latent heat and the reflection coefficient of the considered surface (albedo). For ES0 and ET0 albedo values of 0.15 and 0.20 are used respectively. The evaporative demand is determined by humidity, wind speed and surface roughness. For a free water surface and for the wet bare soil (E0, ES0) a surface roughness value of 0.5 is used. For a more detailed description of the underlying formulae we refer to [[References|Supit et al. (1994)]] and [[References|van der Goot (1997)]].
 +
 +
 +
Note that coefficients of the Angstrom method are required to calculate the atmospheric transmission within the calculation of the net outgoing long wave radiation.
 +
}}
 +
 +
Calculated E0, ES0, and ET0 are stored in object WEATHER_OBS_STATION_CALCULATED.
 +
 +
== Messages to the Project Management Board ==
 +
Information on successfull completion of the various processing steps is sent to the [[Software_Tools#Project_Management_Board|Project Management Board]] (PMB).
 +
 +
{|class="collapsing_table collapsible collapsed"
 +
!List of signals communicated to the Project Management Board (PMB) in connection to the processing of observations from ground weather stations.
 +
|-
 +
|
 +
 +
{| class="wikitable"
 +
!EVENT_ID!!Region!!FREQUENCY!!DELAY!!HOUR!!MINUTE!!THEME!!STEP!!RESOLUTION!!LOCATION
 +
|-
 +
|310||EUR||Daily||1||19||0||Weather - OBS||Delivered||Station||FTP WENR
 +
|-
 +
|303||EUR||Daily||1||19||0||Weather - OBS - 6HourlyRain||Delivered||Station||FTP WENR
 +
|-
 +
|304||EUR||Daily||1||19||0||Weather - OBS - 3HourlyRain||Delivered||Station||FTP WENR
 +
|-
 +
|200||CHN||Daily||1||19||0||Weather - OBS||Delivered||Station||FTP WENR
 +
|}
 +
 +
|}
 +
 +
 +
 +
 +
[[Category:Weather Monitoring]]

Latest revision as of 16:39, 28 March 2022



General description

The processing of observed station weather into the MCYFS involves four steps:

preprocessing of station weather data

Data acquisition from weather stations

Weather stations (black dots) for which data are available for (part of) the period from 1975 until present

The selection of stations is limited to those stations that regularly collect data and can supply data in near real time. Relevant meta data of stations includes station number, station name, latitude, longitude and altitude. This data is available in the object STATIONS.

Currently, data acquisition and processing applies to two regional windows: Europe and China. Mainly examples from Europe are shown in this documentation.

Some of the historic meteorological data were purchased directly from National Meteorological Services. Others were acquired via the GTS. As data are obtained from a variety of different sources, considerable pre-processing was necessary to convert them into a standard format. Around 1992 the historic meteorological data represented approximately 380 stations in the EU, Switzerland, Poland and Slovenia with data from 1949 to 1991 (Burrill and Vossen, 1992). Later the historic sets have been extended with stations in Eastern Europe, western Russia, Maghreb and Turkey. The historic data were converted into consistent units and checked on realistic values. The database was also scanned for inconsistencies, such as successive days with the same value for a variable, or minimum temperatures higher than maximum temperatures (Burrill and Vossen, 1992).

From 1991 to present, meteorological data is received in near-real-time from open data sources and from contracted providers like ECOMET or national or regional meteorological services. Sources include the WMO GTS network, NOAA data access points, regional and national meteorological services, and the access points for non-essential WMO reports. The data arrives either in standardized encoded formats as defined by the WMO or ICAO, or in proprietary formats as used by individual providers. It is first decoded and converted into a generalized structure, including unit and time zone conversions, alignment of reference periods and - where needed - the assignment of station-id. Basic first level data sanity checks are applied. In a next step, the data is converted into the input-format as required by the QUACKME software package. The temporal resolution of the data ranges from 1-hourly to 24-hourly, depending on the parameter. QUACKME is applying data quality checks, calculates derived parameters and daily aggregates and writes the data into the file formats as expected by the EFAS and MCYFS downstream processes.

In recent years, the earlier archives (1975-2004) of Scandinavia and eastern Europe have been enriched. In 2016 data from around 300 Chinese stations have been acquired starting a new service for this region.

Available stations

The stations, stored in object STATIONS holds over 10221 stations distributed over 40 countries in Europe and neighboring countries. Over 5100 of these stations provide weather data in near real time. All weather data is stored in the stations weather database (daily data in object WEATHER_OBS_STATION and 3- and 6-hourly precipitation in object WEATHER_OBS_STATION_RAIN).

Raw station data is collected from various sources:

  • GTS (essential data and data licensed by ECOMET restrictions)
  • NOAA (USA)
  • European National Meteorological Institutes (NMI) (licensed)
  • Various regional networks in Europe

For transmission and international exchange, the internationally exchanged station reports are encoded in formats standardized and maintained by WMO and International Civil Aviation Organizaton (ICAO)

Observations as provided directly by National Meteorological Institutes or regional authorities come from secondary networks and are provided in proprietary formats.

Meteorological stations selected in priority are those located in the agricultural zones and equally distributed over the mainland (instead of over islands - for Portugal, Spain or Greece in particular). In particular, for western Russia (western of Urals) the main areas covered are the agricultural districts.

In the case of China roughly 300 stations were selected meeting the following criteria:

  • Near real time delivery
  • A 20-years archive
  • Located in the main agricultural areas
  • Covering the elements: precipitation, minimum and maximum temperature, humidity and wind speed

The raw station data for China is collected from GTS.

Basic indicators

The basic indicators that are received from weather stations include:

  • Sum of precipitation
  • 2m air temperature
  • Maximum of 2m air temperature
  • Minimum of 2m air temperature
  • Downward directed solar radiation measured at earth's surface (global radiation)
  • Duration of sunshine
  • Total cloud cover
  • Water vapour pressure
  • Relative humidity
  • 10m mean wind speed
  • Snow depth

For WMO SYNOP FM 12 and BUFR FM 94 bulletins, WMO defines regional regulations to consider time zones and national coding practices. The extent of reported parameters and the report frequency differs per country and is for ECOMET member countries affected as well by license restrictions.

The METAR code is standardized through the ICAO. In Europe and China, the WMO-maintained codes SYNOP FM12 and BUFR FM 94 provide higher accuracy for the various parameters and more detail. In these regions, METAR provides only temperature, dew point, visibility, cloud amount and wind speed and is reported in coarser increments for the various parameters. The reporting frequency is determined by the individual flight operation of each airport. Nevertheless, METAR reports are used as well, mostly to fill spatial gaps in areas with less WMO stations.

Observation data from several European regional meteorological networks became available after 2015. The data from regional networks are mostly not available in the standard meteorological formats, but have to be collected and converted individually. The quality of the data is determined by the installed sensors and the siting of the stations. The reported parameters and frequency differ by network.

The following table summarizes basic information on the availability and reporting regulations from the various observing station data sources:

Parameter Reference periods of reports as defined by WMO WMO formats BUFR or SYNOP (*) METAR (**) Regional networks
Sum of precipitation 24-hourly sum, 12-hourly sums, 6-hourly sum, 1-hourly sum reported, depending on region WMO-region and local regulations Europe: 06 UTC: past 24 hours / 00 UTC and 12 UTC: past 6 hours / 06 UTC and 18 UTC: past 12 hours / 1-hourly -- China: Reports 00 UTC for past 24 hours, some stations report 21 UTC for previous 24 hours (***) Not reported in Europe and China Individual, mostly 1-hourly
2 m air temperature Instantaneous value Reported with 0.1 K accuracy Reported as full degrees Mostly 0.1 K accuracy
Maximum 2 m air temperature Maximum of continuous measurement during reference period (****) Europe and China: reported 18 UTC Not reported in Europe and China Individual
Minimum 2 m air temperature Minimum of continuous measurement during reference period (****) Europe and China: reported 06 UTC Not reported in Europe and China Individual
Downward directed surface solar radiation (global radiation) Sum accumulated over past 24 hours, sum past 1 hour Available for some European countries at 00 UTC, 1-hourly Not reported in Europe and China Individual definition, mostly 1-hourly
Duration of sunshine Sum accumulated over past 24 hours Most European countries report at 06 UTC Not reported in Europe and China Individual, mostly 1-hourly
Total cloud cover Instantaneous value Octas 0-8 5 stages, only clouds up to a height of 5000 feet over ground reported Not reported
Measures for the humidity of the air at 2 m above ground: dew point, water vapour pressure and relative humidity Instantaneous value of dew point temperature reported (*****) Reported with 0.1 K accuracy Reported as full degrees to be derived from other humidity parameters like relative humidity and air temperature
10 m mean wind speed Mean over past 10 minutes Meters per second Mostly full knots, occasionally less accuracy during low wind situations Individual definition
Snow depth Instantaneous value, increasing automatization of measurement When a station reports snow depth, it is done in Europe by 06 UTC, in China by 00 UTC not reported Not reported

(*) Main synoptic hours are 00, 06, 12, 18 UTC. Intermediate synoptic hours are 03, 09, 15, 21 UTC. For most European countries, 1-hourly data is used as well.
(**) The report frequency is determined by the airport's schedule and can be as often as 20 minutes. The frequency of reports can change over daytime, weekday, and season.
(***) In BUFR, several countries do not provide the reference period during dry conditions in the FM94 code, supposedly by accident. In this case, it is assumed that the WMO definitions for the reference period are applied.
(****) Europe: Covers past 12 hours. China: Covers past 24 hours.
(*****) Other thermodynamical measures for the humidity of air can be calculated from dew point and air temperature.

Data quality check

The software package Quality Checks Meteorological Data (QUACKME) as developed by the JRC is the main processing tool for completing and quality evaluation of actual meteorological data which is used as input for agro-meteorological models. The data processing workflow with quality control and aggregation can be described as follows.

Near real-time pre-processing (1-hourly reports with extended information at intermediate and main synoptic hours, irregular reports)

  • Near real-time collection of reports from the various data sources.
  • Decoding of the WMO and ICAO standard formats with dedicated decoder software (FMDecode). Reports in other formats from regional, secondary networks are translated into a uniform structure using individual proprietary converters.
  • The data is converted into a generalized structure, including the conversion towards UTC, standard units, the alignment of reference periods and the calculation of derived parameters. Basic sanity checks are applied.

Preparation of QUACKME input data

  • Generate a csv with all available observation data for the period of 24 hours (07 UTC - 06 UTC next day) for the European region and for China, respectively. The format of the csv is described in the QUACKME Technical Guide (pdf)
  • When data from a station are found to be erroneous for a longer time, the station can be listed on a so-called blacklist, either by parameter or for the whole report. Observations from blacklisted station-parameter combinations are not written into the csv. The blacklist is manually checked every three months.If the messages are considered trustworthy again, the station-parameter combination is removed from the blacklist.
  • Generate csv with location specific, near real time forecasts for the same stations and period as the data in the observation-csv. The format of this csv is as well described in the QUACKME Technical Guide.

For a number of weather elements, QUACKME compares the observed values with near-real-time forecast values. The forecast is used as reference for the reasonable range of possible values. The forecasts are obtained through a technique called MOS (Model Output Statistics). Meteorological forecast models, e.g. the ECMWF model, compute the physical status of the atmosphere on a grid, and the results represent the expected situation per grid box. The MOS forecast is using statistical relationships between the observations of a particular station and historic model forecasts for surrounding grid points. Each observing location has its own statistics. In this way, the local conditions at the weather station can be modelled much more accurately. QUACKME is using the individual location forecasts to define time- and location-dependent thresholds for the trustworthiness of station reports, for the elements air temperature (including minimum and maximum), dew point (applies to all derived measures for the humidity of the air), precipitation, and wind speed, respectively. That way, the thresholds consider season, climatology and even the actual weather pattern. A welcome side effect is the high spatial consistency of the statistical MOS approach and therefore of the thresholds. Individual MOS forecasts is used for almost all stations (approx. 5000, state January 2021).

Running the QUACKME modules and interactive data quality checks by the meteorologist

  • This does not apply for precipitation, i.e. for consecutive reports of 0 mm. This rather typical reporting bug is not found when quality checks are applied on to the data of the very day. Due to the mostly “patchy” pattern of precipitation events quality checks accept dry stations in between. To find stations that report consecutively 0 mm several weeks of history need to be considered, see retrospective checks.
  • Correct automatically obvious errors detected while performing these checks;
  • Automatically fill gaps in the database through interpolation based on time consistency criteria;
  • Flag dubious observations which cannot be corrected automatically;
  • Write all automatic corrections and flagged dubious observations to a log file;
  • Have the flagged observations checked and, if necessary, corrected by a trained meteorologist; when a correction is done, the derived parameters are recalculated and the data are written back to the database.

Dedicated trained and qualified meteorologists go through the dubious observation values that are flagged as such by the QUACKME automatic pre-checking program. An interactive system for the visualization of meteorological data is used to graphically visualize and analyze additional information such as:

  • Station observation data
  • Satellite images
  • Precipitation Radar data
  • Analysis and short range forecasts computed by physical models of the atmosphere
  • Short range forecasts for weather station locations

This additional data is used by the analyst to decide on either confirmation or rejection of the observed values.

Conversion to daily values

Once the database has been filled following the method described above, data are aggregated to daily values. This includes the indicators as summarized in the following table:

Parameter Aggregation Reference period Europe Reference period China
Total cloud cover (N) Daily mean 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
Duration of sunshine (Msun) 24-hourly sum 00–24 UTC Not available
Downward directed surface solar radiation (global radiation) (Mrad) 24-hourly sum 00-24 UTC Not available
Minimum 2m air temperature (Tn) Lowest value of continuous reference period (*) 18 previous day -06 UTC 06 UTC previous day – 06 UTC
Maximum 2m air temperature (Tx) Highest value of continuous reference period (**) 06-18 UTC 18 UTC previous day – 18 UTC
Water vapour pressure (e) Daily mean 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
10m mean wind speed (ff10) Daily mean 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
Sum of precipitation (RRR) 24-hourly sum Mostly 06 UTC until 06 UTC next morning Mostly 00 UTC – 00 UTC next day (indicator 2).
For some stations 21 UTC previous day – 21 UTC (indicator 6)
2m air temperature (TT) 03-hourly instantaneous values during daytime 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC
Relative humidity (RH) 03-hourly instantaneous values during daytime 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC
State of soil Instantaneous value (***) 00 UTC following day
Water vapour pressure deficit (vpd) Daily mean 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
Slope of saturation vapour pressure vs. temperature curve slope Daily mean 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC, 21 UTC, 00 UTC next day 18 UTC prev., 21 UTC prev., 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC
Total cloud cover (N) Daytime mean 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC
Low or (when no low clouds) medium clouds (Nh) Daytime mean 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC Not available
Calculated sunshine duration (Csun) 24-hourly sum To be calculated by QUACKME, 0-24 UTC of the day specified To be calculated by QUACKME, 18 UTC previous day - 18 UTC of the day specified
Highest possible global radiation at clear sky (Crad) 24-hourly sum To be calculated by QUACKME, 0-24 UTC of the day specified To be calculated by QUACKME, 18 UTC previous day - 18 UTC of the day specified
Potential evapotranspiration (ETP) 24-hourly sum To be calculated by QUACKME, 0-24 UTC of the day specified To be calculated by QUACKME, 18 UTC previous day - 18 UTC of the day specified
Visibility (VV) Daytime mean 06 UTC, 09 UTC, 12 UTC, 15 UTC, 18 UTC 00 UTC, 03 UTC, 06 UTC, 09 UTC, 12 UTC
Snow depth Instantaneous value 06 UTC 00 UTC

(*)When no minimum is reported but hourly instantaneous temperatures QUACKME estimates the minimum from the hourly local early morning values, see QUACKME Technical Guide (pdf)
(**)When no maximum is reported but hourly instantaneous temperatures QUACKME estimates the maximum from the hourly local afternoon values, see QUACKME Technical Guide (pdf)
(***)Code, for translation see BUFR documentation.

Information on the way the daily element values are constructed/defined is stored in the object WEATHER_OBS_STATION_INFO. Currently this is only done for precipitation e.g. period definition of the daily rainfall sum. Codes are:

  • 0 = real observation 06 - 06 UTC next day
  • 1 = period 06 - 06 UTC next day, short range forecast has been used to cover the complete period
  • 2 = real observation 00 UTC - 24 UTC
  • 3 = real observation 03 UTC - 03 UTC next day
  • 4 = real observation 12 UTC previous day - 12 UTC
  • 5 = real observation 18 UTC previous day - 18 UTC
  • 6 = real observation 21 UTC previous day - 21 UTC

More information on the 3- and 6-hourly precipitation data are stored in object WEATHER_OBS_STATION_RAIN (column IDFLAG). Codes are:

  • 1 = Changed by meteorologist (not applicable)
  • 2 = Automaticaly corrected (not applicable)
  • 3 = Observation
  • 4 = Linear interpolation from observations
  • 5 = Interpolated via MOS from observations
  • 6 = MOS analyses (not managed yet)

Finally, meta data of all stations is checked once a year.

Retrospective checks and blacklisting of suspect stations

Some suspicious station reports are only detectable by checking time series of several weeks. Continuous reports of 0 mm precipitation (instead of a "precipitation not observed" flag) do not stick out in daily rainfall sums, but only by investigating the station's reports over a longer period. Global radiation and cloud cover have a high spatial volatility, and continuous observation or encoding errors at a certain station become more explicit when looking into several weeks of station reports.

For all European stations, the QUACKME output of the past 40 days is inspected each week through time series checks. Provided a station reported on more than half of the tested days, the reports are checked, consulting ECMWF model analysis and short range forecasts for model grid points surrounding the station of request.

The checks are set up as follows:

Precipitation

A station is flagged as suspicious for precipitation when suspicious consecutive zero rainfall reports are detection. Criteria are

  • The observed precipitation sum is 0 mm whan aggregated over the whole checked period.

AND

  • the ECMWF model near real time forecasts during the checked period included at least 10 wet days. A day is considered being wet when more than 0.5 mm precipitation is forecasted by the model's near real time foreast.
Radiation

For each day of the investigated period, the station’s maximum possible daily solar radiation sum is calculated, based on its latitude, the time of year, and using a standard atmospheric optical depth. A station is flagged as being suspicious for radiation when:

  • There are at least 10 days with observed solar radiation exceeding 110 % of the maximum possible amount of solar radiation.

OR

  • There are at least 10 days on which the observed solar radiation remained below 10 % of the maximum possible daily sum of solar radiation.

OR

  • There are at least 10 days with observed radiation of 0 MJ m-2 day-1 whilst the ECMWF short-range forecast analysed solar radiation exceeding 0 MJ m-2 day-1.

OR

  • The total sum of observed solar radiation is less than 25 % of the maximum possible radiation sum for the period, whilst the sum of the model's short-range forecasts for the parameter exceeded 25 % of the maximum possible daily sum of solar radiation. Naturally, the maximum possible radiation period's is only summed up from days with observations being available.
Mean daytime cloudiness

A station is flagged as being suspicious for cloudiness when:

  • A difference of more than 2.5 octa between the daily mean of observed total cloud cover and the daily mean of ECMWF model analysis and short-range forecast for total cloud is found for all days of the investigated period.

OR

  • The reported instantaneous cloudiness was always higher than 4.0 octa whilst the model analysis and short-range forecasted for at least three time steps (hours) in the period a total cloud cover of less then 3.0 octa.

OR

  • For all time steps in the period more than 5 octa total cloud cover was reported.
Duration of sunshine

For each day in the investigated period, the maximum day length is calculated based on the day of the year and the station latitude. Dividing the observed sunshine duration for a day by the calculated day length gives the relative sunshine.
A station is flagged as being suspicious for sunshine duration when:

  • For more than 10 days in the period, the observed duration of sunshine is more than 110% of the calculated day length.

OR Depending on the dominant season during the period:

  • Summer: highest relative sunshine value is less than 30% (i.e. the station is always cloudy).
  • Winter: the lowest relative sunshine value is more than 70% (i.e. the station is always sunny)
  • Spring/autumn: highest relative sunshine value is less than 30% (see summer check) OR the lowest relative sunshine value is more than 70% (see winter check).

The dominant season is determined as the season with the largest number of days in the investigated period. When 50% of the investigated days are winter/summer days, the dominant season will be winter/summer.


When the process flags stations as suspect a final manual inspection by a meteorologist follows. If the time series of the station are found to be wrong the following actions are executed:

  • The station is added to a blacklist: the station is immediately excluded from the operational station list.
  • The erroneous time series are deleted from the objects WEATHER_OBS_STATION_RAIN and WEATHER_OBS_STATION. The erroneous values are flagged (object WEATHER_OBS_STATION_RAIN, column TYPE) or deleted (object WEATHER_OBS_STATION and WEATHER_OBS_STATION_INFO) and deleted values are saved in separate objects (WEATHER_OBS_STATION_ERRORS and WEATHER_OBS_STATION_INFO_ERR).
  • All affected grid cells (object WEATHER_OBS_GRID) and regions (object WEATHER_OBS_REGIONCOVER) are reprocessed at regular time intervals. This also includes the crop simulation results.

Every three months, by the end of the quarter, each station on the blacklist is verified. Afterwards it is decided if stations can return to the operational work flow. Falsely blocked data is back-ordered, added and reprocessed.

Station data availability

Each month an overview is created showing the delivered number of stations per country. Information is also added on sudden changes and follow-up actions. Example monthly overview

Every day, the newly produced data files are compared with those of the previous day. If the number of delivered values for individual countries and parameters decreases significantly, an alert is sent by email is sent to the project team. The threshold above which a decline in the number of values delivered is considered critical depends exponentially on the number of values in the country.

The number of values flagged by the weak, heavy and threshold checks of QUACKME are monitored on a daily and on a monthly basis. Stations that are flagged particularly frequently are identified and the cause can be analysed separately. As a result, stations can be blacklisted or an improvement of the QUACKME checks can be suggested.

The following maps illustrate the available stations (red 0-20% - green 80-100%) for the main elements in a recent year 2019. The main elements (maximum temperature, minimum temperature, precipitation, sun shine, cloud cover, wind speed and vapor pressure) have a good spatial spread over Europe with a relative high spatial density in western and central Europe. Availability of measured radiation is mainly limited to western and central Europe.

Stations tmax 2020.jpg
Stations tmin 2020.jpg
Stations rain 2020.jpg
maximum temperature minimum temperature precipitation
Stations radiation 2020.jpg
Stations sunshine 2020.jpg
Stations cloud 2020.jpg
global radiation sunshine cloud cover
Stations wind 2020.jpg
Stations vapour 2020.jpg
Stations snow 2020.jpg
wind speed 10m vapor pressure snow depth

The following graph shows the increase of observations for the main elements between 1975 and 2019. Most elements have at least 600,000 annual observations which equals over more than 1600 stations in case they would have a complete temporal coverage. However, most stations have temporal gaps and therefore the number of reporting stations is much higher. Since 2004 the number of observations increased up-till a level of around 1,500,000 reported by more than 4500 stations. During the recent years also observations of radiation related elements increased drastically. This is especially true for cloud cover and sunshine. Prior to 1995 these elements have a relative low number of observations meaning that the global radiation of these years, required in MCYFS, is mainly based on the daily temperature range, see Calculation of advanced parameters.

Availability over time 2020.JPG


In general the station density and available data in the monitored areas is considered sufficiently high for the purpose of the project.

Ingestion into the database

After the station weather data passed all checks, daily weather data is exported to a fixed formatted ASCII file (s-file) containing the data of a single day that can be imported in the object WEATHER_OBS_STATION. In the near real time situation a s-file is delivered one day later. For example in the afternoon of day 31 March 2016 the following file is generated: s20160330.dat.


The 3-hourly rainfall data is exported to a plain ASCII file (rrr3h_*.txt file) containing the data of one 3-hourly time step within one single day. This data can be imported in the object WEATHER_OBS_STATION_RAIN. In the near real time service each day 8 rrr3h_*.txt files are generated at once containing data of 8 3-hourly time steps:

  • 09 UTC (06-09 UTC of previous day)
  • 12 UTC (09-12 UTC of previous day)
  • 15 UTC (12-15 UTC of previous day)
  • 18 UTC (15-18 UTC of previous day)
  • 21 UTC (18-21 UTC of previous day)
  • 00 UTC (21-00 UTC of previous day)
  • 03 UTC (00-03 UTC of present day)
  • 06 UTC (03-06 UTC of present day)

For example in the afternoon of day 31 March 2016 the following files are generated: rrr3h_2016033009.txt, rrr3h_2016033012.txt, rrr3h_2016033015.txt, rrr3h_2016033018.txt, rrr3h_2016033021.txt, rrr3h_2016033100.txt, rrr3h_2016033103.txt and rrr3h_2016033106.txt.


The 6-hourly rainfall data is exported to a plain ASCII file (rrr_*.txt file) containing the data of one 6-hourly time step within one single day. This data can be imported in the object WEATHER_OBS_STATION_RAIN. In the near real time service each day 4 rrr_*.txt files are generated at once containing data of 4 6-hourly time steps:

  • 12 UTC (06-12 UTC of previous day)
  • 18 UTC (12-18 UTC of previous day)
  • 00 UTC (18-00 UTC of previous day)
  • 06 UTC (00-06 UTC of present day)

For example in the afternoon of day 31 March 2016 the following files are generated: rrr_2016033012.txt, rrr_2016033018.txt, rrr_2016033100.txt and rrr_2016033106.txt.


Calculation of advanced parameters

Global radiation

Global radiation is the daily sum of incoming solar radiation that reaches the earth surface. It is mainly composed of wavelengths between 0.3 μm and 3 μm. Approximately half of the incoming radiation with wavelengths between 0.4 and 0.7 μm is Photosynthetically Active Radiation (PAR). Global radiation is the driving variable in the growth-determining CO2 assimilation process and thus crop growth models are sensitive to radiation data (van Diepen, 1992).

A major problem is the scarcity of measured global radiation. In cases where no direct observations are available it must be derived from sunshine duration, cloud cover and/or temperature, on the basis of statistical relationships. If measured global radiation is missing, it is based on one of three formulae (Ångström-Prescott, Supit-Van Kappel, and Hargreaves), depending on the availability of meteorological parameters. An important component in these formulae is the amount of Angot radiation which is the extraterrestrial radiation integrated over the day at certain latitude on a certain day. The calculation of the Angot radiation and the three different formulae are described by Supit et al. (1994) and van der Goot (1998a).

Angot radiation

The principle component of all three formulae is the extraterrestrial radiation, or Angot radiation. In fact, all of the three formulae estimate the fraction of Angot radiation actually received at the earth surface. The Angot radiation is calculated as:


The following hierarchical method is used to calculate global radiation for each station (Supit and van Kappel, 1998) in case measured global radiation is missing:

Ångström-Prescott formula

If sunshine duration is available, global radiation is calculated using the equation postulated by Ångström (1924) and modified by Prescott (1940). The two constants in this equation depend on the geographic location.


Supit-Van Kappel formula

When neither measured radiation nor sunshine duration are available, but minimum and maximum temperature and daytime cloud cover are known, the Supit-Van Kappel formula is used. This is an extension of the Hargreaves formula (Supit, 1994). The regression coefficients depend on the geographic location.


Hargreaves formula

When only the minimum and maximum temperatures are known the equation of Hargreaves et al. (1985) is used. The regression coefficients depend on the geographic location.


Any one of the above three methods has an additional upper limit. The maximum calculated global radiation is set to Angot radiation, corrected for atmospheric transmissivity, by multiplying the Angot value with the sum of the Angstrom A and B coefficients.

Deriving Ångström-Prescott, Supit-Van Kappel, and Hargreaves regression constants

The main problem with the application of the Ångström-Prescott, Supit-Van Kappel, and Hargreaves formulae is the quality of the regression constants. Studies by Supit (1994), Supit and van Kappel (1998) and van Kappel and Supit (1998) showed no relationship between latitude and the coefficients for Europe, although such a relation is frequently used to estimate these regression constants. Initially in MCYFS regression constants of Supit and van Kappel (1998) and van Kappel and Supit (1998) for Europe were used. They obtained sets of regression constants for the formulae for as many weather stations as possible, with a geographic distribution that corresponds to the area of interest for the MCYFS. As a result, a set of 256 reference stations was identified for which a relevant set of measured radiation data and other parameters in the formulae existed. For these stations regression constants were calculated based on measured radiation data for the three formulae mentioned above.

In 2012 the regression coefficients of these solar radiation models for Europe were updated using a new set of weather station data (temperature, sunshine and cloudcover) and an alternative training data set: 6 years (2005-2010) of the down-welling surface shortwave radiation flux (DSSF) 30-minutes product derived from Meteosat Second Generation satellite data by the Land Surface Analysis Satellite Applications Facility (LSA SAF) (Bojanowski et al.,2013). For each solar radiation model a set of weather stations was selected having sufficient observations of either sunshine duration, or cloud cover/temperature or only temperature (minimum and maximum) to perform a regression analysis. Results are stored in object STATION_REFERENCE_COEFFICIENTS (CGMS14SYS).

Station archive data for China did not include measured radiation nor sunshine. Therefore radiation was derived from other observed elements namely cloud cover and minimum and maximum temperature. The Hargreaves and Supit-VanKappel models have been trained using modelled radiation by Tang et al., 2013. The 50yrRad database of Tang et al., 2013 containing ‘modelled’ radiation data for 716 CMA stations, has demonstrated its superior performance over previous estimates of locally calibrated Angstrom-Prescott models. While radiation is based on the Hargreaves or Supit-VanKappel models, coefficients of the Angstrom method are still required to calculate net outgoing long wave radiation within the potential evapotranspiration calculation. For determining Angstrom coefficients only the 50yrRad archive was used. Since no sunshine duration data is available, an alternative was sought. Transmissivity was derived by dividing the measured solar radiation at the ground by the solar radiation at the top of the atmosphere. By selecting only the period between day of year 150 and 200 (during mid-summer) the transmissivity is almost constant and can be linked to the Angstrom coefficients.

The program SupitConstants uses this set of data (via the view SUPIT_REFERENCE_STATIONS, CGMS14SYS), consisting of latitude, longitude, altitude and calculated regression constants, to derive the regression constants for all stations in the MCYFS. Interpolation of the regression constants of the reference stations to other stations is based on a distance weighted average of the three nearest stations. This process is carried out once, unless the set of reference stations changes or when new stations are added or when meta data of stations change.


Interpolated regression constants are written in the temporary object SUPIT_CONSTANTS (CGMS14SYS) and copied to object STATIONS (CGMS14SYS). After the regression constants have been established for all stations, global radiation can be calculated by using any one of the above formulae. Finally, the derived daily global radiation of each station is written into object WEATHER_OBS_STATION_CALCULATED (see flowchart).

Evapotranspiration

Daily meteorological station data collected from stations does not contain potential evapotranspiration by crop, wet soils and open water. Potential crop evapotranspiration (ET0) is calculated by the Penman-Monteith equation while potential evapotranspiration of wet soils (ES0) and open water (E0) is calculated by the Penman equation.


Calculated E0, ES0, and ET0 are stored in object WEATHER_OBS_STATION_CALCULATED.

Messages to the Project Management Board

Information on successfull completion of the various processing steps is sent to the Project Management Board (PMB).