Software Tools
Aggregator
This is an ORACLE package containing stored procedures developed to aggregate crop simulations to EMU, grid and regional levels and gridded weather to regional levels while differentiating for different land covers or crops. The package have to be compiled for every regional window: EUR, RUK, IND, CHN, SAM and GLO.
The core of the package is the Aggregator.aggregation procedure that select a specific aggregation procedure given the provided arguments. The arguments can be
- Theme Crop indictors or Weather indicators (values: ‘SIM’ or ’WEATHER’)
- Model Crop model (values: ‘WOFOST’,’WARM’,’BLAST’)
- Meteo Meteo model (values: ‘OBS’,’HRES’,’OPE’,’HIS’,’ERA’,’ENS’,’ENSEXT’,’SEA’)
- Crop Selected crop (values: crop number or null for all crops)
- Start Selected start-date
- End Selected end-date
- From Start resolution (values: ‘STU’,’EMU’,’GRID’,’REGION_LOW’,’REGION_UPPER’)
- To Target resolution (values: ’EMU’,’GRID’,’REGION_LOW’,’REGION_UPPER’)
- Region Selected region (values: region code or null for all regions)
- Continent Selected continent (values: continent code or null for all continents)
- Regional level Select the regional level from which data must be aggregated
Two general types (themes) of aggregation procedures are distinguished:
- Aggregate weather data from GRID resolution to all regional levels for every land cover using the information from table LINK_GRID_REGION_COVER.
- Aggregated crop simulation results from
- STU to EMU resolution
- EMU to GRID resolution
- EMU to Lowest regional level resolution
- GRID to Lowest regional level resolution
- Lowest regional level to Upper regional resolutions
Aggregation to the levels GRID and lowest regional level is not crop specific but based on the land cover associated with the crop parametrization (table CROP_PARAMETRIZATIONS). Area weights of these land covers are given in tables LINK_SMU_GRID_COVER and LINK_SMU_GRID_REGION_COVER. Aggregation from the lowest regional level to upper regional levels is based on crop specific area weights coming from table STAT_REGION_AREAS. Further simulation results of crop parametrization can be linked to different crops available in the statistics. For example in the regional RUK window we simulate winter wheat with parametrization 1 but we aggregate the outputs at regional level for aggregation 1 and 3, thus for winter wheat and winter barley.
| Configuration of the Aggregator |
|---|
Aggregator.aggregation procedure has following parameters:
1) vcTheme => this required parameter is the theme we want aggregate, allowed values: 'SIM' / 'WEATHER'
2) vcModel => this parameter is required only for the crop simulation theme else it is null, allowed values: 'WOFOST' / 'WARM' / 'BLAST' / 'CROPSYST' / 'FROST'
3) vcMeteo => this required parameter selects the weather data set to aggregate, allowed values: 'HRES' / 'OPE' / 'HIS' / 'SEA' / 'ENS' / 'ENSEXT'
4) nCrop => this optional parameter can be used to select a single crop parametrization / aggregation crop to aggregate. When it is null all crops will be selected.
5) dStart => this required parameter selects the start period
6) dEnd => this required parameter selects the end period
7) vcFrom => this required parameter selects the level from which data must be aggregated, allowed values: 'STU' / 'EMU' / 'GRID' / 'REGION_LOW'
8) vcTo => this required parameter selects the level to which data must be aggregated, allowed values: 'EMU' / 'GRID' / 'REGION_LOW' / 'REGION_UPPER'
9) vcRegion => this optional parameter can select only the grids they fall inside the region. When it is null all regions will be included. The code should be a value from database object {{Object|REGIONS}}.
10) vccontinent => this optional parameter can be used to select the continent for which data must be aggregated. Can be used to simply run aggregations in parallel since different continent don't share vcRegions
11) nRegionLevel => this optional parameter can be used to select the regional level from which data must be aggregated.
Examples on how to use the above procedure:
DECLARE
THEME_SIMULATION CONSTANT VARCHAR2(15):='SIM';
THEME_WEATHER CONSTANT VARCHAR2(15):='WEATHER';
AGGREGATOR_STU CONSTANT VARCHAR2(15):='STU';
AGGREGATOR_EMU CONSTANT VARCHAR2(15):='EMU';
AGGREGATOR_GRID CONSTANT VARCHAR2(15):='GRID';
AGGREGATOR_REGION_LOW CONSTANT VARCHAR2(15):='REGION_LOW';
AGGREGATOR_REGION_UPPER CONSTANT VARCHAR2(15):='REGION_UPPER';
METEO_OPERATIONAL CONSTANT VARCHAR2(15):='HRES';
METEO_DETERMINISTIC CONSTANT VARCHAR2(15):='OPE';
METEO_ANALISYS CONSTANT VARCHAR2(15):='HIS';
METEO_REANALISIS CONSTANT VARCHAR2(15):='ERA';
METEO_FORECAST CONSTANT VARCHAR2(15):='ENS';
METEO_FORECAST_EXTEND CONSTANT VARCHAR2(15):='ENSEXT';
METEO_SEASONAL CONSTANT VARCHAR2(15):='SEA';
MODEL_WOFOST CONSTANT VARCHAR2(15):='WOFOST';
MODEL_WARM CONSTANT VARCHAR2(15):='WARM';
MODEL_BLAST CONSTANT VARCHAR2(15):='BLAST';
MODEL_FROSTKILL CONSTANT VARCHAR2(15):='FROST';
MODEL_CROPSYST CONSTANT VARCHAR2(15):='CROPSYST';
BEGIN
Aggregator.Aggregation
(vcTheme => THEME_SIMULATION
,vcModel => MODEL_WOFOST
,vcMeteo => METEO_OPERATIONAL
,nCrop => NULL
,dStart => TO_DATE('19990101','YYYYMMDD')
,dEnd => TO_DATE('20001231','YYYYMMDD')
,vcFrom => AGGREGATOR_STU
,vcTo => AGGREGATOR_EMU
,vcRegion => NULL
,vccontinent => NULL
,nregionlevel => NULL
);
Aggregator.Aggregation
(vcTheme => THEME_SIMULATION
,vcModel => MODEL_WOFOST
,vcMeteo => METEO_OPERATIONAL
,nCrop => NULL
,dStart => TO_DATE('19990101','YYYYMMDD')
,dEnd => TO_DATE('20001231','YYYYMMDD')
,vcFrom => AGGREGATOR_EMU
,vcTo => AGGREGATOR_GRID
,vcRegion => NULL
,vccontinent => NULL
,nregionlevel => NULL
);
Aggregator.Aggregation
(vcTheme => THEME_SIMULATION
,vcModel => MODEL_WOFOST
,vcMeteo => METEO_OPERATIONAL
,nCrop => NULL
,dStart => TO_DATE('19990101','YYYYMMDD')
,dEnd => TO_DATE('20001231','YYYYMMDD')
,vcFrom => AGGREGATOR_EMU
,vcTo => AGGREGATOR_REGION_LOW
,vcRegion => NULL
,vccontinent => NULL
,nregionlevel => NULL
);
Aggregator.Aggregation
(vcTheme => THEME_SIMULATION
,vcModel => MODEL_WOFOST
,vcMeteo => METEO_OPERATIONAL
,nCrop => NULL
,dStart => TO_DATE('19990101','YYYYMMDD')
,dEnd => TO_DATE('20001231','YYYYMMDD')
,vcFrom => AGGREGATOR_REGION_LOW
,vcTo => AGGREGATOR_REGION_UPPER
,vcRegion => NULL
,vccontinent => NULL
,nregionlevel => 2
);
Aggregator.Aggregation
(vcTheme => THEME_SIMULATION
,vcModel => MODEL_WOFOST
,vcMeteo => METEO_OPERATIONAL
,nCrop => NULL
,dStart => TO_DATE('19990101','YYYYMMDD')
,dEnd => TO_DATE('20001231','YYYYMMDD')
,vcFrom => AGGREGATOR_REGION_LOW
,vcTo => AGGREGATOR_REGION_UPPER
,vcRegion => NULL
,vccontinent => NULL
,nregionlevel => 1
);
Aggregator.Aggregation
(vcTheme => THEME_WEATHER
,vcModel => NULL
,vcMeteo => METEO_REANALISYS
,nCrop => NULL
,dStart => TO_DATE('19990101','YYYYMMDD')
,dEnd => TO_DATE('20001231','YYYYMMDD')
,vcFrom => AGGREGATOR_GRID
,vcTo => AGGREGATOR_REGION_UPPER
,vcRegion => NULL
,vccontinent => NULL
,nregionlevel => NULL
);
Aggregator.Aggregation
(vcTheme => THEME_WEATHER
,vcModel => NULL
,vcMeteo => METEO_ANALISYS
,nCrop => NULL
,dStart => TO_DATE('19990101','YYYYMMDD')
,dEnd => TO_DATE('20001231','YYYYMMDD')
,vcFrom => AGGREGATOR_GRID
,vcTo => AGGREGATOR_REGION_UPPER
,vcRegion => NULL
,vccontinent => NULL
,nregionlevel => NULL
);
Aggregator.Aggregation
(vcTheme => THEME_WEATHER
,vcModel => NULL
,vcMeteo => METEO_DETERMINISTIC
,nCrop => NULL
,dStart => TO_DATE('19990101','YYYYMMDD')
,dEnd => TO_DATE('20001231','YYYYMMDD')
,vcFrom => AGGREGATOR_GRID
,vcTo => AGGREGATOR_REGION_UPPER
,vcRegion => NULL
,vccontinent => NULL
,nregionlevel => NULL
);
Aggregator.Aggregation
(vcTheme => THEME_WEATHER
,vcModel => NULL
,vcMeteo => METEO_FORECAST
,nCrop => NULL
,dStart => TO_DATE('19990101','YYYYMMDD')
,dEnd => TO_DATE('20001231','YYYYMMDD')
,vcFrom => AGGREGATOR_GRID
,vcTo => AGGREGATOR_REGION_UPPER
,vcRegion => NULL
,vccontinent => NULL
,nregionlevel => NULL
);
COMMIT;
END;
|
AMDAC
Models of Yield Production is one of the fields covered by the Agriculture Project of the Institute for Remote Sensing Applications at the Joint Research Centre of the Commission of the European Communities in Ispra (Italy). The goal of one of the studies in this field (see operation 3.2 in the MARS Project Call for Proposals: General Conditions and Detailed Specifications of August 1990) was to provide the Agriculture Project with a software package able to perform decoding, filing and quality evaluation of actual meteorological data which are used as input for agro-meteorological models. For this purpose the Actual Meteorological Database Construction (AMDaC) package has been developed by MeteoConsult (Wageningen, The Netherlands), which is described in this manual. AMDAC has been retired in January 2020 and is succeeded by the Quality Checks of Meteorological Data Software that was developed by the Joint Research Centre.
Calplat
Within the framework of the ASEMARS project, the calibration platform Calplat was developed, at Alterra, in close consultation with the Joint Research Centre (JRC) at Ispra. It was developed during 2005 and 2006, and was, in first instance, meant to calibrate the basic crop phenology parameters. Later Calplat has been extended to be able to calibrate a larger set of crop parameters for the WOFOST crop growth model and the LINGRA grass growth model, which are both part of the CGMS monitoring system.
More information
Calplat manual (pdf)
Calibration Manager
The calibration Manager is a Python packages that combines Python wofost (PCSE) with an open optimization tool NLopt. This makes it possible to calibrate a combination of crop parameters at the same time (e.g. TSUM1 and TSUM2, SPAN and SLATB and TDWI, etc.) using one or more target variables (e.g. day of anthesis, day of maturity, lai-max, harvest index etc.). The selected target variables are combined in a combined objective function that is optimized. Additional functionalities are added to the tool, such as normalizing target variables, assigning weights to experiments and calendars, criteria when expert knowledge is taken into account, applying additional crop masks to exclude regional observations of non agricultural areas.
More information
Calibration_Manager
CGMS
The CGMS is the combination of the WOFOST crop growth model, a relational database and a statistical yield prediction module. From 2004 onwards the development of CGMS continued in the framework of the MARSOP2, ASEMARS and MARSOP3 projects, leading to the current version CGMS 10.0.3.2. The linked document below describes version 9.2
| CGMS versions | ||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
More information
CGMS manual (pdf)
CGMS statistical tool
The CGMS statistical tool has been developed for JRC’s MARS project in the framework of the contract study “Actions in Support of the Enlargement of the MARS Crop Yield Forecasting System (MCYFS) Lot I (ASEMARS Lot I)”. The tool is designed for use by the crop analysts and is an improved version of the CGMS statistical module which was in use since 1994 to facilitate national and sub national crop yield forecasting. Time trend analyis of yield statistics is followed by regression or scenario analysis using biophysical indicators to explain yield statistics and search for similar years. Constructed models are used to predict yield of the current growing season.
More information
CGMS Statistical Tool manual (pdf)
CMETEO
The software package has been developed to aggregate grid weather to some levels of administrative and agri-environmental regions. It is a generic package that can be used with input datasets such as observed weather and forecasted weather of several models and regions of interest (ROIs) as supplied in the MARS database. For each of the input dataset the procedure is repeated using weighing methods based on occupied areas. The output of the process is merged into resulting datasets for the choosen theme and resolution.
More information
CMETEO
Control board
In order to facilitate the work of the MARS analysts the COBO (Control board) was developed. The COBO is a mixture of data warehousing and data search engines organised through a common interface based on web-portal concept. COBO represents a base tool for the analyst.
More information
CoBo_(Control_Board)_-_the_tool_used_for_statistical_forecasting
COPdate
To estimate grid specific sowing dates following sowing dates rules, the oracle package COPdate was developed
More information
COPdate
GIS interface
CGMS does not use or need a GIS to produce its results. However, a GIS is necessary for a meaningful presentation of the results, and is also indispensable for the initial creation of the database. The link to the GIS is formed by the concept of the EMU’s, grid cells, administrative regions(NUTS) and Agro-environmental zones. Meteorological data are stored in the resolutions 'grid', 'NUTS' and 'agro-environmental zones'. Simulated yields are stored in the resolutions 'EMU', 'grid' and 'NUTS'.
In the cgms database all grid cells, administrative regions and agri-environmental zones have a unique code or number that can be used to provide a link to a GIS. The EMU's are stored as a unique combination of grid number and SMU number. This unique combination can be used to provide the link to the GIS.
More information
GIS interface manual (pdf)
Remote Sensing Software
More information
REMOTE SENSING SOFTWARE
Image server
ECMWF weather data, aggregated for 10-daily periods and 10-daily and monthly remote sensing based indicators can be downloaded from image servers hosted by the JRC. Documentation, terms of use and download links can be found on the homepages of these image servers.
ISW
To estimate initial soil water content for crop simulations, the implemented approach is to start a water balance long before the actual crop simulation. Using this approach, recharge of the soil moisture by rainfall and water use of a pseudo crop simulation or bare soil will help to find a more representative estimate of the soil moisture level when the actual crop simulation for the year of interest starts.
More information
ISW
LTA_YIELD
LTA_YIELD is a dedicated software package that calculates long term averages of simulated crop yield for each individual geographical feature on a 10 daily base. The geographical features can be be EMU's, grid cells or adminstrative regions (NUTS).
More information
LTA_YIELD
Marsop viewer
The operational MARS services deliver and store large amounts of data. These data vary from static reference layers and input weather data originating from different supplying meteorological sources (weather stations, ECMWF) and data quick looks to the data that is generated in the various operational levels of the MARS services through downscaling, simulation of crop indicators, estimation of crop yields etcetera. The MARS viewers enable users to perform spatial and temporal analysis of these data sets in a customized way. The documentation of the Marsop3 viewer is split into two manuals: an analyst viewer version with full functionality and a web version with limited functionality.
PCSE
PCSE is the abbreviation for Python Crop Simulation Environment. PCSE is a Python package for building crop simulation models, in particular the crop models developed in Wageningen (Netherlands). PCSE provides the environment to implement crop simulation models, the tools for reading ancillary data (weather, soil, agromanagement) and the components for simulating biophysical processes such as phenology, respiration and evapotranspiration. PCSE also includes implementations of the WOFOST and LINTUL3.
Project Management Board
The Project Management Board (PMB) was developed to keep track of the operational processing activities. Within the MCYFS many production lines are operational. The source date come from different locations, generally in large quantities and high frequencies (daily-, decadal, monthly and seasonal updates). Different parties are involved in processing the data before they are injected in a database. And eventually most data is mirrored to a duplicate database elsewhere.
A single error could interrupt one of the production lines and may cause a complete dataset to be inaccessible (including derived results). A real-time overview of the states of the processing steps in the different production lines could help to quickly trace potential problems. This is what the PMB does. It consists of a database that stores all scheduled processing steps and their real-time status. The status of processing steps is updated by the various production lines automatically.
A web-based user interface can generate various overviews of processing steps. By default it displays the events that are delayed and not yet delivered. The user interface can also be used to manually update status information in case issues are resolved (and automatic status request can't be generated).
More information
PMB manual (pdf)
QUACKME
In 2018, the JRC developed the Quality Checks Meteorological Station Data Software (QUACKME) as a successor of AMDAC . The Technical Guide describes the architecture, the implemented data quality checks and the applied data aggregation methods. The Manual describes how the system can be operated.
ReferenceWeather
Oracle package (WEATHER_OBS_LTA) to calculate average station weather.
| Configuration |
|---|
|
First, the time window for looking back and forth can be set. Without setting the time window a default of 7 days (back and forth) is used; a total of 15 days. The window can be set by calling the function 'set_window_limit' with one argument. To calculated the long term averages, function 'stations' is called and configured using four arguments:
Calculation steps:
Important remarks:
|
SupitConstants
Supit constants are need to be available for all weather stations in the table WEATHER_STATION to be able to calculate solar radiation from other weather indicators as measured solar radiation is only seldom available. The SupitContstants application is developed to interpolated supit constants from the table SUPIT_REFERENCE_STATIONS to all weather stations and restuls are stored in the SUPIT_CONSTANTS table.
More information
SupitConstants
Supporting software
Within the MCYFS a number of software packages are developed that support processing steps in automated production lines. Some of these software packages can also be used as a separate tool by analysts.
More information
- REGLISTS (supplies specific lists of regions directed by input parameters)
- DATE_GENERATOR (generic tool to generate specific lists of dates according to the input parameters)
- PROCESS_LOGGING (procedures to send some info to a user interface)
- PROCMAN (generic tool to assist other procedures to run in heterogeneous environments)
