CMETEO

From Agri4castWiki
Revision as of 15:07, 6 September 2013 by Henk (talk | contribs)
Jump to: navigation, search


Introduction

CMETEO stands for CORINE-meteo. The softwaretool CMETEO can be seen an application suite to aggregate data of meteo indicators as found in e.g. the CGMS_EUR_08.GRID_WEATHER dataset or similar data sets at grid resolutions. Think of meteo indicators on temperature (maximum, minimum, mean or daily), windspeed, precipitation, radiation, potential transpiration, evaporation etc. In the context of MARS, the weather indicators are given at a regular grid. These indicators will be aggregated to spatial resolutions like administrative regions (NUTS level 3-0) or agri-environmental regions according to 1) a general area weighted algorithm and 2) a landcover (or even crop) specific area weighted algorithm.

The process

The process can be described in a short way as:

After starting, use the supplied interface to initialise some more parameters holding values that are valid during the execution of the process.

  • parameters according the user name, theme and spatial type: ROI, type of aggregation {plain, median -- in case of weather forecast members}, type of interval {day, dekad}, indication to include a crop specific aggregation {yes,no}, thresholds whether to include a grid cell based on its landcover or crop specific area {a comma seperated list of integer for each threshold needed, integers are in the range of: 0 up to 99}
  • get a list of all days to process
  • get a list of all landcovers that are relevant for the ROI
  • get a list of all cmeteo-members, being day-offsets (days in forecast depth) and members of meteo-ensembles. Mostly, the cmeteo-members refers to only one member and no day-offset (e.g. data sets like CGMS_EUR_08.GRID_WEATHER).

Continue to process regions of level 0 (=countries) belonging to the ROI:

  • get a region (country)
  • get all subregions of all levels contained in that region.
  • gather input for each subregion:
    • initialise an array of weather indicators for all days, all landcovers and all members of cmeteo (day-offsets and member_no's).
    • for each day:
      • collect available weather indicators of all cmeteo-members for the grid cells contained in the subregion.
      • for each threshold:
        • if the size of area is above the threshold, summarise them per cmeteo-member and landcover. The values are weighted with the landcover specific area of the grid cell.
        • optionally, add a crop specific aggregation, weighted for the area of the crop in the region (note that crop area statistics are only available at administrative/region level and thus can only be used in aggregations from NUTS level 3 to higher levels)
        • add this collection of data to the region that contains this subregion to facilitate the aggregation of the regions at higher levels.

As a result an array is filled with aggregated weather indicators per (sub)region, landcover (or even crop), day and cmeteo-member. The records of the array will be merged into the database according the rules:

  • if the record does not contain any result (no aggregation), then delete it's former representation from the database if one exists
  • if the record contains a result (an aggregation), then update it's former representation in the database if one exists
  • if the record contains a result (an aggregation), then insert it into the database if no former representation exists

The procedure stops after processing all indicated regions of level 0 belonging to the ROI.

Environment

The CMETEO package is designed to operate in several environments which may differ in both input (different tables names, definitions, structure, time resolution, etc.) as output datasets (different type of adminstrative resolutions, agri-environmental zones etc). All these environments need to have the basic data (gridded weather, list of regions, land cover or even crop specific areas per grid cell) and a number of CMETEO database objects available for the CMETEO process.

Software tools needed by CMETEO

CMETEO uses functionality which is implemented in several other software tools. The major ones are listed in the following table.

tool remarks
REGLISTS supplies specific lists of regions directed by input parameters defined in DB schema FORALL.
MRS_PROCESS contains common used functionality for processing more complex programs like CMETEO in DB schema MRSMAN.
ADD_PROCESS_LOG procedure to send some info to a logtable in DB schema XMRSMAN.
ADD_PROCESS_METRICS procedure to send some data, retrieved during processing, to a logtable in DB schema XMRSMAN.
update_events_archive procedure, located in the PMB (Project Management Board) DB schema, to signal the successful end of processing.
DATE_GENERATOR generic tool to generate specific lists of dates according to the input parameters defined in DB schema FORALL.

Administration

Before the process will be executed, some checks and tasks, mainly administrative, are done.

  • The module checks if no other instance of CMETEO is active in it's schema and registers itself as active. At any one moment only at most one process per schema can be active.
  • After registering, it checks if all objects are available that are needed for the selected theme (including the ROI).
  • It also redefines some synonyms (dynamic synonyms like CMETEO_REGION_WEATHER and CMETEO_GRID_LANDCOVER_AREAS) according to the demands of the theme.
  • Before processing, it checks the availability of weather indicators for the selected interval.
  • After processing, it collects counts of processed data and some other items and stores them into the metrics table (DB schema XMRSMAN).
  • It signals the Project Management Board that the process has successfully ended.
  • Finally it releases the lock on active processes.

Interface

The CMETEO package is designed to act in a batch file, or to start from a commandline. Both methods use the same procedure, CMETEO_ADMIN.DO_AGGREGATION to start with. This procedure defines the interface consisting of parameters, both obligatory or optional.

To get CMETEO working properly, you supply:

  • the name of the database schema that holds the data to be processed. Although the schema name is not defined as a parameter, this name is part of the interface. You supply this name as the user name when you login to the database to process CMETEO.
  • p_theme: theme, an indication of the meteo indicators you wish to process. See some list of possible themes.
  • p_spatial: spatial indicator, e.g. 'Regions', 'Agri environmental zones'

Optionally, you can supply:

  • qry_region: a set identifiers of regions to be processed.
  • p_yr2start,p_yr2end,p_dy2start,p_dy2end: a day or an interval of days to be processed.
  • p_refresh: an indicator whether or not to refresh the previously aggregated weather indicators.

Details of the interface [suppply list of options per theme/ROI for p_theme and p_spc_res]

Themes and spatial indications used by cmeteo process performed in schema:
schema: process: theme: spatial indicator:
hh cmeteo_hh observed weather regions


  • The theme must be supplied in p_theme. Possible alternatives are listed in the process table of MRSMAN.
  • The spatial indicator must be supplied in p_spc_res. Possible alternatives are listed in the process table of MRSMAN.
  • (Optional) An interval or day to process in the set of: p_yr2start,p_dy2start,p_yr2end,p_dy2end. CMETEO uses several possibilities to define an interval. A coarse way is to mention a year to start and a year to end. In this case all days from the first of January of the starting year to the last day of December of the ending year will be processed. A more finegrained way is to provide a date to start and a date to end in which case only the days in between will be processed, or even one day if both entered dates are the same. You may supply any combination of years and dates. If you don't supply any, the default interval will be choosen, which starts at the first of January of the current year and runs up to the day before today. In fact, no day in the future will be processed.
  • (Optional) an indicator to enable replacement of previously aggregated weather indicators in p_refresh, being one of 'yes', 'no'. The default is 'yes'. This one will be used when no value for this parameter has been supplied.
  • A set identifiers of regions to be processed in qry_region. You may add a query or sql statement that delivers the preferred subset of regions for which you need cmeteo. The regions must be identified by their reg_map_id as found in table REGION_MAPPINGS.

If supplied, the selected set must contain some or all reg_map_ids of level 0 (!) for the relevant ROI, e.g.: select rm.reg_map_id from region_mappings rm where rm.reg_level = 0 and rm.reg_code in ('LU', 'BE', 'NL'); You also may supply a limited list of ids of region mappings which must be comma separated. If you leave this parameter empty, all regions of the ROI will be processed.

Several datasources work with the generic interface of CMETEO. The most important ones are listed in the table below [add intermediate column & add CMETEO_REGIONS].

Datasets used by cmeteo:
dataset containing example in CGMS_EUR_08 name in cmeteo remarks
weather indicators per grid GRID_WEATHER CMETEO_GRID_WEATHER input for cmeteo with standardized attribute names
weather indicators per NUTS-region NUTS_WEATHER CMETEO_NUTS_WEATHER CMETEO_REGION_WEATHER input for cmeteo with standardized attribute names
weather indicators per agri-environmental region ZONAL_WEATHER CMETEO_REGION_WEATHER input for cmeteo with standardized attribute names
landcovers or crops to be evaluated CROP_LANDCOVER CMETEO_LANDCOVERS input for cmeteo with standardized attribute names
landcover areas intersection of grid and NUTS region GRID_PLUS_NUTS_LANDCOVER CMETEO_GRID_NUTS_LCV_AREAS CMETEO_GRID_LANDCOVER_AREAS input for cmeteo with standardized attribute names
areas/surfaces for landcovers per grid and per agri-env. region GRID_PLUS_ENS_LANDCOVER CMETEO_GRID_ZONAL_LCV_AREAS CMETEO_GRID_LANDCOVER_AREAS input for cmeteo, renaming columns to a standardized naming
areas/surfaces for crops per NUTS region AGGREGATION_AREAS CMETEO_AGGREGATION_AREAS input for cmeteo, renaming columns to a standardized naming
weather indicators per region which are rejected CMETEO_REJECTED_WEATHER output for cmeteo, containing weather indicators per region that could not be stored in regular output via cmeteo_region_weather
regions and their mappings CMETEO_REGION_MAPPINGS input for cmeteo