Difference between revisions of "CMETEO"

From Agri4castWiki
Jump to: navigation, search
(Installation of CMETEO, install the package)
 
(66 intermediate revisions by 3 users not shown)
Line 2: Line 2:
 
{{scientific}}
 
{{scientific}}
 
==Introduction==
 
==Introduction==
CMETEO stands for CORINE-meteo. The softwaretool CMETEO can be seen an application suite to aggregate data of meteo indicators as found in e.g. the CGMS_EUR_08.GRID_WEATHER dataset or similar data sets at grid resolutions. Think of meteo indicators on temperature (maximum, minimum, mean or daily), windspeed, precipitation, radiation, potential transpiration, evaporation etc. In the context of MARS, the weather indicators are given at a regular grid. These indicators will be aggregated to spatial resolutions like administrative regions (NUTS level 3-0) or agri-environmental regions according to 1) a general area weighted algorithm and 2) a landcover (or even crop) specific area weighted algorithm.
+
CMETEO stands for CORINE-meteo. The softwaretool CMETEO can be seen as an application suite to aggregate data of meteo indicators as found in e.g. the CGMS12EU.WEATHER_OBS_GRID data set or similar data sets at grid resolutions (meteo grid). Think of meteo indicators on temperature (maximum, minimum, mean or daily), windspeed, precipitation, radiation, potential transpiration, evaporation etc. In the context of MARS, the weather indicators are given at a regular grid. These indicators will be aggregated to spatial resolutions like administrative regions (NUTS level 3-0) or agri-environmental regions according to:
 +
* a general area weighted algorithm  
 +
* a landcover (or even crop) specific area weighted algorithm.
  
 
==The process==
 
==The process==
The process can be described in a short way as:
+
The process can be described in a short way as follows.
  
After starting, use the supplied [[#Interface|interface]] to initialise some more parameters holding values that are valid during the execution of the process.
+
When activated CMETEO first uses the supplied [[#Interface|interface]] parameters to initialise other parameters to steer the process.
* parameters according the user name, theme and spatial type: ROI, type of aggregation {plain, median -- in case of weather forecast members}, type of interval {day, dekad}, indication to include a crop specific aggregation {yes,no}, thresholds whether to include a grid cell based on its landcover or crop specific area {a comma seperated list of integer for each threshold needed, integers are in the range of: 0 up to 99}
+
* parameters according the user name (ROI), theme and spatial type (regions, agri-environmental zones):
 +
** type of aggregation {relevant in case of weather forecast members: plain (in case of one or no members; median (in case of 50/51 members)}
 +
** type of interval {day, dekad}
 +
** indication to include a crop specific aggregation {yes,no}
 +
** thresholds whether to include a grid cell based on its landcover or crop specific area {a comma seperated list of integer for each threshold needed, integers are in the range of: 0 up to 99}
 +
** get a list of all landcovers that are relevant for the ROI
 +
** get a list of all cmeteo-members, being a combination of day-offsets (days in forecast depth) and members of meteo-ensembles. In those cases where no weather forecast data apply, the cmeteo-members refers to only one member and no day-offset e.g. data sets like CMETEO_GRID_WEATHER (see for details the description of the [[#Details of the interface|interface]]).
 
* get a list of all days to process
 
* get a list of all days to process
* get a list of all landcovers that are relevant for the ROI
 
* get a list of all cmeteo-members, being day-offsets (days in forecast depth) and members of meteo-ensembles. Mostly, the cmeteo-members refers to only one member and no day-offset (e.g. data sets like CGMS_EUR_08.GRID_WEATHER).
 
  
 
Continue to process regions of level 0 (=countries) belonging to the ROI:
 
Continue to process regions of level 0 (=countries) belonging to the ROI:
 
* get a region (country)
 
* get a region (country)
* get all subregions of all levels contained in that region.
+
* get all subregions of all levels linked to that region.
 
* gather input for each subregion:
 
* gather input for each subregion:
 
** initialise an array of weather indicators for all days, all landcovers and all members of cmeteo (day-offsets and member_no's).
 
** initialise an array of weather indicators for all days, all landcovers and all members of cmeteo (day-offsets and member_no's).
 
** for each day:
 
** for each day:
*** collect available weather indicators of all cmeteo-members for the grid cells contained in the subregion.
+
*** collect available weather indicators of all cmeteo-members for the grid cells within the subregion.
 
*** for each threshold:
 
*** for each threshold:
**** if the size of area is above the threshold, summarise them per cmeteo-member and landcover. The values are weighted with the landcover specific area of the grid cell.  
+
**** if the size of area is above the threshold, summarise the values per cmeteo-member and landcover. The values are weighted for the landcover specific area of the grid cell.  
 
**** optionally, add a crop specific aggregation, weighted for the area of the crop in the region (note that crop area statistics are only available at administrative/region level and thus can only be used in aggregations from NUTS level 3 to higher levels)
 
**** optionally, add a crop specific aggregation, weighted for the area of the crop in the region (note that crop area statistics are only available at administrative/region level and thus can only be used in aggregations from NUTS level 3 to higher levels)
**** add this collection of data to the region that contains this subregion to facilitate the aggregation of the regions at higher levels.
+
**** add the results to the parent region (that contains this subregion) to facilitate aggregation of the regions at higher levels.
  
 
As a result an array is filled with aggregated weather indicators per (sub)region, landcover (or even crop), day and cmeteo-member.
 
As a result an array is filled with aggregated weather indicators per (sub)region, landcover (or even crop), day and cmeteo-member.
Line 35: Line 41:
 
==Environment==
 
==Environment==
  
The CMETEO package is designed to operate in several environments which may differ in both input (different tables names, definitions, structure, time resolution, etc.) as output datasets (different type of adminstrative resolutions, agri-environmental zones etc). All these environments need to have the basic data (gridded weather, list of regions, land cover or even crop specific areas per grid cell) and a number of CMETEO database objects available for the CMETEO process.
+
The CMETEO package is designed to operate in several environments (i.e. database schemas) which may differ in both input (different tables names, definitions, structure, interval of time, etc.) and output datasets (different type of spatial resolutions, agri-environmental zones etc). All these environments need to have the basic data (gridded weather, list of regions, land cover or even crop specific areas per grid cell) and a number of CMETEO database objects available for the CMETEO process.
  
 
==Software tools needed by CMETEO==
 
==Software tools needed by CMETEO==
CMETEO uses functionality which is implemented in several other software tools. The major ones are listed in the following table.
+
The essential functionality of the process is implemented in the packages: CMETEO_admin and CMETEO_process. Besides CMETEO uses functionality which is implemented in several other software tools. These ones are listed in the following table.
  
 
{|class="wikitable"
 
{|class="wikitable"
 
!tool!!remarks
 
!tool!!remarks
 
|-
 
|-
|[[REGLISTS]] || supplies specific lists of regions directed by input parameters defined in DB schema FORALL.  
+
|[[REGLISTS]] || supplies specific lists of regions directed by input parameters.
 
|-
 
|-
|[[MRS_PROCESS]] || contains common used functionality for processing more complex programs like CMETEO in DB schema MRSMAN.
+
|[[DATE_GENERATOR]] || generic tool to generate lists of dates and some additional attributes based on the input parameters.
 
|-
 
|-
|[[ADD_PROCESS_LOG]] || procedure to send some info to a logtable in DB schema XMRSMAN.
+
|[[PROCMAN]] || package serving configurations for complex programs like CMETEO.
 
|-
 
|-
|[[ADD_PROCESS_METRICS]] || procedure to send some data, retrieved during processing, to a logtable in DB schema XMRSMAN.
+
|[[PROCESS LOGGING]] || set of procedures to send info to the user interface (you may extend these to send to log tables).
 
|-
 
|-
|[[UPDATE_EVENTS_ARCHIVE]] || procedure, located in the PMB (Project Management Board) DB schema, to signal the successful end of processing.
+
|[[UPDATE_EVENTS_ARCHIVE]] || procedure to signal the PMB (Project Management Board) that the processing ended successfully.
 +
|}
 +
 
 +
==Installation of CMETEO==
 +
 
 +
To install CMETEO you need to do the following steps:
 +
* check the environment, if the objects needed by the CMETEO package are available (see [[#Software tools needed by CMETEO|Software tools needed by CMETEO]])
 +
* install the objects of the CMETEO package
 +
* add configurations to PROCMAN for any schema you wish to process CMETEO on
 +
 
 +
In the next paragraphs these steps will be detailed.
 +
 
 +
CMETEO will be installed in the database. So you need to decide in which schema you wish to install. Currently, CMETEO is installed in the MRSMAN schema, which is intended to be the base of applications to process datasets that are in other schemas. As CMETEO is such an application (serving different environments), it is a good choice to install it in the MRSMAN schema. However, it should not be installed in a schema (storing the gridded weather) that is to be processed by CMETEO.
 +
 
 +
Tools or objects needed by CMETEO don't have to be installed necessarily in the same schema as CMETEO. It is sufficient to grant access to these [[#Software tools needed by CMETEO|objects]] to the owner/schema in which CMETEO will be installed.
 +
 
 +
===Installation of CMETEO, check the environment===
 +
 
 +
Check that the following tools or objects are valid and available:
 +
 
 +
Tools, currently installed in the FORALL schema:
 +
* REGLISTS (requires the tables REGION_MAPPINGS and REGION_MAPPINGS_PER_ROI)
 +
* DATE_GENERATOR
 +
 
 +
Tools, currently installed in the MRSMAN schema:
 +
* PROCMAN
 +
* PROCESS LOGGING
 +
Plus the objects TD_NUMBER and TD_ERRORS.
 +
 
 +
Tool, currently installed in the PMB schema:
 +
* UPDATE_EVENTS_ARCHIVE
 +
 
 +
===Installation of CMETEO, install the package===
 +
 
 +
CMETEO is delivered as a package containing scripts for the application software and accompanying objects. They must be installed in this order:
 +
* threshold_value_tp.sql
 +
* all: cmeteo_...._tbl.sql
 +
* cmeteo_region_mappings_vw.sql
 +
* cmeteo_process_pks.sql
 +
* cmeteo_process_pkb.sql
 +
* cmeteo_admin_pks.sql
 +
* cmeteo_admin_pkb.sql
 +
 
 +
In each target db schema (with gridded weather) the user has to install a synonym CMETEO_ADMIN and some synonyms for installed types running the following scripts:
 +
* cmeteo_admin_syn.sql
 +
* td_threshold_value_syn.sql
 +
* td_threshold_values_syn.sql
 +
 
 +
Also, in each target schema some views have to be installed. It's not to say beforehand how this should be created. It depends on the availability of tables in the schema. For schema CGMS12EU scripts are supplied as a part of the installation to be installed.
 +
 
 +
===Installation of CMETEO, configure for schema===
 +
 
 +
CMETEO uses configurations in the PROCMAN tool to do the processing of weather indicators for each database schema in a proper way. Configuration in PROCMAN guides CMETEO to:
 +
* check the availability of datasets (tables, views) needed for the processing
 +
* create standardized entries for these datasets (tables, views)
 +
* get settings for some details while processing
 +
 
 +
Of course, the datasets and the settings must be in accordance with each other. Thus, for each set to be processed a 'process' must be defined which declares the datasets involved and the specification of the details. Several processes can be declared to be executed per schema. E.g. in schema CGMS12EU two processes are declared, named cmeteo-cgms12eu-regions and cmeteo-cgms12eu-zones.
 +
 
 +
For the details some values need to be defined e.g.:
 +
* aggregation method
 +
* event ID for Project Management Board
 +
* list of threshold-indicators to be produced in the spatial aggregation
 +
* etc.
 +
 
 +
In order to check availability of datasets (e.g. for the process of cmeteo-cgms12eu-regions) required datasets need to be defined in PROCMAN, table PRCMN_PROCESS_OBJECTS, with specifications (name, type, owner etc.) of (e.g. CGMS12EU for aggregation to administrative regions):
 +
* CMETEO_AGGREGATION_AREAS
 +
* CMETEO_REGION_GRID_LANDCOVERS
 +
* CMETEO_WEATHER_OBS_GRID
 +
* CMETEO_WEATHER_OBS_REGION (also check for trigger CMETEO_WEATHER_OBS_REGION_IO)
 +
 
 +
To make CMETEO run the process (e.g. cmeteo-cgms12eu-regions), additional object names need to be defined in PROCMAN, table PRCMN_PROCESS_OBJECTS to create (dynamic) synonyms:
 +
* CMETEO_CROPS
 +
* CMETEO_GRID_LANDCOVER_AREAS
 +
* CMETEO_GRID_WEATHER
 +
* CMETEO_LANDCOVERS
 +
* CMETEO_PROCESS
 +
* CMETEO_REGIONS
 +
* CMETEO_REGION_MAPPINGS
 +
* CMETEO_REGION_WEATHER
 +
* CMETEO_REJECTED_WEATHER
 +
 
 +
Finally some parameters needs to be set (table PRCMN_CONFIGURATIONS). The PROCMAN tool is able to use default settings for all parameters needed. So you only should specify values for some parameters that are specific for the process.
 +
 
 +
For CMETEO a limited set of parameters is used. In the table below these paramaeters are listed with their default value and a description with some hints to use.
 +
 
 +
{|class="wikitable"
 +
!PARAMETER_GROUP!!PARAMETER!!ARGUMENT_TYPE!!DEFAULT VALUE!!DESCRIPTION
 
|-
 
|-
|[[DATE_GENERATOR]] || generic tool to generate specific lists of dates according to the input parameters defined in DB schema FORALL.
+
| ||roi||text||replace with a non-virtual roi||The code of a Region of Interest (ROI) for which the aggregation has to take place. Choose one of the ROIs in the table Region_mappings_per_roi in the FORALL schema.
 +
|-
 +
| ||interval dates||text||day||An indicator for the interval between successive dates of gridded weather. Meaningful values are {day, dekade}.
 +
|-
 +
| ||day in interval||text||day1||Indicates the position of the date in the interval, as found in the input (gridded weather), in the interval (see: interval dates). Meaningful values are {day1, last day}. If the interval is day, then only day1 is to be used.
 +
|-
 +
| ||aggregation method||text||plain||Indicates the method to derive the aggregation. Useful values are {plain, median}. Plain just gathers the values for each grid and day and aggregates them. Median first takes the median values of a set of members, then aggregates them.
 +
|-
 +
| ||include crops||text||no||An indicator to add aggregations weighted on areas of crops (rom aggregation areas), not being land covers. You can choose: {yes, no}
 +
|-
 +
| ||thresholds grid||set of integer||0,5||An integer, or a set of integers separated by commas, to define a threshold to select the grids. A grid is taken into account if it's fraction occupied by the land cover is above the threshold. The threshold is expressed as a percentage (100 times the fraction). You can use any value(s) between 0 and 99. If you don't wish to apply a threshold you just set 0.
 +
|-
 +
| ||destination||text||name of table for weather||the name of dataset (table) in which the results finally will be stored. You may prefix if with the schema.
 +
|-
 +
| ||event id||number||-1||the identifier (number) of the entry in the Process Management Board (PMB), table EVENTS, that identifies a type of action. CMETEO uses this id to signal the PMB that the process has successfully completed. see the PMB manual.
 +
|-
 +
| ||group indicator||text||std||(Depricated) A code to guide the selection of regions to process. It's not used any more.
 +
|-
 +
| ||numeric fractions||set of strings||NIL||A name, or a set of names, of indicators that needs to be derived during spatial aggregation (events like rainy days, or temperature above a threshold (by comparing primary values with a threshold). You can supply any set of names. However, you need to supply details for each name in the set. See the examples of NIL in the parameter_group.
 +
|-
 +
|
 +
|-
 +
| NIL||source||text||zero||Indication of the source of a (scalar) value to compare with the threshold(s). In case of NIL (not relevant) it should be the string zero. In other cases (it is thematically relevant) an attribute of a table is used. Currently, you can use {rainfall, temp_max, temp_min, temp_avg}
 +
|-
 +
| NIL||operator||text||ne||Code for the comparison operator. In case of NIL (not relevant) the string ne is used. In other cases (it is thematically relevant) you can use one of {gt, ge, eq, ne, le, lt}.
 +
|-
 +
| NIL||thresholds||set of number||0||A number, or a set of numbers, used as thresholds to compare to the value as found in the source. In case of NIL (not relevant) the value 0 is used. In other cases (it is thematically relevant) it may be number(s) with a fractional part, e.g. 3.25. Numbers must be separated by commas.
 +
|-
 +
| NIL||unit type||text||unit||Code to define the threshold-indicators (how to process the result of the comparison). In case of NIL (not relevant) the string unit is used. In other cases (it is thematically relevant) you may just count them, so adding 1 to the aggregation (e.g. rainy days), or use the remains above (or below) the threshold to summarise (e.g. temperature above a threshold). Meaningful codes are {unit, remains}. Currently, unit results in a value of 0 or 1; remains delivers the remaining part of the source below or above the threshold.
 +
|-
 +
| NIL||format fraction number||text||9.99||The format mask for the value of the fraction to use in the resulting dataset. In case of NIL (not relevant) the string 9.99 is used. In other cases (it is thematically relevant) this mask should be equal to the comparative column in the table where the results will be written.
 +
|}
 +
 
 +
 
 +
The table below also lists the parameters and supplies values for processing weather indicators for Regions in CGMS12EU (as an example). As you see the default values remain in place if applicable.
 +
 
 +
{|class="wikitable"
 +
! PARAMETER_GROUP !! PARAMETER !! ARGUMENT_TYPE !! DEFAULT VALUE !! CGMS12EU (for process: cmeteo-cgms12eu-regions)
 +
|-
 +
| ||aggregation method||text||plain||
 +
|-
 +
| ||day in interval||text||day1||
 +
|-
 +
| ||destination||text||name of table for weather|| weather_obs_region
 +
|-
 +
| ||event id||number||-1|| 19
 +
|-
 +
| ||group indicator||text||std||
 +
|-
 +
| ||include crops||text||no||
 +
|-
 +
| ||interval dates||text||day||
 +
|-
 +
| ||numeric fractions||set of strings||NIL|| FRAC_AREA_TEMP_MAX, FRAC_AREA_TEMP_MIN, FRAC_AREA_PRECIPITATION, AVG_TEMP
 +
|-
 +
| ||roi||text||replace with a non-virtual roi|| EUR
 +
|-
 +
| ||thresholds grid||set of integer||0,5|| 0
 +
|-
 +
| NIL||format fraction number||text||9.99||
 +
|-
 +
| NIL||operator||text||ne||
 +
|-
 +
| NIL||source||text||zero||
 +
|-
 +
| NIL||thresholds||set of number||0||
 +
|-
 +
| NIL||unit type||text||unit||
 +
|-
 +
| FRAC_AREA_TEMP_MIN||format fraction number||text||  ||9.99
 +
|-
 +
| FRAC_AREA_TEMP_MIN||operator||text||  ||le
 +
|-
 +
| FRAC_AREA_TEMP_MIN||source||text||  ||temp_min
 +
|-
 +
| FRAC_AREA_TEMP_MIN||thresholds||set of number||  ||0,-8,-10,-18,-20
 +
|-
 +
| FRAC_AREA_TEMP_MIN||unit type||text||  ||unit
 +
|-
 +
| FRAC_AREA_TEMP_MAX||format fraction number||text||  ||9.99
 +
|-
 +
| FRAC_AREA_TEMP_MAX||operator||text||  ||ge
 +
|-
 +
| FRAC_AREA_TEMP_MAX||source||text||  ||temp_max
 +
|-
 +
| FRAC_AREA_TEMP_MAX||thresholds||set of number||  ||25,30,35,40
 +
|-
 +
| FRAC_AREA_TEMP_MAX||unit type||text||  ||unit
 +
|-
 +
| FRAC_AREA_TEMP_PRECIPITATION||format fraction number||text||  ||9.99
 +
|-
 +
| FRAC_AREA_TEMP_PRECIPITATION||operator||text||  ||ge
 +
|-
 +
| FRAC_AREA_TEMP_PRECIPITATION||source||text||  ||rainfall
 +
|-
 +
| FRAC_AREA_TEMP_PRECIPITATION||thresholds||set of number||  ||1,3,5,10,15,30
 +
|-
 +
| FRAC_AREA_TEMP_PRECIPITATION||unit type||text||  ||unit
 +
|-
 +
| AVG_TEMP||format fraction number||text||  ||99.9
 +
|-
 +
| AVG_TEMP||operator||text||  ||ge
 +
|-
 +
| AVG_TEMP||source||text||  ||temp_avg
 +
|-
 +
| AVG_TEMP||thresholds||set of number||  ||0,2,4,6,8,10
 +
|-
 +
| AVG_TEMP||unit type||text||  ||remains
 
|}
 
|}
  
Line 60: Line 260:
 
Before the process will be executed, some checks and tasks, mainly administrative, are done.
 
Before the process will be executed, some checks and tasks, mainly administrative, are done.
  
* The module checks if no other instance of CMETEO is active in it's schema and registers itself as active. At any one moment only at most one process per schema can be active.
+
* The module checks if no other instances of CMETEO are active in it's schema and registers itself as active. Only one process per schema can be active at once.
 
* After registering, it checks if all objects are available that are needed for the selected theme (including the ROI).
 
* After registering, it checks if all objects are available that are needed for the selected theme (including the ROI).
* It also redefines some synonyms (dynamic synonyms like CMETEO_REGION_WEATHER and CMETEO_GRID_LANDCOVER_AREAS) according to the demands of the theme.
+
* It also redefines some synonyms (dynamic synonyms like CMETEO_REGION_WEATHER and CMETEO_GRID_LANDCOVER_AREAS) according to the demands of the process.
 
* Before processing, it checks the availability of weather indicators for the selected interval.
 
* Before processing, it checks the availability of weather indicators for the selected interval.
* After processing, it collects counts of processed data and some other items and stores them into the metrics table (DB schema XMRSMAN).
+
* After processing, it collects counts of processed data and some other items and informs the user via the tool PROCESS LOGGING.
 
* It signals the Project Management Board that the process has successfully ended.
 
* It signals the Project Management Board that the process has successfully ended.
 
* Finally it releases the lock on active processes.
 
* Finally it releases the lock on active processes.
Line 73: Line 273:
  
 
To get CMETEO working properly, you supply:
 
To get CMETEO working properly, you supply:
* the name of the database schema that holds the data to be processed. Although the schema name is not defined as a parameter, this name is part of the interface. You supply this name as the user name when you login to the database to process CMETEO.
+
* the name of the database schema that holds the data to be processed (although the schema name is not defined as a parameter, this name is part of the interface). You supply this name as the user name when you login to the database to start CMETEO.
 
* p_theme: theme, an indication of the group of meteo indicators you wish to process. See the list below for possible themes.
 
* p_theme: theme, an indication of the group of meteo indicators you wish to process. See the list below for possible themes.
 
* p_spatial: spatial indicator, e.g. 'Regions', 'agri-environmental zones'. See the list below for possible alternatives.
 
* p_spatial: spatial indicator, e.g. 'Regions', 'agri-environmental zones'. See the list below for possible alternatives.
Line 87: Line 287:
  
 
{|class="wikitable"
 
{|class="wikitable"
|+ Themes and spatial indicators used by cmeteo process performed in schema:
+
|+ Themes and spatial indicators as used by CMETEO to perform the process in schema:
!schema: !! process: !! theme: !! spatial indicator:
+
!schema: !! process: !! theme: !! spatial level:
 
|-
 
|-
| cgfs_eur_09 || cmeteo-cgfs-eps-nuts || weather(simulated) eps || NUTS
+
| cgfs14eu  || cmeteo-cgfs14eu-eps-regions || weather(simulated) eps || Regions
 
|-
 
|-
| cgfs_eur_09 || cmeteo-cgfs-eps-zones || weather(simulated) eps || agri-environmental zones
+
| cgfs14eu  || cmeteo-cgfs14eu-eps-zones || weather(simulated) eps || agri-environmental zones
 
|-
 
|-
| cgfs_eur_09 || cmeteo-cgfs-his-nuts || weather(simulated) his || NUTS
+
| cgfs14eu || cmeteo-cgfs14eu-his-regions || weather(simulated) his || Regions
 
|-
 
|-
| cgfs_eur_09 || cmeteo-cgfs-his-zones || weather(simulated) his || agri-environmental zones
+
| cgfs14eu || cmeteo-cgfs14eu-his-zones || weather(simulated) his || agri-environmental zones
 
|-
 
|-
| cgfs_eur_09 || cmeteo-cgfs-ope-nuts || weather(simulated) ope || NUTS
+
| cgfs14eu  || cmeteo-cgfs14eu-ope-regions || weather(simulated) ope || Regions
 
|-
 
|-
| cgfs_eur_09 || cmeteo-cgfs-ope-zones || weather(simulated) ope || agri-environmental zones
+
| cgfs14eu  || cmeteo-cgfs14eu-ope-zones || weather(simulated) ope || agri-environmental zones
 
|-
 
|-
| cgms12eu || cmeteo-cgms12eu-nuts || weather(observed) || Regions
+
| cgms12eu || cmeteo-cgms12eu-regions || weather(observed) || Regions
 
|-
 
|-
 
| cgms12eu || cmeteo-cgms12eu-zones || weather(observed) || agri-environmental zones
 
| cgms12eu || cmeteo-cgms12eu-zones || weather(observed) || agri-environmental zones
 
|-
 
|-
 
| cgms_asia_08 || cmeteo-cgms-asia || Weather - CGMS - Analysis || Regions
 
| cgms_asia_08 || cmeteo-cgms-asia || Weather - CGMS - Analysis || Regions
|-
 
| cgms_eur_08 || cmeteo-cgms-nuts || weather(observed) || NUTS
 
|-
 
| cgms_eur_08 || cmeteo-cgms-zones || weather(observed) || agri-environmental zones
 
 
|-
 
|-
 
| gwsi_e_12 || cmeteo-gwsi_e || Weather - GWSI - Analysis || Regions
 
| gwsi_e_12 || cmeteo-gwsi_e || Weather - GWSI - Analysis || Regions
Line 118: Line 314:
  
  
* The theme must be supplied in p_theme. See the table above for possible alternatives. It is an extraction of the process table in MRSMAN.
+
* The theme must be supplied in p_theme. See the table above for possible alternatives. It is an extraction of the process table (PRCMN_PROCESSES) in MRSMAN.
* The spatial indicator must be supplied in p_spc_res. See the table above for possible alternatives. It is an extraction of the process table in MRSMAN.
+
* The spatial level must be supplied in p_spatial. See the table above for possible alternatives. It is an extraction of the process table (PRCMN_PROCESSES) in MRSMAN.
* (Optional) An interval or day to process in the set of: p_yr2start,p_dy2start,p_yr2end,p_dy2end. CMETEO uses several possibilities to define an interval. A coarse way is to mention a year to start and a year to end. In this case all days from the first of January of the starting year to the last day of December of the ending year will be processed. A more fine grained way is to provide a date to start and a date to end in which case only the days in between will be processed, or even one day if both entered dates are the same. You may supply any combination of years and dates. If you don't supply any, the default interval will be chosen, which starts at the first of January of the current year and runs up to the day before today. In fact, no day in the future will be processed.
+
* (Optional) An interval or day to process in the set of: p_yr2start (yyyy),p_dy2start (ddmmyyyy),p_yr2end (yyyy),p_dy2end (ddmmyyyy). CMETEO uses several possibilities to define an interval. A coarse way is to mention a year to start (p_yr2start) and a year to end (p_yr2end). In this case all days from the first of January of the starting year to the last day of December of the ending year will be processed. A more fine grained way is to provide a date to start (p_dy2start) and a date to end (p_dy2end) in which case only the days in between will be processed, or even one day if both entered dates are the same. You may supply any combination of years and dates. Note that in case you enter both years and dates the entered dates will overrule entered years. If you don't supply any, the default interval will be chosen, which starts at the first of January of the current year and runs up to the day before today. In fact, no day in the future will be processed.
* (Optional) an indicator to enable replacement of previously aggregated weather indicators in p_refresh, being one of 'yes', 'no'. The default is 'yes'. This one will be used when no value for this parameter has been supplied.  
+
* (Optional) an indicator (p_refresh) to enable update of previously aggregated weather indicators, being one of 'yes', 'no'. The default is 'yes'. This one will be used when no value for this parameter has been supplied.  
* A set identifiers of regions to be processed in qry_region. You may add a query or sql statement that delivers the preferred subset of regions for which you need cmeteo. The regions must be identified by their reg_map_id as found in table [[REGION_MAPPINGS]].
+
* (Optional) A set identifiers of regions (q_regmap) to be processed. You may add a query or sql statement that delivers the preferred subset of regions for which you need cmeteo. The regions must be identified by their reg_map_id as found in table {{Object|REGION_MAPPINGS}} in db schema FORALL.
 
If supplied, the selected set must contain some or all reg_map_ids of level 0 (!) for the relevant ROI,
 
If supplied, the selected set must contain some or all reg_map_ids of level 0 (!) for the relevant ROI,
 
e.g.: select rm.reg_map_id from region_mappings rm where rm.reg_level = 0 and rm.reg_code in ('LU', 'BE', 'NL');
 
e.g.: select rm.reg_map_id from region_mappings rm where rm.reg_level = 0 and rm.reg_code in ('LU', 'BE', 'NL');
Line 131: Line 327:
  
 
{|class="wikitable"
 
{|class="wikitable"
|+ Datasets used by cmeteo:
+
|+ Datasets (in CGMS12EU) used by CMETEO:
!dataset containing: !! base set in CGMS_EUR_08 !! intermediate set for cmeteo !! name in cmeteo !! remarks
+
!dataset containing: !! base set !! adapted for CMETEO!! name in CMETEO!! remarks
 
|-
 
|-
|weather indicators per grid                      || {{Object|GRID_WEATHER}}   || {{Object|CMETEO_GRID_WEATHER}} || {{Object|CMETEO_GRID_WEATHER}}  || input for cmeteo; intermediate set contains standardized attribute names
+
|weather indicators per gridcell for meteodata || {{Object|WEATHER_OBS_GRID}} || {{Object|CMETEO_WEATHER_OBS_GRID}} (view) || {{Object|CMETEO_GRID_WEATHER}}  || input for cmeteo; intermediate set contains standardized attribute names
 
|-
 
|-
|weather indicators per NUTS-region               || {{Object|NUTS_WEATHER}}   || {{Object|CMETEO_NUTS_WEATHER}} || {{Object|CMETEO_REGION_WEATHER}} || output of cmeteo; intermediate set contains standardized attribute names
+
|weather indicators per administrative region || {{Object|WEATHER_OBS_REGION}} || {{Object|CMETEO_WEATHER_OBS_REGION}} (view) || {{Object|CMETEO_REGION_WEATHER}} || output of cmeteo; intermediate set contains standardized attribute names
 
|-
 
|-
|weather indicators per agri-environmental region || {{Object|ZONAL_WEATHER}} || {{Object|CMETEO_ZONAL_WEATHER}} || {{Object|CMETEO_REGION_WEATHER}} || output of cmeteo; intermediate set contains standardized attribute names  
+
|weather indicators per agri-environmental region   || {{Object|WEATHER_OBS_AEZ}}   || {{Object|CMETEO_WEATHER_OBS_AEZ}} (view) || {{Object|CMETEO_REGION_WEATHER}} || output of cmeteo; intermediate set contains standardized attribute names  
 
|-
 
|-
|landcovers or crops to be evaluated             || {{Object|CROP_LANDCOVER}} || {{Object|CMETEO_LANDCOVERS}} || {{Object|CMETEO_LANDCOVERS}}    || input for cmeteo; intermediate set contains standardized attribute names
+
|landcovers or crops to be evaluated                 || {{Object|CMETEO_LANDCOVER_EU}} (in schema FORALL) || {{Object|CMETEO_LANDCOVERS}} (synonym) || {{Object|CMETEO_LANDCOVERS}}    || input for cmeteo; intermediate set contains standardized attribute names
 
|-
 
|-
|landcover areas intersection of grid and NUTS region || {{Object|GRID_PLUS_NUTS_LANDCOVER}} || {{Object|CMETEO_GRID_NUTS_LCV_AREAS}} || {{Object|CMETEO_GRID_LANDCOVER_AREAS}} || input for cmeteo; intermediate set contains standardized attribute names
+
|areas/surfaces for landcovers per grid and per admin. region || {{Object|LINK_REGION_GRID_LANDCOVER}} || {{Object|CMETEO_REGION_GRID_LANDCOVERS}} (view) || {{Object|CMETEO_GRID_LANDCOVER_AREAS}} || input for cmeteo; intermediate set contains standardized attribute names
 
|-
 
|-
|areas/surfaces for landcovers per grid and per agri-env. region || {{Object|GRID_PLUS_ENS_LANDCOVER}} || {{Object|CMETEO_GRID_ZONAL_LCV_AREAS}} || {{Object|CMETEO_GRID_LANDCOVER_AREAS}} || input for cmeteo; intermediate set contains standardized attribute names
+
|areas/surfaces for landcovers per grid and per agri-env. region || {{Object|LINK_AEZ_GRID_LANDCOVER}} || {{Object|CMETEO_AEZ_GRID_LANDCOVERS}} (view) || {{Object|CMETEO_GRID_LANDCOVER_AREAS}} || input for cmeteo; intermediate set contains standardized attribute names
 
|-
 
|-
| areas/surfaces for crops per NUTS region || {{Object|AGGREGATION_AREAS}} || {{Object|CMETEO_AGGREGATION_AREAS}} || {{Object|CMETEO_AGGREGATION_AREAS}} || input for cmeteo
+
| areas/surfaces for crops per administrative region || {{Object|AGGREGATION_AREAS}} || {{Object|CMETEO_AGGREGATION_AREAS}} (view) || {{Object|CMETEO_AGGREGATION_AREAS}} || input for cmeteo
 
|-
 
|-
 
| weather indicators per region which are rejected ||  ||  || {{Object|CMETEO_REJECTED_WEATHER}} || output of cmeteo, containing weather indicators per region and date that could not be stored in regular output dataset via cmeteo_region_weather
 
| weather indicators per region which are rejected ||  ||  || {{Object|CMETEO_REJECTED_WEATHER}} || output of cmeteo, containing weather indicators per region and date that could not be stored in regular output dataset via cmeteo_region_weather
 
|-
 
|-
| regions and their mappings  ||  || {{Object|CMETEO_REGION_MAPPINGS}} || {{Object|CMETEO_REGIONS}} || input for cmeteo  
+
| regions and their mappings  || {{Object|CMETEO_REGION_MAPPINGS}} (in schema MRSMAN) || {{Object|CMETEO_REGION_MAPPINGS}} (synonym) || {{Object|CMETEO_REGION_MAPPINGS}} || input for cmeteo
 +
|-
 +
| regions to process || {{Object|CMETEO_REGIONS}} (in schema MRSMAN) || {{Object|CMETEO_REGIONS}} (synonym) || {{Object|CMETEO_REGIONS}} || input/output for cmeteo  
 
|}
 
|}
 +
 +
 +
In various schemas datasets are available that contain weather indicators, e.g. for meteo gridcells or regions of some type. However, they do vary both in name of the set and in names of the attributes. Also the collection of the attributes may vary (with or without day_offsets, with or without members of the meteo ensemble). In fact, all the sets contain the indicators that are essential for the processing of CMETEO. To prevent adaption of the CMETEO application every time another dataset in a schema must be processed or a new schema will be included into the processing scheme of CMETEO, attention was paid to the standardisation of the structure of datasets both as input and as output. The standardized dataset for input is called CMETEO_GRID_WEATHER. For each variant of a dataset containing grid weather a view is created that presents all attributes needed for the CMETEO process, either as data selected from the dataset or a dummy value. CMETEO just reads this data.
 +
 +
For the output of CMETEO also a view is created for each variant of output. In some schemas several of such views can exist. The name of a view hints, in most cases, to the name of the dataset on which it is based. CMETEO uses a synonym CMETEO_REGION_WEATHER, denoting the proper view, to access the data and to write the results. At the start of the processing, CMETEO is able to create the synonym to point to the right view (for either regions or AEZ).
 +
  
  
Line 158: Line 362:
 
|-
 
|-
 
|
 
|
In various schemas, datasets containing weather indicators do vary both in name of the set and in names of the attributes. Also the collection of the attributes may vary (with or withouth meteo-members). However, all the sets contain the indicators that are essential to process CMETEO. The writing of results to objects is organized as follows. CMETEO_REGION_WEATHER is a replacement for a variety of objects to which CMETEO will write the aggregated results. The results of CMETEO must be placed into datasets with an appropriate spatial indication like NUTS regions or agri-environmental zones. In general, datasets for NUTS-regions are pre- or postfixed with 'NUTS'; datasets for agri_environmental zones are pre- or postfixed with 'ZONAL'. To serve both type of spatial indications the object CMETEO_REGION_WEATHER is implemented in a dynamic synonym that switches between administrative regions or agri-environmental zones depending on the selected target resolution.<br>
+
As CMETEO only can use one structure for grid weather, some logic must be applied to transform the various sets into unified representations. This transformation is implemented as a view on the original dataset containing meteo data of grid cells while renaming attributes and eventually supplying neutral data values for missing attributes (e.g. members and offset days). The dataset {{Object|CMETEO_GRID_WEATHER}} is composed of the most wide combination of attributes of the various underlying datasets.
As CMETEO only can use one structure for grid weather, some logic must be applied to transform the various sets into unified representations. This transformation is implemented as a view (standard selection) on each of the original datasets while renaming the attributes and eventually supplying neutral data values for missing attributes (e.g. members and offset days). The dataset CMETEO_GRID_WEATHER is composed of the most wide combination of attributes of the various underlying datasets.
 
  
Below a summary of some sets in various schemas and the definition of CMETEO_GRID_WEATHER
+
Below a summary of some sets in various schemas and the definition of {{Object|CMETEO_GRID_WEATHER}}
  
 
{|class="wikitable"
 
{|class="wikitable"
! !! !! !! !! !! !! !! !!domain
+
! !! !! !! !! !! !! !! CMETEO !!domain
 
|-
 
|-
 
!schema:
 
!schema:
|CGMS_EUR_08
+
|CGMS12EU
|CGFS_EUR_09
+
|CGFS14EU
|CGFS_EUR_09
+
|CGFS14EU
|CGFS_EUR_09
+
|CGFS14EU
 
|CGMS_ASIA_08
 
|CGMS_ASIA_08
 
|GWSI
 
|GWSI
Line 177: Line 380:
 
|-
 
|-
 
!dataset:
 
!dataset:
|grid_weather
+
|weather_obs_grid
|ecmwf_his_downscaled_data
+
|weather_his_grid
|ecmwf_eps_downscaled_data
+
|weather_eps_grid
|ecmwf_ope_downscaled_data
+
|weather_ope_grid
 
|ecmwf_downscaled_data
 
|ecmwf_downscaled_data
 
|grid_weather_model_10
 
|grid_weather_model_10
Line 187: Line 390:
 
|-
 
|-
 
!attributes:
 
!attributes:
|grid_no                 || grid_no         ||grid_no         ||grid_no         ||ecmwf_grid_no    ||grid_id              ||grid_no          ||NN number(10)
+
|grid_no             || grid_no         || grid_no         || grid_no         || ecmwf_grid_no    || grid_id              || grid_no          || NN number(10)
|-                                                                                                                      
+
|-
| ||day                 || mdate            ||mdate            ||mdate            ||mdate            ||year, dekade        ||day              ||NN date
+
| || day             || day            || day            || day            || mdate            || year, dekade_in_year || day              || NN date
|-                                                                                                                      
+
|-
| || - 1)               || - 1)             ||date_offset     ||date_offset     || - 1)            || - 1)                ||day_offset      ||NN number(10)
+
| || - 1)           || - 1)           || day_offset     || day_offset     || - 1)            || - 1)                || day_offset      || NN number(10)
|-                                                                                                                      
+
|-
| || - 2)               || - 2)             ||member_no       ||member_no       || - 2)            || - 2)                ||member_no        ||NN number(10)
+
| || - 2)           || - 2)           || member_no       || member_no       || - 2)            || - 2)                || member_no        || NN number(10)
|-                                                                                                                      
+
|-
| ||maximum_temperature  || temperature_max ||temperature_max ||temperature_max ||temperature_max  ||temperature_maximum  ||max_temperature  ||NN number(3,1)
+
| || temperature_max || temperature_max || temperature_max || temperature_max || temperature_max  || temperature_maximum  || max_temperature  || NN number(3,1)
|-                                                                                                                      
+
|-
| ||minimum_temperature  || temperature_min ||temperature_min ||temperature_min ||temperature_min  ||temperature_minimum  ||min_temperature  ||NN number(3,1)
+
| || temperature_min || temperature_min || temperature_min || temperature_min || temperature_min  || temperature_minimum  || min_temperature  || NN number(3,1)
|-                                                                                                                      
+
|-
| || - 3)                || temperature_mean ||temperature_mean ||temperature_mean ||temperature_mean ||temperature_mean    ||mean_temperature ||NN number(3,1)
+
| || temperature_avg || temperature_avg || temperature_avg || temperature_avg || temperature_mean || temperature_mean    || mean_temperature || NN number(3,1)
|-                                                                                                                      
+
|-
| || - 3)               || temperature_dew ||temperature_dew ||temperature_dew ||temperature_dew  || - 3)                ||dew_temperature  ||NN number(3,1)
+
| || - 3)           || temperature_dew || temperature_dew || temperature_dew || temperature_dew  || - 3)                || dew_temperature  || NN number(3,1)
|-                                                                                                                      
+
|-
| ||vapour_pressure      || vapour_pressure ||vapour_pressure ||vapour_pressure ||vapour_pressure  || - 3)                ||vapour_pressure  ||NN number(4,2)
+
| || humidity_06h    ||  - 4)          ||  - 4)          ||  - 4)          ||  - 4)            ||  - 4)                ||  - 4)            ||
|-                                                                                                                      
+
|-
| ||windspeed           || wind_speed       ||wind_speed       ||wind_speed       ||wind_speed      || - 3)                ||wind_speed      ||NN number(5,1)
+
| || humidity_09h    ||  - 4)          ||  - 4)          ||  - 4)          ||  - 4)            ||  - 4)                ||  - 4)            ||
|-                                                                                                                      
+
|-
| ||rainfall            || precipitation   ||precipitation   ||precipitation   ||precipitation    ||precipitation        ||precipitation    ||NN number(4,1)
+
| || humidity_12h    ||  - 4)          ||  - 4)          ||  - 4)          ||  - 4)            ||  - 4)                ||  - 4)            ||
|-                                                                                                                      
+
|-
| ||e0                   || e0               ||e0               ||e0               ||e0              || - 3)                ||e0              ||NN number(4,2)
+
| || humidity_15h    ||  - 4)          ||  - 4)          ||  - 4)          ||  - 4)            ||  - 4)                ||  - 4)            ||
|-                                                                                                                      
+
|-
| ||es0                 || es0             ||es0             ||es0             ||es0              || - 3)                ||es0              ||NN number(4,2)
+
| || humidity_18h    ||  - 4)          ||  - 4)          ||  - 4)          ||  - 4)            ||  - 4)                ||  - 4)            ||
|-                                                                                                                      
+
|-
| ||et0                 || et0             ||et0             ||et0             ||et0              ||et0                  ||et0              ||NN number(4,2)
+
| || vapourpressure  || vapourpressure || vapourpressure || vapourpressure || vapour_pressure  || - 3)                || vapour_pressure  || NN number(4,2)
|-                                                                                                                      
+
|-
| ||calculated_radiation || radiation_global ||radiation_global ||radiation_global ||radiation_global ||global_radiation    ||global_radiation ||NN number(6)
+
| || windspeed       || windspeed       || windspeed       || windspeed       || wind_speed      || - 3)                || wind_speed      || NN number(5,1)
|-                                                                                                                      
+
|-
| ||snow_depth          || snow_depth       ||snow_depth      ||snow_depth       ||snow_depth      ||snow_depth          ||snow_depth      ||number(6,2)
+
| || precipitation  || precipitation   || precipitation   || precipitation   || precipitation    || precipitation        || precipitation    || NN number(4,1)
 +
|-
 +
| || e0             || e0             || e0             || e0             || e0              || - 3)                || e0              || NN number(4,2)
 +
|-
 +
| || es0             || es0             || es0             || es0             || es0              || - 3)                || es0              || NN number(4,2)
 +
|-
 +
| || et0             || et0             || et0             || et0             || et0              || et0                  || et0              || NN number(4,2)
 +
|-
 +
| || radiation      || radiation      || radiation      || radiation      || radiation_global || global_radiation    || global_radiation || NN number(6)
 +
|-
 +
| || snowdepth      || snowdepth       || - 3)            || snowdepth       || snow_depth      || snow_depth          || snow_depth      || number(6,2)
 +
|-
 +
| ||  - 3)          || snowweq        || snowweq        || snowweq        ||  - 3)            ||  - 3)                || snow_weq        || number(6,2)
 
|}
 
|}
  
 +
 +
NN stands for: "not null" i.e. the attribute in this domain may not be filled with null, so must contain a value.<br>
 
1) If an attribute for the offset of the day isn't available, the day-offset in CMETEO_GRID_WEATHER will be filled with a neutral value (0). This value doesn't influence the processing.<br>
 
1) If an attribute for the offset of the day isn't available, the day-offset in CMETEO_GRID_WEATHER will be filled with a neutral value (0). This value doesn't influence the processing.<br>
 
2) For sets that doesn't contain an attribute for members, a member_no: 1 will be supplied in CMETEO_GRID_WEATHER.<br>
 
2) For sets that doesn't contain an attribute for members, a member_no: 1 will be supplied in CMETEO_GRID_WEATHER.<br>
 
3) A dummy value (0) for this weather indicator will be supplied to CMETEO_GRID_WEATHER for sets lacking such an attribute. CMETEO will process this value and produces a result for it. However these results will not be inserted into the output table.<br>
 
3) A dummy value (0) for this weather indicator will be supplied to CMETEO_GRID_WEATHER for sets lacking such an attribute. CMETEO will process this value and produces a result for it. However these results will not be inserted into the output table.<br>
 +
4) This attribute, as included in some dataset, is not processed in CMETEO.<br>
 
|}
 
|}
  
  
 
{|class="collapsing_table collapsible collapsed"
 
{|class="collapsing_table collapsible collapsed"
!Specifications of the CMETEO_REGION_WEATHER
+
!Specifications of CMETEO_REGION_WEATHER
 
|-
 
|-
 
|
 
|
CMETEO_REGION_WEATHER is a replacement for a variety of datasets, both for themes (e.g. CGFS-HIS, CGMS) and for spatial indications (e.g. NUTS regions, agri-environmental zones), to which CMETEO need to write the aggregated results. In general, datasets for NUTS-regions are pre- or postfixed with 'NUTS'; datasets for agri_environmental zones are pre- or postfixed with 'ZONAL'.<br>
+
CMETEO_REGION_WEATHER is a replacement for a variety of datasets for themes (e.g. cmeteo-cgms12eu-regions), to which CMETEO need to write the aggregated results. In general, datasets for regions are indicated with 'REGION'; datasets for agri_environmental zones are indicated with 'AEZ'.In various schemas, datasets containing weather indicators do vary both in name of the set and in names of the attributes. Also the collection of the weather indicators may vary. However, all the sets contain the essential indicators for which CMETEO delivers output. Per theme, the structure of datasets for both spatial levels are very similar. As CMETEO only can use one structure for aggregated weather, some logic must be applied to transform the various sets into unified representations. This transformation is implemented as a view (with the possibility to insert/update values via specific triggers) on each of the original datasets while renaming the attributes. As the dataset CMETEO_REGION_WEATHER delivers the most wide combination of attributes of the various underlying datasets, some precautions must be taken to prevent some useless attributes being placed into the final dataset. Functionality for this is located in a technical construct (trigger) which is connected to the views.<br>
In various schemas, datasets containing weather indicators do vary both in name of the set and in names of the attributes. Also the collection of the weather indicators may vary. However, all the sets contain the essential indicators for which CMETEO delivers output. Per theme, the structure of datasets for both spatial indications are very similar. As CMETEO only can use one structure for region weather, some logic must be applied to transform the various sets into unified representations. This transformation is implemented as a view (with the possibility to insert/update values via specific triggers) on each of the original datasets while renaming the attributes. As the dataset CMETEO_REGION_WEATHER delivers the most wide combination of attributes of the various underlying datasets, some precautions must be taken to prevent some useless attributes being placed into the final dataset. Functionality for this is located in a technical construct (trigger) which is connected to the views.<br>
+
 
In a dynamical way CMETEO_REGION_WEATHER will be linked to the proper dataset according to the theme and the appropriate spatial indication to deliver the results of CMETEO.
+
In a dynamical way CMETEO_REGION_WEATHER will be linked to the proper dataset (either region or AEZ) according to the theme and the appropriate spatial indication to deliver the results of CMETEO.
  
 
Below a summary of some sets in various schemas and the definition of CMETEO_REGION_WEATHER
 
Below a summary of some sets in various schemas and the definition of CMETEO_REGION_WEATHER
  
 
{|class="wikitable"
 
{|class="wikitable"
! !! !! !! !! !! !! !! !!domain
+
! !! !! !! !! !! !! !! CMETEO !!domain
 
|-
 
|-
 
!schema:
 
!schema:
|CGMS_EUR_08
+
|CGMS12EU
|CGFS_EUR_09
+
|CGFS14EU
|CGFS_EUR_09
+
|CGFS14EU
|CGFS_EUR_09
+
|CGFS14EU
 
|CGMS_ASIA_08
 
|CGMS_ASIA_08
 
|GWSI
 
|GWSI
Line 250: Line 468:
 
|-
 
|-
 
!datasets:
 
!datasets:
|nuts_weather
+
|weather_obs_region
|ecmwf_his_downscaled_nuts
+
|weather_his_region
|ecmwf_eps_downscaled_nuts
+
|weather_eps_region
|ecmwf_ope_downscaled_nuts
+
|weather_ope_region
|nuts_weather
+
|ecmwf_downscaled_region
 
|region_weather_model_10
 
|region_weather_model_10
 
|cmeteo_region_weather
 
|cmeteo_region_weather
Line 260: Line 478:
 
|-
 
|-
 
|
 
|
|zonal_weather
+
|weather_obs_aez
|ecmwf_his_downscaled_zonal
+
|weather_his_aez
|ecmwf_eps_downscaled_zonal
+
|weather_eps_aez
|ecmwf_ope_downscaled_zonal
+
|weather_ope_aez
 
|
 
|
 
|
 
|
Line 270: Line 488:
 
|-
 
|-
 
!attributes:
 
!attributes:
   |reg_map_id           || reg_map_id       ||reg_map_id       ||reg_map_id       ||reg_map_id       ||reg_map_id           ||reg_map_id      ||NN number(10)
+
   |reg_map_id     ||reg_map_id     ||reg_map_id     ||reg_map_id     ||reg_map_id       ||reg_map_id         ||reg_map_id      ||NN number(10)
|-                                                                                                                          
+
|-
| ||landcover_id         || landcover_id     ||landcover_id     ||landcover_id     ||landcover_id     ||landcover_id         ||landcover_id    ||NN number(10)
+
| ||landcover_id   ||landcover_id   ||landcover_id   ||landcover_id   ||landcover_id     ||landcover_id       ||landcover_id    ||NN number(10)
|-                                                                                                                          
+
|-
| ||threshold           || threshold       ||threshold       ||threshold       ||threshold         ||threshold           ||threshold        ||NN number(3)
+
| ||threshold       ||threshold       ||threshold       ||threshold       ||threshold       ||threshold           ||threshold        ||NN number(3)
|-                                                                                                                          
+
|-
| ||day                 || mdate            ||mdate            ||mdate            ||day              ||year, dekade        ||day              ||NN date
+
| ||day             ||day            ||day            ||day            ||mdate            ||year, dekad_in_year ||day              ||NN date
|-                                                                                                                          
+
|-
| || - 1)               || - 1)             ||date_offset     ||date_offset     || - 1)             || - 1)               ||day_offset      ||NN number(10)
+
| || - 1)           ||- 1)           ||day_offset     ||day_offset     || - 1)           || - 1)               ||day_offset      ||NN number(10)
|-                                                                                                                          
+
|-
| || - 2)               || - 2)             ||member_no       ||member_no       || - 2)             || - 2)               ||member_no        ||NN number(10)
+
| || - 2)           ||- 2)           ||member_no       ||member_no       || - 2)           || - 2)               ||member_no        ||NN number(10)
|-                                                                                                                      
+
|-
| ||maximum_temperature  || temperature_max ||temperature_max ||temperature_max ||max_temperature  ||temperature_maximum ||max_temperature  ||NN number(3,1)
+
| ||temperature_max ||temperature_max ||temperature_max ||temperature_max ||temperature_max  ||temperature_maximum ||max_temperature  ||NN number(3,1)
|-                                                                                                                      
+
|-
| ||minimum_temperature  || temperature_min ||temperature_min ||temperature_min ||min_temperature  ||temperature_minimum ||min_temperature  ||NN number(3,1)
+
| ||temperature_min ||temperature_min ||temperature_min ||temperature_min ||temperature_min  ||temperature_minimum ||min_temperature  ||NN number(3,1)
|-                                                                                                                      
+
|-
| ||daily_temperature    || temperature_mean ||temperature_mean ||temperature_mean ||daily_temperature ||temperature_mean     ||mean_temperature ||NN number(3,1)
+
| ||temperature_avg ||temperature_avg ||temperature_avg ||temperature_avg ||temperature_mean ||temperature_mean   ||mean_temperature ||NN number(3,1)
|-                                                                                                                          
+
|-
| || - 3)               || temperature_dew ||temperature_dew ||temperature_dew ||dew_temperature  || - 3)               ||dew_temperature  ||NN number(3,1)
+
| || - 3)           ||temperature_dew ||temperature_dew ||temperature_dew ||temperature_dew  || - 3)               ||dew_temperature  ||NN number(3,1)
|-                                                                                                                          
+
|-
| ||vapour_pressure      || vapour_pressure ||vapour_pressure ||vapour_pressure ||vapour_pressure   || - 3)               ||vapour_pressure  ||NN number(4,2)
+
| ||vapourpressure  ||vapourpressure ||vapourpressure ||vapourpressure ||vapour_pressure || - 3)               ||vapour_pressure  ||NN number(4,2)
|-                                                                                                                          
+
|-
| ||windspeed           || wind_speed       ||wind_speed       ||wind_speed       ||wind_speed       || - 3)               ||wind_speed      ||NN number(5,1)
+
| ||windspeed       ||windspeed       ||windspeed       ||windspeed       ||wind_speed       || - 3)               ||wind_speed      ||NN number(5,1)
|-                                                                                                                          
+
|-
| ||rainfall            || precipitation   ||precipitation   ||precipitation   ||precipitation     ||precipitation       ||precipitation    ||NN number(4,1)
+
| ||precipitation  ||precipitation   ||precipitation   ||precipitation   ||precipitation   ||precipitation       ||precipitation    ||NN number(4,1)
|-                                                                                                                          
+
|-
| ||e0                   || e0               ||e0               ||e0               ||e0               || - 3)               ||e0              ||NN number(4,2)
+
| ||e0             ||e0             ||e0             ||e0             ||e0               || - 3)               ||e0              ||NN number(4,2)
|-                                                                                                                          
+
|-
| ||es0                 || es0             ||es0             ||es0             ||es0               || - 3)               ||es0              ||NN number(4,2)
+
| ||es0             ||es0             ||es0             ||es0             ||es0             || - 3)               ||es0              ||NN number(4,2)
|-                                                                                                                          
+
|-
| ||et0                 || et0             ||et0             ||et0             ||et0               ||et0                 ||et0              ||NN number(4,2)
+
| ||et0             ||et0             ||et0             ||et0             ||et0             ||et0                 ||et0              ||NN number(4,2)
|-                                                                                                                          
+
|-
| ||calculated_radiation || radiation_global ||radiation_global ||radiation_global ||global_radiation  ||global_radiation     ||global_radiation ||NN number(6)
+
| ||radiation      ||radiation      ||radiation      ||radiation      ||radiation_global ||global_radiation   ||global_radiation ||NN number(6)
|-                                                                                                                          
+
|-
| ||snow_depth           || snow_depth      ||snow_depth       ||snow_depth      ||snow_depth        ||snow_depth           ||snow_depth      ||number(6,2)
+
| ||snowdepth      ||snowdepth      || - 3)           ||snowdepth      ||snow_depth      ||snow_depth         ||snow_depth      ||number(6,2)
 +
|-
 +
| || - 3)           ||snowweq        ||snowweq        ||snowweq        ||snowweq          || - 3)              ||snow_weq        ||number(6,2)
 +
|-
 +
| ||fractions 4)    ||fractions 4)    ||fractions 4)    ||fractions 4)    ||nil-fractions 5) ||nil-fractions 5)    ||fractions 4)    ||td_threshold_values
 
|}
 
|}
  
 +
NN stands for: "not null" i.e. the attribute in this domain may not be filled with null, so must contain a value.<br>
 
1) The value for the offset of the day as produced by CMETEO will not be inserted into indicated sets.<br>
 
1) The value for the offset of the day as produced by CMETEO will not be inserted into indicated sets.<br>
 
2) The value for the member as produced by CMETEO will not be inserted into indicated sets.<br>
 
2) The value for the member as produced by CMETEO will not be inserted into indicated sets.<br>
 
3) As input for this attribute was lacking, the dummy value (0) produced by CMETEO will be omitted.<br>
 
3) As input for this attribute was lacking, the dummy value (0) produced by CMETEO will be omitted.<br>
 +
4) This dataset also contains aggregated values of so-called threshold-indicators describing an event (rain, hot, cold or not) or a value corrected for a certain threshold (temperature sum). The daily aggregated value could be regarded as the fraction of this event (or the average temperature sum above a threshold) for the selected subregion. All these fractions are collected in a structure of varying length in the view based on this dataset.<br>
 +
5) This dataset does not contain the attributes as mentioned under 4). The so called nil-fractions delivers a dummy in the view based on this dataset.<br>
 
|}
 
|}
  
Line 317: Line 542:
 
|-
 
|-
 
|
 
|
CMETEO_AGGREGATION_AREAS is a replacement for some of datasets from which CMETEO needs to gather input concerning crop and region specific areas. Note this is only used if the user chooses to have an additional crop specific aggregation from lower to higher administrative regions.<br>
+
CMETEO_AGGREGATION_AREAS is a replacement for some of datasets of which CMETEO needs to gather input concerning crop and region specific areas. Note this is only used if the user chooses to have an additional crop specific aggregation from lower to higher administrative regions.<br>
  
 
Below a summary of some sets in various schemas and the definition of CMETEO_AGGREGATION_AREAS
 
Below a summary of some sets in various schemas and the definition of CMETEO_AGGREGATION_AREAS
  
 
{|class="wikitable"
 
{|class="wikitable"
! !! !! !! !! !! !!domain
+
! !! !! !! !! !! CMETEO !!domain
 
|-
 
|-
 
!schema:
 
!schema:
|CGMS_EUR_08
+
|CGMS12EU
|CGFS_EUR_09
+
|CGFS14EU
 
|CGMS_ASIA_08
 
|CGMS_ASIA_08
 
|GWSI
 
|GWSI
Line 333: Line 558:
 
|-
 
|-
 
!dataset:
 
!dataset:
|aggregation_areas
+
|aggregation_areas 1)
| - 1)
 
|aggregation_areas
 
 
| - 2)
 
| - 2)
 +
|aggregation_areas 1)
 +
| - 3)
 
|cmeteo_aggregation_areas
 
|cmeteo_aggregation_areas
 
|
 
|
 
|-
 
|-
 
!attributes:
 
!attributes:
   |crop_no             || ||crop_no        || ||crop_no          ||NN number(10)
+
   |crop_no                 || ||crop_no        || ||crop_no          ||NN number(10)
 
|-                                                                 
 
|-                                                                 
| ||nuts_code            || ||nuts_code      || ||reg_code        ||NN varchar2(10)
+
| ||reg_map_id > reg_code 4) || ||nuts_code      || ||reg_code        ||NN varchar2(10)
 
|-                                                                 
 
|-                                                                 
| ||year                 || ||year            || ||year            ||NN number(4)
+
| ||year                     || ||year            || ||year            ||NN number(4)
 
|-                                                                 
 
|-                                                                 
| ||area_cultivated     || ||area_cultivated || ||area_cultivated  ||number
+
| ||area_cultivated         || ||area_cultivated || ||area_cultivated  ||number
 
|-                                                                 
 
|-                                                                 
| ||source               || ||source          || ||source          ||NN varchar2(40)
+
| ||source                   || ||source          || ||source          ||NN varchar2(40)
|-                                             
 
| ||day                  || ||day            || ||                ||
 
|-                                             
 
| ||percentage          || ||percentage      || ||                ||
 
 
|}
 
|}
  
1) CGFS_EUR_09 uses the dataset of CGMS_EUR_08.<br>
+
1) not every attribute of the dataset will be used in CMETEO.<br>
2) For GWSI no aggregation for crops is planned. It uses the dummy dataset of MRSMAN.<br>
+
2) CGFS14EU uses the dataset of CGMS12EU.<br>
 +
3) For GWSI no aggregation for crops is planned. It uses the dummy dataset of MRSMAN.<br>
 +
4) As a remains of a former version of CMETEO, here the region code is needed in stead of the reg_map_id
 
|}
 
|}
  
Line 365: Line 588:
 
|-
 
|-
 
|
 
|
CMETEO_GRID_LANDCOVER_AREAS is a replacement for a some of datasets from which CMETEO needs to gather input concerning land cover and grid cell specific areas.<br>
+
CMETEO_GRID_LANDCOVER_AREAS is a replacement for a some of datasets of which CMETEO needs to gather input concerning land cover and grid cell specific areas.<br>
  
 
Below a summary of some sets in various schemas and the definition of CMETEO_GRID_LANDCOVER_AREAS
 
Below a summary of some sets in various schemas and the definition of CMETEO_GRID_LANDCOVER_AREAS
  
 
{|class="wikitable"
 
{|class="wikitable"
! !! !! !! !! !! !!domain
+
! !! !! !! !! !! CMETEO !!domain
 
|-
 
|-
 
!schema:
 
!schema:
|CGMS_EUR_08
+
|CGMS12EU
|CGFS_EUR_09
+
|CGFS14EU
 
|CGMS_ASIA_08
 
|CGMS_ASIA_08
 
|GWSI
 
|GWSI
Line 381: Line 604:
 
|-
 
|-
 
!datasets:
 
!datasets:
|grid_plus_nuts_landcover
+
|link_region_grid_landcover
 
| - 1)
 
| - 1)
 
| - 2)
 
| - 2)
Line 389: Line 612:
 
|-
 
|-
 
|
 
|
|grid_plus_ens_landcover
+
|link_aez_grid_landcover
 
| - 1)
 
| - 1)
 
|
 
|
Line 397: Line 620:
 
|-
 
|-
 
!attributes:
 
!attributes:
   |grid_no                 || || || ||grid_no          ||NN number(10)
+
   |grid_no       || ||grid_no 3)  ||grid_no      ||grid_no          ||NN number(10)
|-                                                
+
|-                                                                
| ||nuts_code/ens_code      || || || ||                 ||
+
| ||reg_map_id    || ||reg_map_id  ||reg_map_id  ||reg_map_id      ||NN number(10)
|-                                                
+
|-                                                                
| ||landcover_id           || || || ||landcover_id    ||NN number(3)
+
| ||landcover_id   || ||landcover_id ||landcover_id ||landcover_id    ||NN number(3)
|-                                                
+
|-                                                                
| ||area                   || || || ||area            ||NN number(10)
+
| ||area           || ||area        ||area        ||area            ||NN number(10)
|-                                                 
 
| ||reg_code (only in gpel) || || || ||                ||
 
|-                                 
 
| ||reg_map_id              || || || ||reg_map_id      ||NN number(10)
 
 
|}
 
|}
  
1) CGFS_EUR_09 uses the dataset of CGMS_EUR_08.<br>
+
1) CGFS14EU uses the dataset of CGMS12EU.<br>
2) In this schema the dataset is defined by a view on table FORALL.GRID_REGION_LANDCOVERS.<br>
+
2) In this schema the dataset is defined as a view on table FORALL.GRID_REGION_LANDCOVERS.<br>
 +
3) Some conversion on grid_no is done in order to get the right range of grid cells as used in the ROI ASIA.<br>
 
|}
 
|}
  
 
[[Category:Software Tools]]
 
[[Category:Software Tools]]

Latest revision as of 19:25, 24 February 2014



Introduction

CMETEO stands for CORINE-meteo. The softwaretool CMETEO can be seen as an application suite to aggregate data of meteo indicators as found in e.g. the CGMS12EU.WEATHER_OBS_GRID data set or similar data sets at grid resolutions (meteo grid). Think of meteo indicators on temperature (maximum, minimum, mean or daily), windspeed, precipitation, radiation, potential transpiration, evaporation etc. In the context of MARS, the weather indicators are given at a regular grid. These indicators will be aggregated to spatial resolutions like administrative regions (NUTS level 3-0) or agri-environmental regions according to:

  • a general area weighted algorithm
  • a landcover (or even crop) specific area weighted algorithm.

The process

The process can be described in a short way as follows.

When activated CMETEO first uses the supplied interface parameters to initialise other parameters to steer the process.

  • parameters according the user name (ROI), theme and spatial type (regions, agri-environmental zones):
    • type of aggregation {relevant in case of weather forecast members: plain (in case of one or no members; median (in case of 50/51 members)}
    • type of interval {day, dekad}
    • indication to include a crop specific aggregation {yes,no}
    • thresholds whether to include a grid cell based on its landcover or crop specific area {a comma seperated list of integer for each threshold needed, integers are in the range of: 0 up to 99}
    • get a list of all landcovers that are relevant for the ROI
    • get a list of all cmeteo-members, being a combination of day-offsets (days in forecast depth) and members of meteo-ensembles. In those cases where no weather forecast data apply, the cmeteo-members refers to only one member and no day-offset e.g. data sets like CMETEO_GRID_WEATHER (see for details the description of the interface).
  • get a list of all days to process

Continue to process regions of level 0 (=countries) belonging to the ROI:

  • get a region (country)
  • get all subregions of all levels linked to that region.
  • gather input for each subregion:
    • initialise an array of weather indicators for all days, all landcovers and all members of cmeteo (day-offsets and member_no's).
    • for each day:
      • collect available weather indicators of all cmeteo-members for the grid cells within the subregion.
      • for each threshold:
        • if the size of area is above the threshold, summarise the values per cmeteo-member and landcover. The values are weighted for the landcover specific area of the grid cell.
        • optionally, add a crop specific aggregation, weighted for the area of the crop in the region (note that crop area statistics are only available at administrative/region level and thus can only be used in aggregations from NUTS level 3 to higher levels)
        • add the results to the parent region (that contains this subregion) to facilitate aggregation of the regions at higher levels.

As a result an array is filled with aggregated weather indicators per (sub)region, landcover (or even crop), day and cmeteo-member. The records of the array will be merged into the database according the rules:

  • if the record does not contain any result (no aggregation), then delete it's former representation from the database if one exists
  • if the record contains a result (an aggregation), then update it's former representation in the database if one exists
  • if the record contains a result (an aggregation), then insert it into the database if no former representation exists

The procedure stops after processing all indicated regions of level 0 belonging to the ROI.

Environment

The CMETEO package is designed to operate in several environments (i.e. database schemas) which may differ in both input (different tables names, definitions, structure, interval of time, etc.) and output datasets (different type of spatial resolutions, agri-environmental zones etc). All these environments need to have the basic data (gridded weather, list of regions, land cover or even crop specific areas per grid cell) and a number of CMETEO database objects available for the CMETEO process.

Software tools needed by CMETEO

The essential functionality of the process is implemented in the packages: CMETEO_admin and CMETEO_process. Besides CMETEO uses functionality which is implemented in several other software tools. These ones are listed in the following table.

tool remarks
REGLISTS supplies specific lists of regions directed by input parameters.
DATE_GENERATOR generic tool to generate lists of dates and some additional attributes based on the input parameters.
PROCMAN package serving configurations for complex programs like CMETEO.
PROCESS LOGGING set of procedures to send info to the user interface (you may extend these to send to log tables).
UPDATE_EVENTS_ARCHIVE procedure to signal the PMB (Project Management Board) that the processing ended successfully.

Installation of CMETEO

To install CMETEO you need to do the following steps:

  • check the environment, if the objects needed by the CMETEO package are available (see Software tools needed by CMETEO)
  • install the objects of the CMETEO package
  • add configurations to PROCMAN for any schema you wish to process CMETEO on

In the next paragraphs these steps will be detailed.

CMETEO will be installed in the database. So you need to decide in which schema you wish to install. Currently, CMETEO is installed in the MRSMAN schema, which is intended to be the base of applications to process datasets that are in other schemas. As CMETEO is such an application (serving different environments), it is a good choice to install it in the MRSMAN schema. However, it should not be installed in a schema (storing the gridded weather) that is to be processed by CMETEO.

Tools or objects needed by CMETEO don't have to be installed necessarily in the same schema as CMETEO. It is sufficient to grant access to these objects to the owner/schema in which CMETEO will be installed.

Installation of CMETEO, check the environment

Check that the following tools or objects are valid and available:

Tools, currently installed in the FORALL schema:

  • REGLISTS (requires the tables REGION_MAPPINGS and REGION_MAPPINGS_PER_ROI)
  • DATE_GENERATOR

Tools, currently installed in the MRSMAN schema:

  • PROCMAN
  • PROCESS LOGGING

Plus the objects TD_NUMBER and TD_ERRORS.

Tool, currently installed in the PMB schema:

  • UPDATE_EVENTS_ARCHIVE

Installation of CMETEO, install the package

CMETEO is delivered as a package containing scripts for the application software and accompanying objects. They must be installed in this order:

  • threshold_value_tp.sql
  • all: cmeteo_...._tbl.sql
  • cmeteo_region_mappings_vw.sql
  • cmeteo_process_pks.sql
  • cmeteo_process_pkb.sql
  • cmeteo_admin_pks.sql
  • cmeteo_admin_pkb.sql

In each target db schema (with gridded weather) the user has to install a synonym CMETEO_ADMIN and some synonyms for installed types running the following scripts:

  • cmeteo_admin_syn.sql
  • td_threshold_value_syn.sql
  • td_threshold_values_syn.sql

Also, in each target schema some views have to be installed. It's not to say beforehand how this should be created. It depends on the availability of tables in the schema. For schema CGMS12EU scripts are supplied as a part of the installation to be installed.

Installation of CMETEO, configure for schema

CMETEO uses configurations in the PROCMAN tool to do the processing of weather indicators for each database schema in a proper way. Configuration in PROCMAN guides CMETEO to:

  • check the availability of datasets (tables, views) needed for the processing
  • create standardized entries for these datasets (tables, views)
  • get settings for some details while processing

Of course, the datasets and the settings must be in accordance with each other. Thus, for each set to be processed a 'process' must be defined which declares the datasets involved and the specification of the details. Several processes can be declared to be executed per schema. E.g. in schema CGMS12EU two processes are declared, named cmeteo-cgms12eu-regions and cmeteo-cgms12eu-zones.

For the details some values need to be defined e.g.:

  • aggregation method
  • event ID for Project Management Board
  • list of threshold-indicators to be produced in the spatial aggregation
  • etc.

In order to check availability of datasets (e.g. for the process of cmeteo-cgms12eu-regions) required datasets need to be defined in PROCMAN, table PRCMN_PROCESS_OBJECTS, with specifications (name, type, owner etc.) of (e.g. CGMS12EU for aggregation to administrative regions):

  • CMETEO_AGGREGATION_AREAS
  • CMETEO_REGION_GRID_LANDCOVERS
  • CMETEO_WEATHER_OBS_GRID
  • CMETEO_WEATHER_OBS_REGION (also check for trigger CMETEO_WEATHER_OBS_REGION_IO)

To make CMETEO run the process (e.g. cmeteo-cgms12eu-regions), additional object names need to be defined in PROCMAN, table PRCMN_PROCESS_OBJECTS to create (dynamic) synonyms:

  • CMETEO_CROPS
  • CMETEO_GRID_LANDCOVER_AREAS
  • CMETEO_GRID_WEATHER
  • CMETEO_LANDCOVERS
  • CMETEO_PROCESS
  • CMETEO_REGIONS
  • CMETEO_REGION_MAPPINGS
  • CMETEO_REGION_WEATHER
  • CMETEO_REJECTED_WEATHER

Finally some parameters needs to be set (table PRCMN_CONFIGURATIONS). The PROCMAN tool is able to use default settings for all parameters needed. So you only should specify values for some parameters that are specific for the process.

For CMETEO a limited set of parameters is used. In the table below these paramaeters are listed with their default value and a description with some hints to use.

PARAMETER_GROUP PARAMETER ARGUMENT_TYPE DEFAULT VALUE DESCRIPTION
roi text replace with a non-virtual roi The code of a Region of Interest (ROI) for which the aggregation has to take place. Choose one of the ROIs in the table Region_mappings_per_roi in the FORALL schema.
interval dates text day An indicator for the interval between successive dates of gridded weather. Meaningful values are {day, dekade}.
day in interval text day1 Indicates the position of the date in the interval, as found in the input (gridded weather), in the interval (see: interval dates). Meaningful values are {day1, last day}. If the interval is day, then only day1 is to be used.
aggregation method text plain Indicates the method to derive the aggregation. Useful values are {plain, median}. Plain just gathers the values for each grid and day and aggregates them. Median first takes the median values of a set of members, then aggregates them.
include crops text no An indicator to add aggregations weighted on areas of crops (rom aggregation areas), not being land covers. You can choose: {yes, no}
thresholds grid set of integer 0,5 An integer, or a set of integers separated by commas, to define a threshold to select the grids. A grid is taken into account if it's fraction occupied by the land cover is above the threshold. The threshold is expressed as a percentage (100 times the fraction). You can use any value(s) between 0 and 99. If you don't wish to apply a threshold you just set 0.
destination text name of table for weather the name of dataset (table) in which the results finally will be stored. You may prefix if with the schema.
event id number -1 the identifier (number) of the entry in the Process Management Board (PMB), table EVENTS, that identifies a type of action. CMETEO uses this id to signal the PMB that the process has successfully completed. see the PMB manual.
group indicator text std (Depricated) A code to guide the selection of regions to process. It's not used any more.
numeric fractions set of strings NIL A name, or a set of names, of indicators that needs to be derived during spatial aggregation (events like rainy days, or temperature above a threshold (by comparing primary values with a threshold). You can supply any set of names. However, you need to supply details for each name in the set. See the examples of NIL in the parameter_group.
NIL source text zero Indication of the source of a (scalar) value to compare with the threshold(s). In case of NIL (not relevant) it should be the string zero. In other cases (it is thematically relevant) an attribute of a table is used. Currently, you can use {rainfall, temp_max, temp_min, temp_avg}
NIL operator text ne Code for the comparison operator. In case of NIL (not relevant) the string ne is used. In other cases (it is thematically relevant) you can use one of {gt, ge, eq, ne, le, lt}.
NIL thresholds set of number 0 A number, or a set of numbers, used as thresholds to compare to the value as found in the source. In case of NIL (not relevant) the value 0 is used. In other cases (it is thematically relevant) it may be number(s) with a fractional part, e.g. 3.25. Numbers must be separated by commas.
NIL unit type text unit Code to define the threshold-indicators (how to process the result of the comparison). In case of NIL (not relevant) the string unit is used. In other cases (it is thematically relevant) you may just count them, so adding 1 to the aggregation (e.g. rainy days), or use the remains above (or below) the threshold to summarise (e.g. temperature above a threshold). Meaningful codes are {unit, remains}. Currently, unit results in a value of 0 or 1; remains delivers the remaining part of the source below or above the threshold.
NIL format fraction number text 9.99 The format mask for the value of the fraction to use in the resulting dataset. In case of NIL (not relevant) the string 9.99 is used. In other cases (it is thematically relevant) this mask should be equal to the comparative column in the table where the results will be written.


The table below also lists the parameters and supplies values for processing weather indicators for Regions in CGMS12EU (as an example). As you see the default values remain in place if applicable.

PARAMETER_GROUP PARAMETER ARGUMENT_TYPE DEFAULT VALUE CGMS12EU (for process: cmeteo-cgms12eu-regions)
aggregation method text plain
day in interval text day1
destination text name of table for weather weather_obs_region
event id number -1 19
group indicator text std
include crops text no
interval dates text day
numeric fractions set of strings NIL FRAC_AREA_TEMP_MAX, FRAC_AREA_TEMP_MIN, FRAC_AREA_PRECIPITATION, AVG_TEMP
roi text replace with a non-virtual roi EUR
thresholds grid set of integer 0,5 0
NIL format fraction number text 9.99
NIL operator text ne
NIL source text zero
NIL thresholds set of number 0
NIL unit type text unit
FRAC_AREA_TEMP_MIN format fraction number text 9.99
FRAC_AREA_TEMP_MIN operator text le
FRAC_AREA_TEMP_MIN source text temp_min
FRAC_AREA_TEMP_MIN thresholds set of number 0,-8,-10,-18,-20
FRAC_AREA_TEMP_MIN unit type text unit
FRAC_AREA_TEMP_MAX format fraction number text 9.99
FRAC_AREA_TEMP_MAX operator text ge
FRAC_AREA_TEMP_MAX source text temp_max
FRAC_AREA_TEMP_MAX thresholds set of number 25,30,35,40
FRAC_AREA_TEMP_MAX unit type text unit
FRAC_AREA_TEMP_PRECIPITATION format fraction number text 9.99
FRAC_AREA_TEMP_PRECIPITATION operator text ge
FRAC_AREA_TEMP_PRECIPITATION source text rainfall
FRAC_AREA_TEMP_PRECIPITATION thresholds set of number 1,3,5,10,15,30
FRAC_AREA_TEMP_PRECIPITATION unit type text unit
AVG_TEMP format fraction number text 99.9
AVG_TEMP operator text ge
AVG_TEMP source text temp_avg
AVG_TEMP thresholds set of number 0,2,4,6,8,10
AVG_TEMP unit type text remains

Administration

Before the process will be executed, some checks and tasks, mainly administrative, are done.

  • The module checks if no other instances of CMETEO are active in it's schema and registers itself as active. Only one process per schema can be active at once.
  • After registering, it checks if all objects are available that are needed for the selected theme (including the ROI).
  • It also redefines some synonyms (dynamic synonyms like CMETEO_REGION_WEATHER and CMETEO_GRID_LANDCOVER_AREAS) according to the demands of the process.
  • Before processing, it checks the availability of weather indicators for the selected interval.
  • After processing, it collects counts of processed data and some other items and informs the user via the tool PROCESS LOGGING.
  • It signals the Project Management Board that the process has successfully ended.
  • Finally it releases the lock on active processes.

Interface

The CMETEO package is designed to act in a batch file, or to start from a commandline. Both methods use the same procedure, CMETEO_ADMIN.DO_AGGREGATION to start with. This procedure defines the interface consisting of parameters, both obligatory or optional.

To get CMETEO working properly, you supply:

  • the name of the database schema that holds the data to be processed (although the schema name is not defined as a parameter, this name is part of the interface). You supply this name as the user name when you login to the database to start CMETEO.
  • p_theme: theme, an indication of the group of meteo indicators you wish to process. See the list below for possible themes.
  • p_spatial: spatial indicator, e.g. 'Regions', 'agri-environmental zones'. See the list below for possible alternatives.

N.B. you must supply the parameters for theme and spatial indicators exactly as they are defined.

Optionally, you can supply:

  • qry_region: a set identifiers of regions to be processed.
  • p_yr2start,p_yr2end,p_dy2start,p_dy2end: a day or an interval of days to be processed.
  • p_refresh: an indicator whether or not to refresh the previously aggregated weather indicators.

Details of the interface

Themes and spatial indicators as used by CMETEO to perform the process in schema:
schema: process: theme: spatial level:
cgfs14eu cmeteo-cgfs14eu-eps-regions weather(simulated) eps Regions
cgfs14eu cmeteo-cgfs14eu-eps-zones weather(simulated) eps agri-environmental zones
cgfs14eu cmeteo-cgfs14eu-his-regions weather(simulated) his Regions
cgfs14eu cmeteo-cgfs14eu-his-zones weather(simulated) his agri-environmental zones
cgfs14eu cmeteo-cgfs14eu-ope-regions weather(simulated) ope Regions
cgfs14eu cmeteo-cgfs14eu-ope-zones weather(simulated) ope agri-environmental zones
cgms12eu cmeteo-cgms12eu-regions weather(observed) Regions
cgms12eu cmeteo-cgms12eu-zones weather(observed) agri-environmental zones
cgms_asia_08 cmeteo-cgms-asia Weather - CGMS - Analysis Regions
gwsi_e_12 cmeteo-gwsi_e Weather - GWSI - Analysis Regions
gwsi_t_12 cmeteo-gwsi_t Weather - GWSI - Analysis Regions


  • The theme must be supplied in p_theme. See the table above for possible alternatives. It is an extraction of the process table (PRCMN_PROCESSES) in MRSMAN.
  • The spatial level must be supplied in p_spatial. See the table above for possible alternatives. It is an extraction of the process table (PRCMN_PROCESSES) in MRSMAN.
  • (Optional) An interval or day to process in the set of: p_yr2start (yyyy),p_dy2start (ddmmyyyy),p_yr2end (yyyy),p_dy2end (ddmmyyyy). CMETEO uses several possibilities to define an interval. A coarse way is to mention a year to start (p_yr2start) and a year to end (p_yr2end). In this case all days from the first of January of the starting year to the last day of December of the ending year will be processed. A more fine grained way is to provide a date to start (p_dy2start) and a date to end (p_dy2end) in which case only the days in between will be processed, or even one day if both entered dates are the same. You may supply any combination of years and dates. Note that in case you enter both years and dates the entered dates will overrule entered years. If you don't supply any, the default interval will be chosen, which starts at the first of January of the current year and runs up to the day before today. In fact, no day in the future will be processed.
  • (Optional) an indicator (p_refresh) to enable update of previously aggregated weather indicators, being one of 'yes', 'no'. The default is 'yes'. This one will be used when no value for this parameter has been supplied.
  • (Optional) A set identifiers of regions (q_regmap) to be processed. You may add a query or sql statement that delivers the preferred subset of regions for which you need cmeteo. The regions must be identified by their reg_map_id as found in table REGION_MAPPINGS in db schema FORALL.

If supplied, the selected set must contain some or all reg_map_ids of level 0 (!) for the relevant ROI, e.g.: select rm.reg_map_id from region_mappings rm where rm.reg_level = 0 and rm.reg_code in ('LU', 'BE', 'NL'); You also may supply a limited list of ids of region mappings which must be comma separated. If you leave this parameter empty, all regions of the ROI will be processed.

Several datasources work with the generic interface of CMETEO. The most important ones are listed in the table below.

Datasets (in CGMS12EU) used by CMETEO:
dataset containing: base set adapted for CMETEO name in CMETEO remarks
weather indicators per gridcell for meteodata WEATHER_OBS_GRID CMETEO_WEATHER_OBS_GRID (view) CMETEO_GRID_WEATHER input for cmeteo; intermediate set contains standardized attribute names
weather indicators per administrative region WEATHER_OBS_REGION CMETEO_WEATHER_OBS_REGION (view) CMETEO_REGION_WEATHER output of cmeteo; intermediate set contains standardized attribute names
weather indicators per agri-environmental region WEATHER_OBS_AEZ CMETEO_WEATHER_OBS_AEZ (view) CMETEO_REGION_WEATHER output of cmeteo; intermediate set contains standardized attribute names
landcovers or crops to be evaluated CMETEO_LANDCOVER_EU (in schema FORALL) CMETEO_LANDCOVERS (synonym) CMETEO_LANDCOVERS input for cmeteo; intermediate set contains standardized attribute names
areas/surfaces for landcovers per grid and per admin. region LINK_REGION_GRID_LANDCOVER CMETEO_REGION_GRID_LANDCOVERS (view) CMETEO_GRID_LANDCOVER_AREAS input for cmeteo; intermediate set contains standardized attribute names
areas/surfaces for landcovers per grid and per agri-env. region LINK_AEZ_GRID_LANDCOVER CMETEO_AEZ_GRID_LANDCOVERS (view) CMETEO_GRID_LANDCOVER_AREAS input for cmeteo; intermediate set contains standardized attribute names
areas/surfaces for crops per administrative region AGGREGATION_AREAS CMETEO_AGGREGATION_AREAS (view) CMETEO_AGGREGATION_AREAS input for cmeteo
weather indicators per region which are rejected CMETEO_REJECTED_WEATHER output of cmeteo, containing weather indicators per region and date that could not be stored in regular output dataset via cmeteo_region_weather
regions and their mappings CMETEO_REGION_MAPPINGS (in schema MRSMAN) CMETEO_REGION_MAPPINGS (synonym) CMETEO_REGION_MAPPINGS input for cmeteo
regions to process CMETEO_REGIONS (in schema MRSMAN) CMETEO_REGIONS (synonym) CMETEO_REGIONS input/output for cmeteo


In various schemas datasets are available that contain weather indicators, e.g. for meteo gridcells or regions of some type. However, they do vary both in name of the set and in names of the attributes. Also the collection of the attributes may vary (with or without day_offsets, with or without members of the meteo ensemble). In fact, all the sets contain the indicators that are essential for the processing of CMETEO. To prevent adaption of the CMETEO application every time another dataset in a schema must be processed or a new schema will be included into the processing scheme of CMETEO, attention was paid to the standardisation of the structure of datasets both as input and as output. The standardized dataset for input is called CMETEO_GRID_WEATHER. For each variant of a dataset containing grid weather a view is created that presents all attributes needed for the CMETEO process, either as data selected from the dataset or a dummy value. CMETEO just reads this data.

For the output of CMETEO also a view is created for each variant of output. In some schemas several of such views can exist. The name of a view hints, in most cases, to the name of the dataset on which it is based. CMETEO uses a synonym CMETEO_REGION_WEATHER, denoting the proper view, to access the data and to write the results. At the start of the processing, CMETEO is able to create the synonym to point to the right view (for either regions or AEZ).