Difference between revisions of "Talk:CMETEO"

Revision as of 14:41, 30 June 2011

Introduction

Cmeteo stands for CORINE-meteo. Cmeteo can be seen an application suite to aggregate weather indicators from grid resolution to a regional type of division. These indicators will be aggregated to spatial themas like NUTS or agri environmental regions, or translated/adjusted for landcovers of interest like crops, etc. Translation will be accompanied by aggregation. The cmeteo package is designed to operate in several environments which may differ in both input as output datasets. All these environments need to have the basic data (both input and output) available for the CMeteo process.

Software tools needed by cmeteo

Cmeteo uses functionality which is implemented in several other software tools. The major ones are listed in the following table.

tool	remarks
REGLISTS	supplies specific lists of regions directed by input parameters.
MRS_PROCESS	contains common used functionality for processing more complex programs like cmeteo.
ADD_PROCESS_LOG	procedure to send some info to a logtable.
ADD_PROCESS_METRICS	procedure to send some data, retrieved during processing, to a logtable.
MRS_STATS	generic tool to collect some metrics during processing.
MRS_HTTP	generic tool to handle some messages.
DATE_GENERATOR	generic tool to generate specific lists of dates according to the input paramaters.

Interface

Cmeteo uses many database objects. The objects that contain the primary datasets like weather data, landcover types and aggregation areas are listed in the table below:

object name	Content	Environment
CMETEO_GRID_WEATHER	Weather indicators per grid	CGMS / CGFS / ASIA / GWSI
CMETEO_EPS_GRID_WEATHER	Weather indicators per grid (ECMWF EPS model)	CGFS
CMETEO_HIS_GRID_WEATHER	Weather indicators per grid (ECMWF HIS model)	CGFS
CMETEO_OPE_GRID_WEATHER	Weather indicators per grid (ECMWF OPE model)	CGFS
CMETEO_REGION_WEATHER	Weather indicators per administrative region	CGMS / CGFS / ASIA / GWSI
CMETEO_NUTS_WEATHER	Weather indicators per NUTS region	CGMS
CMETEO_EPS_NUTS_WEATHER	Weather indicators per NUTS region (ECMWF EPS model)	CGFS
CMETEO_HIS_NUTS_WEATHER	Weather indicators per NUTS region (ECMWF HIS model)	CGFS
CMETEO_OPE_NUTS_WEATHER	Weather indicators per NUTS region (ECMWF OPE model)	CGFS
CMETEO_ZONAL_WEATHER	Weather indicators per Agri-environmental region	CGMS
CMETEO_EPS_ZONAL_WEATHER	Weather indicators per Agri-environmental region (ECMWF EPS model)	CGFS
CMETEO_HIS_ZONAL_WEATHER	Weather indicators per Agri-environmental region (ECMWF HIS model)	CGFS
CMETEO_OPE_ZONAL_WEATHER	Weather indicators per Agri-environmental region (ECMWF OPE model)	CGFS
CMETEO_AGGREGATION_AREAS	Cultivated areas per crop for Administrative region per year	CGMS / CGFS / ASIA / GWSI
CMETEO_GRID_LANDCOVER_AREAS	Areas for intersections of grid and administrative region	CGMS / CGFS / ASIA / GWSI
CMETEO_LANDCOVERS	Landcovers types or crops to be evaluated	CGMS / CGFS / ASIA / GWSI
CMETEO_GRID_NUTS_LCV_AREAS	Areas for intersections of grid and NUTS region	CGMS
CMETEO_GRID_ZONAL_LCV_AREAS	Areas for intersections of grid and Agri-environmental region	CGMS
CMETEO_REJECTED_WEATHER	Weather indicators per region that could not be stored via CMETEO_REGION_WEATHER	CGMS / CGFS / ASIA / GWSI
CMETEO_REGION_MAPPINGS	Administrative regions and their mappings (relations)	CGMS / CGFS / ASIA / GWSI

Supporting objects that are required by Cmeteo:

object name	Content	Environment
CMETEO_PROCESS		CGMS / CGFS / ASIA / GWSI
CMETEO		CGMS / CGFS / ASIA / GWSI
CMETEO_HIS_NUTS_WEATHER_IO		CGFS
CMETEO_HIS_ZONAL_WEATHER_IO		CGFS
CMET_REGION_WTH		ASIA / GWSI
CMET_NUTS_WTH		CGMS
CMET_ZONAL_WTH		CGMS
PRCMN_PARAMETERS		CGMS / CGFS / ASIA / GWSI
PRCMN_PROCEDURES		CGMS / CGFS / ASIA / GWSI
PRCMN_PROCESS_OBJECTS		CGMS / CGFS / ASIA / GWSI
PRCMN_PROCESS_RUNS		CGMS / CGFS / ASIA / GWSI
PRCMN_PROCESSES		CGMS / CGFS / ASIA / GWSI
PROCMAN		CGMS / CGFS / ASIA / GWSI

CMETEO_GRID_WEATHER

CMETEO_GRID_WEATHER is the object that contains the weather indicators in grid resolution that are needed as input for Cmeteo in a unified structure and with a uniform object name. In each environment (cgms / cgfs / asia / gwsi) the weather indicators in grid resolution are stored in tables with different names and structures. Also the collection of the attributes may vary. Cmeteo requires that the weather indicators in grid resolution are stored in an object with a specific structure and specific object name. Therefore some logic must be applied to transform the sets from various environments into a unified structure. This transformation is implemented by using views and/or synonyms on the original datasets while renaming the attributes and eventually suppliyng neutral or valuable data for missing attributes. The unified object CMETEO_GRID_WEATHER is composed of the most wide combination of attributes of the various underlying datasets.

Summery of structure conversion

Object name conversion

Environment	Primary data source	Intermediate object	Unified object
CGFS	ECMWF_EPS_DOWNSCALED_DATA (table)	CMETEO_EPS_GRID_WEATHER (view)	CMETEO_GRID_WEATHER (synonym)
CGFS	ECMWF_HIS_DOWNSCALED_DATA (table)	CMETEO_HIS_GRID_WEATHER (view)	CMETEO_GRID_WEATHER (synonym)
CGFS	ECMWF_OPE_DOWNSCALED_DATA (table)	CMETEO_OPE_GRID_WEATHER (view)	CMETEO_GRID_WEATHER (synonym)
ASIA	ECMWF_DOWNSCALED_DATA (table)		CMETEO_GRID_WEATHER (view)
CGMS	GRID_WEATHER (table)		CMETEO_GRID_WEATHER (view)
GWSI	GRID_WEATHER_MODEL_10 (table)		CMETEO_GRID_WEATHER (view)

The intermediate objects are views in the CGFS environment to convert the structure of the primary data source into a unified structure. They are alternately used by the synonym CMETEO_GRID_WEATHER to acquire the unified object name.

Structure conversion

Object name						domain
Environment	CGMS	CGFS	ASIA	GWSI
dataset:	grid_weather	ecmwf_his_downscaled_data	ecmwf_eps_downscaled_data	ecmwf_downscaled_data	cmeteo_grid_weather
attributes:	grid_no	grid_no	grid_no	ecmwf_grid_no	grid_no	NN number(10)
	day	mdate	mdate	mdate	day	NN date
	- 1)	- 1)	date_offset	- 1)	day_offset	NN number(10)
	- 2)	- 2)	member_no	- 2)	member_no	NN number(10)
	maximum_temperature	temperature_max	temperature_max	temperature_max	max_temperature	NN number(3,1)
	minimum_temperature	temperature_min	temperature_min	temperature_min	min_temperature	NN number(3,1)
	- 3)	temperature_mean	temperature_mean	temperature_mean	mean_temperature	NN number(3,1)
	- 4)	temperature_dew	temperature_dew	temperature_dew	dew_temperature	NN number(3,1)
	vapour_pressure	vapour_pressure	vapour_pressure	vapour_pressure	vapour_pressure	NN number(4,2)
	windspeed	wind_speed	wind_speed	wind_speed	wind_speed	NN number(5,1)
	rainfall	precipitation	precipitation	precipitation	precipitation	NN number(4,1)
	e0	e0	e0	e0	e0	NN number(4,2)
	es0	es0	es0	es0	es0	NN number(4,2)
	et0	et0	et0	et0	et0	NN number(4,2)
	calculated_radiation	radiation_global	radiation_global	radiation_global	global_radiation	NN number(6)
	snow_depth	snow_depth	snow_depth	snow_depth	snow_depth	number(6,2)

1) If an attribute for the offset of the day isn't available, the day-offset in cmeteo grid weather will be filled with a neutral value (0). This value doesn't influence the processing.
2) For sets that doesn't contain an attribute for members, a member_no: 1 will be supplied in cmeteo grid weather.
3) If an attribute for the mean or daily temperature doesn't exist, this value will be estimated as the mean value of minimum and maximum temperature of the day. Cmeteo will process this value and produces a result for it.
4) A dummy value (0) for the dew temperature wil be supplied to cmeteo grid weather for sets lacking such an attribute. Cmeteo will process this value and produces a result for it, so precautions must be taken to supply these results to the output table.

Specifications of cmeteo_region_weather

Cmeteo_region_weather is a replacement for a variety of datasets, both for themes (e.g. cgfs-his, cgms) and for resolutions (e.g. NUTS regions, agri-environmental zones), to which cmeteo need to fill up with output. In general, datasets for NUTS-regions are pre- or postfixed with 'nuts'; datasets for agri_environmental zones are pre- or postfixed with 'zonal'.
In various schemas, datasets containing weather indicators do vary both in name of the set and in names of the attributes. Also the collection of the attributes may vary. However, all the sets contain the essential indicators for which cmeteo delivers output. Per theme, the structure of datasets for both resolutions are very similar. As cmeteo only can use one structure for region weather, some logic must be applied to transform the various sets into unified representations. This transformation is implemented as a view (with the possibility to insert/update values) on each of the original datasets while renaming the attributes. As the dataset cmeteo_region_weather delivers the most wide combination of attributes of the various underlying datasets, some precautions must be taken to prevent some useless attributes being placed into the final dataset. Functionality for this is located in a technical construct (trigger) which is connected to the views.
In a dynamical way cmeteo_region_weather will be linked to the proper dataset according to the theme and the appropriate spacial resolution to deliver the results of cmeteo.

Below a summary of some sets in various schemas and the definition of cmeteo_region_weather

						domain
schema:	CGMS_EUR_08	CGFS_EUR_09	CGFS_EUR_09	CGMS_ASIA_08
datasets:	nuts_weather	ecmwf_his_downscaled_nuts	ecmwf_eps_downscaled_nuts	nuts_weather	cmeteo_region_weather
	zonal_weather	ecmwf_his_downscaled_zonal	ecmwf_eps_downscaled_zonal
attributes:	reg_map_id	reg_map_id	reg_map_id	reg_map_id	reg_map_id	NN number(10)
	landcover_id	landcover_id	landcover_id	landcover_id	landcover_id	NN number(10)
	threshold	threshold	threshold	threshold	threshold	NN number(3)
	day	mdate	mdate	day	day	NN date
	- 1)	- 1)	date_offset	- 1)	day_offset	NN number(10)
	- 2)	- 2)	member_no	- 2)	member_no	NN number(10)
	maximum_temperature	temperature_max	temperature_max	max_temperature	max_temperature	NN number(3,1)
	minimum_temperature	temperature_min	temperature_min	min_temperature	min_temperature	NN number(3,1)
	daily_temperature	temperature_mean	temperature_mean	daily_temperature	mean_temperature	NN number(3,1)
	- 3)	temperature_dew	temperature_dew	dew_temperature	dew_temperature	NN number(3,1)
	vapour_pressure	vapour_pressure	vapour_pressure	vapour_pressure	vapour_pressure	NN number(4,2)
	windspeed	wind_speed	wind_speed	wind_speed	wind_speed	NN number(5,1)
	rainfall	precipitation	precipitation	precipitation	precipitation	NN number(4,1)
	e0	e0	e0	e0	e0	NN number(4,2)
	es0	es0	es0	es0	es0	NN number(4,2)
	et0	et0	et0	et0	et0	NN number(4,2)
	calculated_radiation	radiation_global	radiation_global	global_radiation	global_radiation	NN number(6)
	snow_depth	snow_depth	snow_depth	snow_depth	snow_depth	number(6,2)

1) The value for the offset of the day as produced by cmeteo will not be inserted into indicated sets.
2) The value for the member as produced by cmeteo will not be inserted into indicated sets.
3) As input for this attribute was lacking cmeteo produced a dummy value (0). This value must be omitted.

Specifications of cmeteo_aggregation_areas

Cmeteo_aggregation_areas is a replacement for a some of datasets from which cmeteo need to gather input concerning sizes of surfaces.

Below a summary of some sets in various schemas and the definition of cmeteo_aggregation_areas

					domain
schema:	CGMS_EUR_08	CGFS_EUR_09	CGMS_ASIA_08
dataset:	aggregation_areas	- 1)	aggregation_areas	cmeteo_aggregation_areas
attributes:	crop_no		crop_no	crop_no	NN number(10)
	nuts_code		nuts_code	reg_code	NN varchar2(10)
	year		year	year	NN number(4)
	area_cultivated		area_cultivated	area_cultivated	number
	source		source	source	NN varchar2(40)
	day		day
	percentage		percentage

1) CGFS_EUR_09 uses the dataset of CGMS_EUR_08.

Specifications of cmeteo_grid_landcover_areas

Cmeteo_grid_landcover_areas is a replacement for a some of datasets from which cmeteo need to gather input concerning sizes of surfaces.

Below a summary of some sets in various schemas and the definition of cmeteo_grid_landcover_areas

					domain
schema:	CGMS_EUR_08	CGFS_EUR_09	CGMS_ASIA_08
datasets:	grid_plus_nuts_landcover	- 1)	grid_landcover_areas 2)	cmeteo_grid_landcover_areas
	grid_plus_ens_landcover	- 1)
attributes:	grid_no			grid_no	NN number(10)
	nuts_code/ens_code
	landcover_id			landcover_id	NN number(3)
	area			area	NN number(10)
	reg_code (only in gpel)
	reg_map_id			reg_map_id	NN number(10)

1) CGFS_EUR_09 uses the dataset of CGMS_EUR_08.
2) In CGMS_ASIA_08 a dataset has to be defined.

Examples

@@ Line 107: / Line 107: @@
 {|class="collapsing_table collapsible collapsed"
-!Conversion paths
+!Summery of structure conversion
 |-
 |
+====Object name conversion====
 {|class="wikitable"
 !Environment !! Primary data source !! Intermediate object !! Unified object
@@ Line 126: / Line 127: @@
 |}
 The intermediate objects are views in the CGFS environment to convert the structure of the primary data source into a unified structure. They are alternately used by the synonym {{Object|CMETEO_GRID_WEATHER}} to acquire the unified object name.
-|}
-{|class="collapsing_table collapsible collapsed"
+====Structure conversion====
-!Summery of structure conversion
-|-
-|
 {|class="wikitable"
 !Object name !! !! !! !! !! !!domain