Valid Method

From Wikicap - European Commission

JRC "Area measurement tool validation method"

The validation method is designed to determine the intrinsic tool precision.

It should be set to limit as much as possible ‘external’ error factors (e.g. bad use of instrument, non-respect of parcel border …).

The validation test is made under ‘artificial/controlled’ conditions and has to be differentiated from a proficiency test that is performed under ‘real operating’ conditions. The result of the validation is strictly related to the whole set of method of measurement and not only to the device. Therefore, the certificate or the validation report remains valid as long as the operators respects the tested method conditions.

The quality of a measurement tool can be characterized by a number of parameters such as its bias, precision and accuracy. Assuming there is no bias, it can also be characterized by its reproducibility limit, which is the parameter used to determine the technical tolerance. The buffer validation method for both GNSS devices and ortho-imagery, is summarised in the following flow chart:

OTSCfigachap53.png

A. Data collection

It is highly recommended to contact JRC at the earliest stage of the exercise to check that the whole protocol of data acquisition (i.e. number and choice of parcels, number of repeated measurements, number of sets…) is in line with the present guidance.

A. 1. For validation of GNSS devices

The test covers the whole set of: hardware, software, settings and method.

The test parcels should have unambiguous borders to ensure that all measurements cover the same object. For instance, the borders could be marked with wooden sticks with a density of at least 1 peg per 25m. Objects should be of variable sizes (at least covering the range over which the GNSS should be working, for instance between 0.4ha and 4ha) and shapes (at least one elongated parcel should be included).

Number of fields

The more fields chosen for the test, the more reliable the assessment: more data collected give more points on the receiver characteristic curve. It is recommended to take at least 6 fields with sizes spread along the typical size range of the country.

Shape

The shape of the fields should vary from very simple shapes (e.g. rectangular) to some irregular shapes with high perimeter to area ratio.

Obstructions of the horizon

Validation test results conducted showed, a high impact of horizon obstruction on the buffer, e.g. parcels with trees to the south often show a significantly higher R-limit than parcels without tree obstruction. Selecting most test parcels in an open horizon environment is likely to result in a (low) buffer, which may not be appropriate to the usual conditions in which the device is used. Terrain characteristic and the type of cultivation is an important issue due to possible disturbances of the satellite signal. In mountainous areas or on fields (partly) bordered by trees, the “visibility” of the satellites may be limited, which may result in higher measurement errors. If parcels with partially obstructed borders are common in the region or country, such parcels should be included in the test to reflect the average conditions of measurement in the region/country.

Parce set GNSS.png

Example of validation test fields set

Borders of the test field

The test fields should have easily accessible borders (no stones, bushes to cross, marshy spots etc.) to allow operators walking without difficulty around them.

Reference value

Standard deviation of measurement repetitions should be estimated against a reference area of the considered parcel. The reference area of the test parcels should be established with a surveying tools or GNSS RTK measurements. It can also be taken from a LPIS reference area if a reference shape corresponds to it.

Repetitions

The GNSS constellation should be considered as relatively stable while collecting data on each field. In other words, the time needed to measure one field four times is short enough (normally 10 to 30 minutes, depending on the perimeter of the field) to consider the satellite constellation as stable. The four measurements taken in a short period of time, by the same operator, will allow to derive the repeatability variance of the area measurements.

Runs (sets) of measurements

The revisit time for the GNSS satellite constellation is equal to ~12h. In order to make measurements under different constellations (i.e. different number of satellites and different satellites in view), the different runs should start at different times of the day. At least 1.5 - 2 hours should be left between two successive runs to consider that the satellite constellations have changed. The variance between runs of measurements will be used to derive the reproducibility variance of the area measurements.

OTSCfigcchap53.png

Example of organization of field measurements for 6 parcels with 2 operators

Settings

The test must be performed with exactly the same settings that will be used during OTS check work (max DOP, S/N ratio, logging interval).

Method

In best practice, it is recommended to perform most of the measurements in continuous mode. However, in real field controls there are situations where the GNSS device has to be used in continuous AND vertex measurement modes (so-called stop-and-go method).

Therefore, so the device must be validated for both modes (continuous logging and log of vertices).

For area measurement done using vertices, the distance between two successive vertices should not be greater than 25m. This is to “simulate” the natural landscape measurement conditions, where the borders are rarely straight and data are logged more frequently than when measuring rectangles. However, in case of very elongated straight parcels (i.e. more than 400, 500 meters long) the distance between two successive vertices can be extended to 100, 150 meters (but marked clearly with a density of at least one marker/peg for 20-25m). On each of the vertices, the operator should remain several seconds to get several records per point. In continuous mode also, the operator should remain some seconds at each corner. Drastic change of direction of the parcel border.

Avoiding systematic errors

Operators should not disturb each other while measuring so if possible one operator should measure one parcel at one time. In order to avoid potential systematic errors related to left/right handed operators,

The direction of walking should be both in clockwise and anticlockwise.

e.g.: for all the runs: 1st repetition – always clockwise, 2nd repetitions always anticlockwise, 3rd – clockwise, 4th – anticlockwise.

A.2. For validation of orthoimagery

On orthoimagery, repetitions of measurement are less time consuming than field measurement, thus the minimum number of parcels and repetitions could be increased to at least 30 parcels. In order to ease the work, it is advised to select parcels corresponding to LPIS reference parcels (reference area already available). Otherwise, it will be necessary to measure the reference area on field using a surveying tools or GNSS RTK measurements. Area measurements of these parcels performed by at least 6 operators (for proper statistical analysis) Each operator performing at least 4 measurements (repetitions) of each parcel (for proper statistical analysis) For what concerns the parcels’ selection:

- Parcels selected should be a representative sample of the control area zone (strongly related to the parcel structure)
- Parcel sizes should cover the range observed in the control area
S: small
M: medium
L: large
- Parcel shapes should vary
SF1: compact
SF2: elongated
SF3: very elongated
- Some parcels should be selected with easily identifiable borders as to avoid interpretation problems and lead to some parcels rejections later on during the analysis

Remarks concerning choice of the parcel size and shape ranges

Before parcel set definition, (size and shape) statistical analysis of the parcel structure on the area to be controlled should be performed. In the first step parcel areas are sorted and 5% of outlying values are discarded (percentiles: 97.5% and 2.5%). In the next step the remaining range is divided by 3 equal parts (small, medium and large size). Parcel area and perimeter allow for Shape Factor calculation (SF=(perimeter/4)^2/area). The same procedure should be performed for SF (compact, elongated and very elongated parcels).

Example of parcel set (30 parcels):

- good border: 15 parcels
S and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
M and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
L and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
- "fuzzy" border: 15 parcels
S and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
M and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels
L and SF1/SF2/SF3 - 2/2/1 i.e 5 parcels

Remarks concerning choice of scale for digitisation

Considering the pixel size of imagery to validate (i.e. <= 0.5m), the scale for digitization should be around 1/1.000. Nevertheless, the scale should not be fixed to 1/1.000 and should be adapted on a parcel per parcel basis.

Example of parcel border

Some examples are provided hereafter in order to illustrate the concepts of ‘easy border’, ‘fuzzy border’ and ‘borders leading to interpretation problems’.

OTSCfigdchap53.png

Examples of ‘easy’ and ‘fuzzy’ borders that should be part of the parcels sample

OTSCfigechap53.png

Examples of parcels with limits difficult to delineate (photo interpret) and that should not be part of the selected parcels sample.

Precise, commons and detailed instructions (the same as used for creation of reference parcels – LPIS if parcel based LPIS available) have to be given to the photo interpreters.

For field parcels, pegs are provided to clearly identify parcel borders. For ortho-images it is not possible to provide the equivalent of pegs otherwise there will be a risk that photo interpreters digitize these vertices/lines not doing interpretation of the image.

Therefore, as guide for interpretation, a shape surrounding the parcel to be measured, has to be created in advance for all parcels in the sample, and shown on the screen during the measurement.

OTSCfigfchap53.png

As for the GNSS validation protocol, the reference area and perimeter of the test parcels should be established (from LPIS if available or performing field measurement using surveying tools or GNSS RTK measurements).

B. Statistical analysis

Collected data

The results of the area measurements performed in validation process should be collected like in Table A of ISO 5725. In each cell, there is measured parcel area (in this case in square meters). Each 4 repetitions define one set, or used in ISO 5725 one laboratory, or sometimes called one run. So one operator delivers 3 sets (in GNSS validation, see diagram above and table below). Level in ISO 5725 means in our validation procedure - parcel (6 parcels: A, B, C, D, E and F - 6 levels).

Table A

OTSCfiggchap53.png

Basic statistics

Next for each parcel mean area and standard deviation for each set is calculated (Table B and C in ISO 5725-2).

OTSCfighchap53.png

One should make sure to avoid rounding the values of the mean and standard deviation statistics. Slight differences can propagate and create numerical issues (e.g. negative value where a positive value is expected).

Outliers detection

The statistical parameters concerning the buffer tolerance value should be calculated only for the dataset free of outliers. Therefore the collected data need to be tested for outliers see flowchart below. Cohran's test is described in chapter 7.3.3 and Grubss' test in chapter 7.3.4 of ISO 5725-2.

- Cochran’s test (in blue) checks variation of standard deviation between classes.
- Grubbs’ test for single observation (in dashed green) checks variation of observed value in class (standard deviation is calculated within class).
- Grubb’s test for 1 outlying mean (in yellow) or for 2 outlying means (in dashed orange) checks variation of means between classes, standard deviation is calculated between classes.
Test parm.png
OTSCfigjchap53.png

During the outliers' detection, follow the flowchart above and perform the tests according ISO 5725-2.

Two of the four tests are always performed at least once (in plain color rectangles in the flowchart): the Cochran’s and the Grubbs’ test for 1 outlying mean.

The remaining two tests are performed only in specific conditions (see the dashed rectangles in the flowchart steps II.1, II.2, III.2 and IV.1).

After the outliers removing tables: A, B and C are modified. It is recommended to put the reason of outliers discarding in table A. Single observations or all sets can be removed. In such a case, the number of sets and the number of observations must be updated for detecting the potential remaining outliers (e.g. the critical value for the Grubb’s test on 1 outlying mean is 1,973 for 6 sets and becomes 1,764 for 5 sets). For the details, please refer to Tables 4 and 5 of ISO 5725-2.

- Calculation of repeatability and reproducibility

After outliers detection the uncertainty of parcel area measurements is estimated.

ISO 5725-1 sets the general definitions for:

- Repeatability standard deviation: the standard deviation of test results obtained under repeatability conditions.
- Repeatability conditions: conditions where independent test results are obtained with the same method on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time.
- Reproducibility standard deviation: the standard deviation of test results obtained under reproducibility conditions.
- Reproducibility conditions: conditions where test results are obtained with the same method on identical test items in different laboratories with different operators using different equipment.

- If the computed between-laboratory standard deviation sLj^2 is negative, it must be manually set to 0. In such case, srj^2=sRj^2.

OTSCfigkchap53.png
OTSCfiglchap53.png

In the example, the computed between-laboratory standard deviation sLj^2 are negative for the parcels E and F. One can see that in that case srj^2=sRj^2.

- Verification of bias and influence of pooling factors.

According the inter-laboratory tests performed in the past and during ongoing projects, the significant, repeatable and rigid influence of parcel border quality on the value of reproducibility was observed. Any other factors should not influence the validation results. However, we observed in some cases the bias, influence of the operator, walking direction, measurements day, parcel area and size etc. But generally we expect no bias and no influence of any factors presented in the table below. Therefore, we recommend performing bias test (basing on the formulas in IS0 5725-2 chapter 7.4.5, ISO 5275-4 chapter 4.7.2 or T-Student test). Influence of the other factors is recommended to verifying using ANOVA analysis.

Table influ.PNG

C. Buffer tolerance estimation

Repeatability and reproducibility standard deviations are for each parcels in square meters. Reproducibility limit in area values [m^2] depends on parcels so for the standardization it should be divided by reference perimeter to obtain reproducibility limit in buffer values [m].

According ISO 5725-6 chapter: 4.1.4: when examining two single test results obtained under reproducibility conditions, the comparison shall be made with the reproducibility limit (in our case: buffer limit): RL=2,8 (sRj in buffer).

OTSCfignchap53.png

D. Classification of the buffer width for a measurement tool

Reproducibility limit calculated in validation process allows classifying the area measurement method to the one of the following classes:

- (1) "1.0m" for RL inside (0.75m, 1.0m];
- (2) "0.75m" for RL inside (0.50m, 0.75m];
- (3) "0.50m" for RL below 0.50m.

Mean value of repeatability in buffer for our example: 0,61 m so the validated method is classified into class (2): buffer limit taken to the control should be: 0,75 m.

E. Report from validation

Report from validation should include following information:

- (1) Validated equipment: type of GNSS receiver with software (type and version), serial numbers of each entity, metadata about orthoimagery (type, resolution, uncertainty from quality control etc.);
- (2) In the case of GNSS receivers:
device settings (elevation mask, max DOP, etc.);

o details about the validated method: vertex + number of logs per vertex / continuous + logging interval;

use of differential correction and type of correction;
measurements with or without external antenna.
- (3) reference parcels areas and perimeters, details about the measurements method applied for reference measurement;
- (4) design of parcels set;
- (5) table A with some explanation if needed (notice about the not normal procedure, applied equipment if changed or shared between operators, gross error etc.);
- (6) basic statistic before outliers discarding: table B and C;
- (7) results of outliers testing: table A with the removed single observations and/or all sets;
- (8) repeatability and reproducibility standard deviation in [m^2] and in buffer values [m];
- (9) results of bias analysis and ANOVA analysis;
- (10) buffer limit and class of the validated method;
- (11) Test data.

An example of xls file containing the collected data from an area measurement validation test can be found on WikiCAP Template.

F. Documentation needed when the statistical analysis is to be made or validated by JRC

1. In case a MS decides to entrust JRC with the statistical analysis after data collection, the following information should be sent to JRC for final analysis:

- Report from validation of the test carried out by MS (points: 1-5). Last four points (6, 7, 8 and 9) will be prepared by JRC (outliers detection, repeatability and reproducibility standard deviation calculation, buffer limit determination and result of method classification).
- Detailed description of validation procedure (protocol) should be delivered.
- Raw measurements data
In the case of GNSS - a copy of the measurement protocol indicating parcel id, date and time, set, repetition, operator, direction of measurement, area measured, perimeter measured.
In the case of orthoimagery validation all vector files should be delivered.

A technical report and the data will document the whole validation process; they will help JRC to evaluate and analyze the data as well as to draw conclusions on the tolerance to be used with that device and measurement method. The final statement on the performance of the system will be prepared by the JRC on the basis of the test results. 2. In case a MS decides to perform the validation tests and the statistical analysis by itself a technical report, data and the sheets with the statistical analysis (templates to be asked to JRC) should be sent to the JRC for validation, final assessment and publication of the results on the JRC web page.

Tool already validated or certified

NOTA BENE: the list of tools validated so far can be found under that LINK.



End of pages on the validation method

Go back to the Navigation page OTSC