AGB Benchmark

Introduction

ERS R&D team undertook a benchmarking initiative to identify the optimal AGB data provider, tailored to ERS’s specific needs.

The methodology for this benchmarking involved a comprehensive, multi-step process designed to evaluate potential AGB data providers objectively and thoroughly. A key aspect of our methodology was obtaining an extensive AGB dataset, covering around 50,000 hectares of varied land in Mozambique. This dataset, derived from Terrestrial Laser Scanning (TLS) and Airborne Laser Scanning (ALS), served as our benchmark for evaluating the performance of various AGB data providers.

The subsequent sections of this report will detail the methodology employed in this benchmarking exercise, clarifying the steps taken to effectively evaluate AGB data providers. Our assessment criteria included the precision of AGB estimates, the methodologies for calculating uncertainty, and the technical infrastructure, particularly focusing on APIs and automated estimation capabilities.

Methodology

DATA ACQUISITION

The benchmarking began with acquiring a detailed AGB dataset for 50,000 hectares of diverse terrain in Mozambique. This dataset, meticulously gathered using TLS and ALS technologies, provided a robust, high-resolution reference for our comparative analysis.

PARTICIPANT ENGAGEMENT

We engaged various AGB data providers in this exercise, inviting them to provide AGB estimates for the specified reference area. Precise geographic coordinates were supplied to each participant, who then used their proprietary models and methodologies to produce AGB estimates.

COMPARATIVE ANALYSIS

The comparison of AGB maps involves unique challenges, such as potential pixel misalignment due to localization inaccuracies. Our analysis adopted a multi-faceted approach to address these challenges:

Visual Comparison: An initial visual comparison involved generating AGB maps of the reference region using a consistent value scale, allowing us to quickly spot major disparities in estimates across providers.

Geometrical Analysis: For a deeper dive into the precision of AGB estimations, we performed a geometrical analysis. This entailed selecting specific geometries within the reference area, equivalent to sub-polygons, and calculating the total estimated AGB from each provider for these polygons. This method provided insights into the providers' capacity to accurately determine AGB sequestration in specific areas, moving beyond mere pixel-level analysis.

EVALUATION CRITERIA

Our benchmarking evaluation was comprehensive, extending beyond precision. We scrutinised the methodologies used by providers to calculate uncertainty and examined their technical infrastructure, with particular emphasis on the availability and functionality of Application Programming Interfaces (APIs) and automated estimation processes.

LIMITATIONS & FUTURE IMPROVEMENTS

The evaluation currently targets a specific 50,000-hectare section of Mozambique's Tropical Dry Forests. This site was strategically selected for its ecological diversity, which includes dense, mixed, and sparsely vegetated forest zones, thereby providing a comprehensive dataset.

It is important to consider the limitations of this approach; AGB models can perform very differently across various biomes, each with unique structural and biomass characteristics that may necessitate tailored evaluation methods.

Future plans include expanding the study to encompass multiple areas, with the objective of enhancing the representativeness of this benchmark.

Technical Comparison

In this session, our goal is to evaluate the technical capabilities of participants in relation to our requirements. Essential criteria include:

1. The provider must offer an API or a system that enables automatic estimations and result retrieval, integrating seamlessly into our certification workflow without the need for human intervention.

2. Results must be provided in a Raster GeoTIFF format that is readily downloadable.

3. Accompanying each result, there must be a clearly defined uncertainty or error range.

4. The system should be adept at accurately processing multiple polygons.

5. Results should be generated and available within a 24-hour timeframe.

AGB Comparison

This global comparison aids in understanding the dynamics and distribution of values globally.

REFERENCE AGB vs PARTICIPANTS AGB

DISTRIBUTION COMPARAISON

Area Comparison

In this section, we have identified specific zones within the benchmark area and calculated the total AGB contained within these zones. We selected this method as it closely mirrors the calculations performed in our workflows, where we extract the total AGB of the restoration area to estimate the baseline.

Here is a summary for some relevant sites:

SITE 3 - HIGH AGB VALUES - 145 HA

SITE 17 - MIXED AGB VALUES - 5 818 HA

SITE 2 - VERY LOW AGB VALUES - 180 HA

SITE 8 - MIXED AGB VALUES - 180 HA

Uncertainity Propagation

When estimating the net GHG removals of a specific area, it's crucial to transform the AGB measurement from tons of dry matter per hectare for each pixel, as supplied by the AGB provider, into the total AGB in tons of dry matter. This conversion yields the aggregate sequestration value for the area. During this conversion, accurately propagating uncertainty from the pixel level to the area level becomes essential to maintain precision.

To do so we can adopt different approach: 

• Considering variables as independent, so the area level uncertainty can be estimated by summing pixel level variance.
• Considering variables as non-independent, mainly because of the spatial auto-correlation, so using a Monte Carlo approach

The Monte Carlo approach, while powerful, presents challenges in application, especially when AGB providers lack multiple AGB maps of the same region for varied metrics. This benchmark study contrasts two approaches to uncertainty propagation: one employing multiple provided AGB maps, and the other a simulated Monte Carlo method.

MONTE CARLO

In this protocol, we utilised a control sample consisting of 100 AGB maps of a specific region. For each geometry in our test sample, we computed the total AGB for each band using all 100 AGB maps, applying the following steps:

• Total AGB Calculation

  Iterate over each pixel within the project area:

  • Calculate the area covered by each pixel.
  • Compute the contribution of each pixel to the total AGB by multiplying the AGB value by the area covered by the pixel.

Where AGBi,j represents the mean AGB of the pixel at row i and column j and Si,j represents the area covered by the pixel.

• Computing Confidence Interval for Geometry

For each geometry, we derived a confidence interval based on the multiple AGB estimations through the following steps:

• Calculate the sample mean:

where n is the number of samples and xiis the ith sample.

• Determine the Standard Error of the Mean (SEM):

where s is the sample standard deviation, and n is the sample size.

• Calculate the Margin of Error (ME) using t-score:

where t is the t-score from the t-distribution corresponding to the desired confidence level and degrees of freedom (n − 1). It can be obtained using the inverse of the t-distribution cumulative distribution function (CDF):

• Compute the Confidence Interval (CI):

SIMULATED MONTE CARLO

This method uses a simulated Monte Carlo approach, integrating pixel-level uncertainty to estimate total AGB for a project area. Prior to simulation, a normality check using the Shapiro-Wilk test is performed. A mean p-value of 0.677 indicated normal distribution of the data. We performed the following steps:

• Normality Check Using Shapiro-Wilk Test

  Before running the Monte Carlo simulation, validate that the pixel-level noise in the reference data follows a normal distribution:

  • Run the Shapiro-Wilk test on each array of pixels covering the same area. The null hypothesis of this test is that the data is normally distributed.
  • Check the p-value from the test. A high p-value (typically >0.05) indicates that the null hypothesis cannot be rejected, suggesting that the data is normally distributed.

We obtain a mean p-value of 0.677 which is higher than 0.05, so our data can be considered as normally distributed.

• Monte Carlo Simulation

  For the Monte Carlo simulation we achieved the following steps:

  • Generate a simulated AGB map by adding normally distributed random noise to the AGB values, with the standard deviation equal to the uncertainty values for each pixel.
  • Calculate the total AGB for the simulation by summing the AGB values of all pixels within the project area, adjusted for the area covered by each pixel.
  • Store the total AGB for each simulation.

• Confidence Interval Calculation and margin of error

  Calculate the confidence interval and margin of error for the total AGB estimates from the simulations. Follow the steps outlined in the previous methodology.

SUMMING VARIANCES

This method involves converting uncertainty to variance and summing these variances to calculate the total AGB and variance for the project area.

• Total AGB and Variance Calculation

  Iterate over each pixel within the project area:

  • Calculate the area covered by each pixel.
  • Compute the contribution of each pixel to the total AGB by multiplying the AGB value by the area covered by the pixel.

Where AGBi,j represents the mean AGB of the pixel at row i and column j and Si,j represents the area covered by the pixel.

• Compute the variance contribution of each pixel. The variance for each pixel is the square of the uncertainty value, multiplied by the squared area covered by the pixel.

Where Si,j represents the area covered by the pixel and Ui,j the uncertainty value of the same pixel.

• Confidence Interval Calculation

  Use the total standard deviation (the square root of the total variance) to calculate the confidence interval for the total AGB.

• Margin of Error

  Calculate the margin of error using the total standard deviation and the z-score corresponding to the desired confidence level (e.g., 1.96 for a 95% confidence interval).

RESULTS

The main goal of this comparison is to highlight the differences in results produced by various uncertainty propagation methods. Illustrated below is a scatter plot delineating the relationship between the relative margin of error and surface area for each method, distinguished by varying colours:

Observations from the analysis reveal that both the simulated Monte Carlo and Summing variances methodologies tend to overestimate the margin of error in smaller areas. But, this difference becomes smaller in areas over 1000 hectares, showing that the results become more similar for larger areas.

A broader perspective indicates that, generally, for areas exceeding 500 hectares, the margin of error falls below the 1% threshold. This finding underscores the potential accuracy of these methods in larger-scale applications and highlights the importance of method selection based on the area size and precision requirements of the project.

CONCLUSION

For large areas exceeding 1000 hectares, the choice of uncertainty propagation method appears to have minimal influence, with the uncertainty in the estimated total AGB falling below 1% and thus being relatively negligible. Both the Simulated Monte Carlo and Variance Summing methods tend to yield comparable outcomes. However, the Simulated Monte Carlo technique is the recommended approach in the Aboveground Woody Biomass Product Validation Good Practices Protocol document document. Consequently, ERS will adopt this method for its assessments.

Uncertainity Comparison

In this section, we aim to compare how participants handle uncertainties calculation. We acknowledge that the process of estimating AGB can be complex and filled with inherent uncertainties. Therefore, understanding and quantifying these uncertainties is crucial in evaluating the reliability and accuracy of the AGB estimates provided by the participants.

ERS relies on the best practices described in the Aboveground Woody Biomass Product Validation Good Practices Protocol which are:

• The estimation of AGB error must consider the entire process, from field measurements to modelling errors, including those associated with allometric equations.
• The propagation of uncertainty through these various stages must be effectively managed.
• A 95% confidence interval should be utilised.

The uncertainty comparison was conducted based on the methodologies used by each participant to calculate uncertainty. Given that the methodologies varied across participants, the comparison was done in a way that accommodated these variations, providing a fair and comprehensive comparison.

🟢: Perfectly handled

🟡: Adequately handled

🔴: Incorrectly handled