Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/NASAARSET/PM2.5_AQ_Online_2026/llms.txt

Use this file to discover all available pages before exploring further.

F_compute_metrics calculates a comprehensive suite of standard performance metrics that quantify how well a satellite or model PM2.5 estimate agrees with collocated ground-based measurements. Given a merged DataFrame containing both measured and estimated values at the same locations and times, the function returns a dictionary of scalar statistics — including bias, normalised bias, RMSE, normalised RMSE, the coefficient of determination (R²), and both spatial and temporal Pearson correlation coefficients — and optionally prints a formatted summary to the notebook output. Use this function as the final evaluation step of the ARSET PM2.5 validation workflow, after merging the satellite product (from F_subset_and_combine) with ground observations (from F_get_OpenAQ_from_API or F_get_SINCA_from_web) into a single DataFrame. The returned dictionary can also be stored and compared across products, regions, or time periods to support the homework exercises.

​
Parameters

​
f_data
pandas.DataFrame
required
A pandas.DataFrame containing both measured and estimated PM2.5 values. The DataFrame must have a three-level MultiIndex corresponding to latitude, longitude, and time (in that order). This is the native format returned by F_get_OpenAQ_from_API and F_get_SINCA_from_web after a merge with collocated satellite data.
​
s_time
str
default:"'time_month_start'"
Name of the time index level in the three-level (lat, lon, time) MultiIndex. This is used to identify unique time steps for temporal correlation and for grouping data by time period. The default 'time_month_start' matches the time index level name used in the case-study data-processing workflow.
​
s_value
str
default:"'Average'"
Name of the column containing the measured (ground-truth) values. Default is 'Average', which is the column name assigned to the averaged OpenAQ or SINCA measurements in the merged DataFrame. Change to 'PM2.5' if working directly with the raw observation DataFrame.
​
s_estimate
str
default:"'SatPM'"
Name of the column containing the estimated (satellite or model) values. Default is 'SatPM'. Adjust this to match the variable name used when merging the satellite dataset into the DataFrame (e.g., 'MERRA2_CNN').
​
b_print
bool
default:"True"
If True, all computed metrics are printed to standard output in a human-readable table. Set to False when calling the function in a loop (e.g., iterating over multiple years or regions) to suppress verbose output.

​
Return Value

​
d_metrics
dict[str, float]
A dictionary of computed metric values. Keys and their definitions:
KeyDescription
'Estimated Mean'Mean of the satellite/model estimates (µg/m³)
'Measured Mean'Mean of the ground observations (µg/m³)
'Bias'Mean difference: mean(estimate) − mean(observed) (µg/m³)
'Mean Normalized Bias'Bias normalised by the measured mean (dimensionless)
'RMSE'Root-mean-square error (µg/m³)
'Mean Normalized RMSE'RMSE normalised by the measured mean (dimensionless)
'R²'Coefficient of determination (overall fit)
'Spatial Correlation (median)'Median Pearson r across locations, computed per time step
'Spatial Correlation (min)'Minimum spatial Pearson r across all time steps
'Spatial Correlation (max)'Maximum spatial Pearson r across all time steps
'Temporal Correlation (median)'Median Pearson r across time, computed per location
'Temporal Correlation (min)'Minimum temporal Pearson r across all locations
'Temporal Correlation (max)'Maximum temporal Pearson r across all locations

​
Usage Example

The following example follows the complete validation workflow for the Biobío case study: load satellite data, retrieve ground observations, merge them, then compute metrics. 
import numpy as np
import pandas as pd
import glob

# --- Step 1: Load satellite data ---
l_files = sorted(glob.glob('/data/SatPM2.5/monthly/SatPM_2023*.nc'))

a_data_SatPM = F_subset_and_combine(
    l_files   = l_files,
    n_lon_min = -74.0, n_lat_min = -39.0,
    n_lon_max = -71.0, n_lat_max = -36.0,
    t_start   = np.datetime64('2023-01-01'),
    t_end     = np.datetime64('2023-12-31'),
    F_time_from_file = time_from_filename,
)

# --- Step 2: Retrieve ground observations ---
f_data_monitors_OpenAQ = F_get_OpenAQ_from_API(
    n_lon_min = -74.0, n_lat_min = -39.0,
    n_lon_max = -71.0, n_lat_max = -36.0,
    t_start   = np.datetime64('2023-01-01'),
    t_end     = np.datetime64('2023-12-31'),
)

# --- Step 3: Collocate and average to monthly means ---
# (collocation details handled in the case-study notebook)
# f_data_combined_gridded has columns: 'Average' (observed), 'SatPM' (estimated),
# and a (lat, lon, time_month_start) MultiIndex

# --- Step 4: Compute metrics ---
d_metrics = F_compute_metrics(
    f_data     = f_data_combined_gridded,
    s_time     = 'time_month_start',
    s_value    = 'Average',
    s_estimate = 'SatPM',
    b_print    = True,
)

# Example printed output:
# Estimated Mean          :  12.34 µg/m³
# Measured Mean           :  11.87 µg/m³
# Bias                    :   0.47 µg/m³
# Mean Normalized Bias    :   0.04
# RMSE                    :   3.21 µg/m³
# Mean Normalized RMSE    :   0.27
# R²                      :   0.81
# Spatial Correlation     : median=0.78  min=0.52  max=0.94
# Temporal Correlation    : median=0.85  min=0.61  max=0.97

​
Comparing two satellite products

products = {'SatPM': 'SatPM', 'MERRA2_CNN': 'MERRA2_CNN'}
results = {}

for product_name, col_name in products.items():
    results[product_name] = F_compute_metrics(
        f_data     = f_data_combined_gridded,
        s_estimate = col_name,
        b_print    = False,   # suppress per-iteration output
    )

# Summarise
for name, metrics in results.items():
    print(f"{name}: R²={metrics['R²']:.2f}, RMSE={metrics['RMSE']:.2f} µg/m³")
Three-level MultiIndex requirement. The f_data DataFrame must have exactly three index levels (latitude, longitude, time). If your DataFrame was constructed differently, use df.set_index(['lat', 'lon', 'time']) before calling this function.
Rows containing NaN in either the measured (s_value) or estimated (s_estimate) column are excluded from all metric calculations. If a large fraction of your data is NaN — for example, due to cloud cover in the satellite product or data gaps in the monitor record — the reported metrics may represent only a non-representative subset of the time series. Check coverage with f_data[[s_value, s_estimate]].notna().mean() before interpreting results.
Spatial and temporal correlation statistics are reported as median, min, and max rather than a single scalar in order to capture the variability of correlations across locations (temporal) and time steps (spatial). A low minimum alongside a high median indicates that performance degrades at specific sites or seasons — a useful diagnostic for further investigation.
  • Performance Metrics Guide — theoretical background on each metric and recommended interpretation thresholds.
  • F_subset_and_combine — load and clip the satellite data that provides the s_estimate column.
  • F_get_OpenAQ_from_API — retrieve the ground observations that provide the s_value column.
  • F_get_SINCA_from_web — alternative ground-observation source for Chilean monitors.
  • F_plot_map — visualise the spatial distribution of bias or RMSE by plotting per-site metrics as point data.

Build docs developers (and LLMs) love

Get started for freeTalk to us