Environment Setup for PM2.5 Satellite Training Notebooks

The ARSET PM2.5 notebooks are designed to run entirely inside Google Colab with data persisted to Google Drive. There is no local Python environment to manage. This page describes every component of that environment in detail: where data lives on your Drive, what each Python package does, how authentication works, and how to resolve common setup problems.

Google Drive Folder Structure

All downloaded data and saved subsets are written to Google Drive so that files persist across Colab sessions. The notebooks recommend (and the code paths assume) the following folder hierarchy:

MyDrive/
└── Colab Notebooks/
    └── ARSET/
        └── PM25_Training_2026/        ← s_folder_Main
            ├── SatPM_Data/            ← s_folder_SatPM
            └── MERRA2_CNN_Data/       ← s_folder_MERRA2_CNN

These paths appear verbatim in the notebook code cells:

s_folder_SatPM     = '/content/drive/MyDrive/Colab Notebooks/ARSET/PM25_Training_2026/SatPM_Data'
s_folder_MERRA2_CNN = '/content/drive/MyDrive/Colab Notebooks/ARSET/PM25_Training_2026/MERRA2_CNN_Data'
s_folder_Main      = '/content/drive/MyDrive/Colab Notebooks/ARSET/PM25_Training_2026'

You can use a different path, but you must update all three s_folder_* variables in the notebook to match. The folder names SatPM_Data and MERRA2_CNN_Data are referenced by name in several cells — keep them consistent.

What Gets Written to Each Folder

Folder	Files written
`PM25_Training_2026/`	`Monitor_data_subset.csv` — ground monitor data for the region
`SatPM_Data/`	`V6GL03.CNNPM25.GL.YYYYMM-YYYYMM.nc` (one per month), `SatPM_data_subset.nc`
`MERRA2_CNN_Data/`	MERRA-2 CNN hourly NetCDF files (one per day), `MERRA2_CNN_data_subset.nc`

The *_subset.nc and *_subset.csv files are condensed spatial/temporal subsets saved after the first full download. On subsequent sessions, you can reload them directly with xr.open_dataset and pd.read_csv without re-downloading from NASA servers.

Google Colab Environment

Google Colab provides a hosted Linux runtime with Python 3.x and most scientific packages pre-installed. Each Colab session is ephemeral — the runtime resets when it disconnects or after 12 hours of inactivity. Packages installed with !pip install and variables stored in memory are lost on reset; files written to Google Drive are permanent. The notebooks mount Google Drive at the start of the setup cell:

from google.colab import drive
drive.mount('/content/drive')

After running this line, Colab will present an authentication dialog in the cell output asking you to authorize Drive access. Once authorized, your entire Drive is accessible at /content/drive/MyDrive/.

Python Packages

The setup cell installs two packages not bundled with Colab’s default environment, then imports everything else from the pre-installed runtime:

!pip install --quiet cartopy earthaccess

The --quiet flag suppresses verbose pip output. Installation typically takes 60–90 seconds on a fresh runtime. All other packages listed below are already present in the Colab environment.

Package Reference

Data Analysis
Data Access
Plotting
System

Package	Import	Role in the notebooks
`numpy`	`import numpy as np`	Array arithmetic, `datetime64` time handling, `np.arange` for month iteration, NaN masking
`pandas`	`import pandas as pd`	Ground monitor data frames with `(lat, lon, time)` multi-index, monthly resampling, CSV I/O
`xarray`	`import xarray as xr`	Loading and subsetting NetCDF gridded datasets (`xr.open_dataset`, `.loc[{...}]`, `.mean(dim='time')`)

Package	Import	Role in the notebooks
`earthaccess`	`import earthaccess`	Authenticates with NASA Earthdata and provides `earthaccess.search_data()` / `earthaccess.download()` for the MERRA-2 CNN product
`requests`	`import requests`	HTTP requests to the OpenAQ v3 REST API and the Chilean SINCA monitoring network web pages
`urllib`	`import urllib`	`urllib.request.urlretrieve()` downloads SatPM monthly NetCDF files directly from the S3 bucket
`re`	`import re`	Regular expression parsing of SINCA HTML pages to extract station coordinates and data links
`time`	`import time`	`time.sleep(0.2)` rate-limits requests to the SINCA web scraper
`urllib.parse`	`from urllib.parse import urljoin`	Resolves relative URLs on SINCA station pages into absolute URLs
`BeautifulSoup`	`from bs4 import BeautifulSoup`	HTML parsing for SINCA station pages (station names, graph URLs, form options)
`getpass`	`import getpass`	`getpass.getpass()` securely prompts for the OpenAQ API key without echoing it to the cell output
`os`	`import os`	`os.path.join`, `os.path.exists` for constructing file paths and checking whether data is already downloaded

Package	Import	Role in the notebooks
`matplotlib`	`import matplotlib.pyplot as plt`	Base figure and axis objects; `plt.subplots`, `plt.get_cmap` for colormaps
`cartopy`	`import cartopy`	Geospatial projections (`cartopy.crs.PlateCarree`), natural features (coastlines, borders, lakes, state boundaries) overlaid on maps
`seaborn`	`import seaborn as sns`	Statistical scatter plots and regression visualizations used in the performance evaluation section

Package	Import	Role in the notebooks
`warnings`	`import warnings`	`warnings.filterwarnings('ignore')` suppresses deprecation warnings from xarray and cartopy
`google.colab`	`from google.colab import drive`	`drive.mount()` attaches Google Drive to the Colab filesystem

NASA Earthdata is the authentication system used by earthaccess to access data held in the NASA Common Metadata Repository (CMR). It is required specifically for downloading the MERRA-2 CNN (MERRA2_CNN_HAQAST_PM25) product — the SatPM data is hosted on a public S3 bucket and does not require authentication.

Getting an Account

Go to urs.earthdata.nasa.gov
Click Register and fill in the form — registration is free and immediate
No application approval is needed for the data used in this training

When the setup cell calls earthaccess.login(), the package checks (in order) for credentials in:

A ~/.netrc file (not present in a fresh Colab runtime)
Environment variables EARTHDATA_USERNAME and EARTHDATA_PASSWORD
An interactive prompt in the cell output

In Colab, the interactive prompt is used. Type your Earthdata username and password when prompted. earthaccess stores the session in memory and uses it for all subsequent search_data and download calls in the same runtime.

# Triggered during the setup cell:
earthaccess.login()

# Used later when downloading MERRA-2 CNN data:
l_results = earthaccess.search_data(
    short_name='MERRA2_CNN_HAQAST_PM25',
    bounding_box=(n_lon_min, n_lat_min, n_lon_max, n_lat_max),
    temporal=(str(t_start), str(t_end))
)
l_files_MERRA2_CNN = earthaccess.download(l_results, s_folder_MERRA2_CNN)

If your Colab runtime resets, you must re-run the setup cell and re-enter your Earthdata credentials. The downloaded NetCDF files on Drive are intact, but the earthaccess session is lost.

OpenAQ API Key

OpenAQ v3 requires an API key for all requests. The key is passed as the X-API-Key HTTP header in every call made by the F_get_OpenAQ_from_API function.

Getting a Key

Visit docs.openaq.org/using-the-api/api-key
Register for a free OpenAQ account
Copy your API key from your account dashboard

How the Key Is Used

The setup cell stores the key in a module-level variable:

s_key_openAQ = getpass.getpass('Enter OpenAQ API key:')

The F_get_OpenAQ_from_API function uses it as the default value for its s_key parameter:

def F_get_OpenAQ_from_API(n_lon_min, n_lat_min, n_lon_max, n_lat_max,
                          t_start, t_end,
                          l_variables=['PM2.5'],
                          b_reference=True,
                          b_lowcost=False,
                          s_key=s_key_openAQ):
    ...
    response_parameters = requests.get(
        'https://api.openaq.org/v3/parameters',
        headers={'X-API-Key': s_key}
    )

The function queries three OpenAQ v3 endpoints in sequence: /v3/parameters (to look up parameter IDs), /v3/locations (to find sensor IDs within the bounding box), and /v3/sensors/{id}/measurements (to download hourly readings). All three pass the key as the X-API-Key header.

The Case Study notebook uses OpenAQ for comparison purposes but ultimately relies on SINCA data (Chile’s national network) for the analysis, because OpenAQ lacked the unvalidated hourly fire-episode readings present in SINCA. The Homework notebook uses OpenAQ as the sole ground truth source.

Complete Setup Cell Reference

For convenience, the full setup cell exactly as it appears in both notebooks:

# Install & Import the required packages:
!pip install --quiet cartopy earthaccess

# basic packages:
import os
import getpass
from google.colab import drive
import warnings
warnings.filterwarnings('ignore')

# data analysis packages:
import numpy as np
import pandas as pd
import xarray as xr

# data access packages:
import earthaccess
import requests
import urllib
import re
import time
from urllib.parse import urljoin
from bs4 import BeautifulSoup

# plotting packages:
import matplotlib.pyplot as plt
import cartopy
import seaborn as sns

# Connect to Google Drive:
drive.mount('/content/drive')

# Store login information:
earthaccess.login()
s_key_openAQ = getpass.getpass('Enter OpenAQ API key:')

Troubleshooting

The pip install step fails or takes very long

Colab’s runtime environment can occasionally be slow to resolve dependencies for cartopy, which has several compiled C extensions. Try:

Runtime → Disconnect and delete runtime, then reconnect and re-run the setup cell on a fresh runtime
If cartopy specifically fails, check that your Colab runtime type is set to Python 3 (not an older version) under Runtime → Change runtime type

earthaccess.login() hangs or raises an authentication error

Double-check your Earthdata username and password at urs.earthdata.nasa.gov — passwords are case-sensitive
Ensure you have accepted the EULA for the MERRA2_CNN_HAQAST_PM25 product; some NASA datasets require application-specific EULAs. Search for the product in Earthdata Search and accept any agreements before running the notebook
If prompted repeatedly, the session may not be persisting; try calling earthaccess.login(strategy="interactive") explicitly

OpenAQ API returns 401 Unauthorized

Your API key is invalid or expired. Log in to your OpenAQ account at explore.openaq.org to verify the key is active. Re-run the getpass line to enter the correct key:

s_key_openAQ = getpass.getpass('Enter OpenAQ API key:')

Google Drive fails to mount or shows 'Drive not mounted'

Make sure you authorized Drive access in the popup dialog when drive.mount('/content/drive') ran
If the dialog did not appear, try Runtime → Restart runtime, re-run the setup cell, and watch for the authorization prompt
In some institutional Google accounts, Drive access from third-party apps may be restricted by your organization’s admin

SatPM download raises a urllib error or returns a 404

SatPM files are served from a public AWS S3 bucket. The URL pattern is:

https://s3.amazonaws.com/satpmdata/V6GL03/FineResolution/GL/Monthly/
{YEAR}/V6GL03.CNNPM25.GL.{YYYYMM}-{YYYYMM}.nc

A 404 typically means the requested month has not yet been published in the product. Check the SatPM data access page to confirm coverage for your time range.

cartopy maps render blank or raise a projection error

cartopy downloads Natural Earth shapefiles on first use. If the runtime loses its internet connection mid-download, the shapefile cache can be corrupted. Run:

import cartopy.io.shapereader as shpreader
shpreader.natural_earth(resolution='110m', category='cultural', name='admin_0_countries')

to trigger a fresh download. If the error persists, disconnect and delete the runtime, then restart.

MERRA-2 CNN download is very slow

The MERRA-2 CNN product contains hourly data and files are large. For a three-month window (e.g., June–August 2019), earthaccess.download may fetch 90+ daily files. Downloads run sequentially by default. This is expected behavior — the files are cached to Drive, so you only need to download them once. Use the re-load cells in the Analysis section on subsequent sessions.

xarray raises a decode error when opening a NetCDF file

Partially downloaded files (e.g., from a dropped connection) can cause corrupt NetCDF files. Delete the affected file from Google Drive and re-run the download cell. The SatPM download cell checks os.path.exists before downloading, so it will skip already-downloaded files and only re-fetch the missing one.

Get Started

Data Sources

Notebooks

Analysis Guide

Google Drive Folder Structure

What Gets Written to Each Folder

Google Colab Environment

Python Packages

Package Reference

Getting an Account

OpenAQ API Key

Getting a Key

How the Key Is Used

Complete Setup Cell Reference

Troubleshooting

Build docs developers (and LLMs) love

Get Started

Data Sources

Notebooks

Analysis Guide

Documentation Index

​Google Drive Folder Structure

​What Gets Written to Each Folder

​Google Colab Environment

​Python Packages

​Package Reference

​NASA Earthdata Login

​Getting an Account

​How earthaccess.login() Works

​OpenAQ API Key

​Getting a Key

​How the Key Is Used

​Complete Setup Cell Reference

​Troubleshooting

Build docs developers (and LLMs) love

Google Drive Folder Structure

What Gets Written to Each Folder

Google Colab Environment

Python Packages

Package Reference

NASA Earthdata Login

Getting an Account

How `earthaccess.login()` Works

OpenAQ API Key

Getting a Key

How the Key Is Used

Complete Setup Cell Reference

Troubleshooting