Updating GeoViable Environmental Data Layers Guide

GeoViable maintains fresh environmental data through an automated pipeline that runs on the first day of every month. The script scripts/update_layers.py downloads the latest Shapefiles from MITECO and CNIG, validates them against a SHA-256 hash to avoid redundant reloads, reprojects and clips them to Galicia, then replaces the previous data in a single atomic database transaction. If any step fails, the transaction is rolled back automatically and the previously loaded data remains intact.

Automated Update Schedule

The update job runs as a cron entry inside the geoviable-api container:

# /etc/cron.d/geoviable — day 1 of each month at 03:00 UTC
0 3 1 * * cd /app && python -m scripts.update_layers >> /var/log/geoviable_update.log 2>&1

Parameter	Value
Script	`scripts/update_layers.py`
Container	`geoviable-api`
Schedule	Day 1 of each month at 03:00 UTC
Log file	`/var/log/geoviable_update.log` (inside container)

Manual Trigger

To run the update outside of the regular schedule — for example, immediately after a MITECO data release:

docker compose exec geoviable-api python -m scripts.update_layers

To load data from manually downloaded ZIP files instead (see Manual Download Fallback):

docker compose exec geoviable-api python -m scripts.load_initial_data

You can also load a single layer:

docker compose exec geoviable-api python -m scripts.load_initial_data --layer red_natura_2000

Or inspect column mappings without loading anything:

docker compose exec geoviable-api python -m scripts.load_initial_data --inspect

After any update run, call GET /api/v1/layers/status to confirm that all 7 layers report a records_count greater than 0 and a recent last_updated timestamp. A zero count on any layer means the update failed or the table has not been loaded yet.

Update Strategy: TRUNCATE + INSERT

Each layer update uses a single atomic transaction:

TRUNCATE TABLE <layer> — removes all existing rows.
INSERT (via gdf.to_postgis(if_exists='append')) — writes the new dataset.
If the download or insert fails → automatic ROLLBACK → previous data is preserved.
If the insert succeeds → COMMIT → a layer_update_log entry is written with status = 'success', records_loaded, and file_hash.
VACUUM ANALYZE runs outside the transaction in AUTOCOMMIT mode.

This guarantees that the layer tables are never left empty, even if a monthly download from MITECO fails or returns a corrupted file.

Update Flow

Find download link

The script fetches the MITECO or CNIG download page and uses requests + BeautifulSoup to extract ZIP file links matching a layer-specific regex pattern (e.g. red_natura.*\.zip).

Download the ZIP

The matched URL is downloaded using an HTTPAdapter with 3 automatic retries and a 120-second timeout.

SHA-256 hash check

A SHA-256 digest of the downloaded bytes is computed and compared against the hash stored in layer_update_log from the last successful update. If the hashes match, the layer is skipped (status = 'skipped') and no database write occurs.

Extract Shapefile

The ZIP is unpacked into a temporary directory and the first .shp file is read into a GeoPandas GeoDataFrame. Files whose names suggest non-peninsular regions (Canarias, Baleares, Ceuta, Melilla) are automatically deprioritized.

Reproject to EPSG:25830

If the GeoDataFrame’s CRS is not already EPSG:25830, it is reprojected with gdf.to_crs(epsg=25830). If no CRS is detected, EPSG:4258 (ETRS89) is assumed. Any Z coordinates are stripped before insertion to match the 2D schema columns.

Filter to Galicia bounding box

Records that do not intersect the Galicia bounding box [-9.5, 41.5, -6.5, 44.0] (EPSG:4326) are discarded. For water-related layers (zonas_inundables, dominio_publico_hidraulico, masas_agua_superficial, masas_agua_subterranea) an additional demarcation filter keeps only records from Galicia-Costa and Miño-Sil.

Map columns to database schema

Shapefile field names are renamed to match the PostGIS table schema (e.g. SITE_CODE → codigo, SUP_HA → superficie_ha). Missing fields are set to NULL. For zonas_inundables, the periodo_retorno value (T100 or T500) is injected if the source file does not contain it.

TRUNCATE + INSERT (atomic transaction)

Inside a single SQLAlchemy transaction: the target table is truncated and the processed GeoDataFrame is inserted with gdf.to_postgis(if_exists='append'). A failure at any point rolls back both operations.

VACUUM ANALYZE

After a successful commit, VACUUM ANALYZE <table> runs in AUTOCOMMIT mode to update query planner statistics and reclaim space.

Write to layer_update_log

A row is inserted into layer_update_log recording status, records_loaded, source_url, file_hash, and finished_at. On failure, error_message is populated instead.

Download Strategy

MITECO download pages do not provide stable direct file URLs. The update pipeline uses a two-level approach:

Level 1 — requests + BeautifulSoup

The preferred method. Fetches the page HTML and scans all <a href> elements for links matching the layer’s file pattern regex. Works for static pages and most MITECO download portals.

Level 2 — Playwright (fallback)

Used when download links are rendered by JavaScript. Launches a headless Chromium browser, waits for a[href$=".zip"] elements to appear, then extracts the URL. Supports ARM64 via playwright install chromium.

Data Source URLs

Layer	Source Page
Red Natura 2000	https://www.miteco.gob.es/es/biodiversidad/servicios/banco-datos-naturaleza/informacion-disponible/rednatura_2000_desc.html
Zonas inundables (SNCZI)	https://www.miteco.gob.es/es/cartografia-y-sig/ide/descargas/agua/descargas_agua_snczi.html
Dominio Público Hidráulico	https://www.miteco.gob.es/es/cartografia-y-sig/ide/descargas/agua/dph-y-zonas-asociadas.html
Vías pecuarias	https://centrodedescargas.cnig.es/CentroDescargas/
Espacios Naturales Protegidos	https://www.miteco.gob.es/es/biodiversidad/servicios/banco-datos-naturaleza/informacion-disponible/enp_descargas.html
Masas de agua superficiales	https://www.miteco.gob.es/es/cartografia-y-sig/ide/descargas/agua/masas-de-agua-phc-2022-2027.html
Masas de agua subterráneas	Same page as surface water (separate file on the same page)

MITECO restructures its download pages periodically and does not maintain stable direct-download URLs. If the automated update fails with “No download links found on the source page”, the scraping pattern for that layer needs to be updated — or you can fall back to the manual download method below.

Manual Download Fallback

For the initial data load, or as a fallback when automated downloads fail, you can download the Shapefiles manually from the pages above and load them using load_initial_data.py.

Download the ZIP files manually

Visit each source page, download the national Shapefile ZIP, and place the files in the DATA_DIR directory (default: /app/data inside the container, mounted from the host).Expected filenames:

red_natura_2000.zip
zonas_inundables_t100.zip
zonas_inundables_t500.zip
dph.zip
vias_pecuarias.zip
enp.zip
masas_agua_superficial.zip
masas_agua_subterranea.zip

Copy files into the container (if needed)

docker compose cp ./data/red_natura_2000.zip geoviable-api:/app/data/
# Repeat for each file

Run load_initial_data.py

# Load all layers
docker compose exec geoviable-api python -m scripts.load_initial_data

# Or load a single layer
docker compose exec geoviable-api python -m scripts.load_initial_data --layer red_natura_2000

Verify the result

docker compose exec geoviable-db \
  psql -U geoviable -d geoviable \
  -c "SELECT layer_name, status, records_loaded, finished_at FROM layer_update_log ORDER BY finished_at DESC LIMIT 10;"

Hydrographic Basin Filtering

Water-related layers are filtered by hydrographic demarcation after the bounding-box clip. The primary demarcations that cover Galicia are:

Demarcación	Coverage in Galicia	MVP priority
Galicia-Costa	Atlantic-draining basins (majority of the territory)	✅ Required
Miño-Sil	Miño river basin and tributaries (eastern Galicia)	✅ Required
Cantábrico Occidental	Small strip in northern Lugo	Optional
Duero	Small area in southeastern Ourense	Optional

The automated update script filters to Galicia-Costa and Miño-Sil for all water layers. Cantábrico Occidental and Duero cover marginal zones and can be added to filter_demarcacion in LayerConfig if needed.

Checking Update Status

# Via API
curl https://your-domain/api/v1/layers/status

# Via psql — last update per layer
docker compose exec geoviable-db psql -U geoviable -d geoviable -c "
  SELECT DISTINCT ON (layer_name)
    layer_name,
    status,
    records_loaded,
    finished_at,
    error_message
  FROM layer_update_log
  ORDER BY layer_name, started_at DESC;
"

The SHA-256 file hash check means that if MITECO publishes an identical file (same bytes) twice, the second run will be logged with status = 'skipped' and no database write will occur. This prevents unnecessary TRUNCATE + INSERT cycles on months when the source data has not changed.

Get Started

Using GeoViable

Deployment

Data & Database

Troubleshooting

Updating GeoViable Environmental Data Layers Guide

Automated Update Schedule

Manual Trigger

Update Strategy: TRUNCATE + INSERT

Update Flow

Download Strategy

Level 1 — requests + BeautifulSoup

Level 2 — Playwright (fallback)

Data Source URLs

Manual Download Fallback

Hydrographic Basin Filtering

Checking Update Status

Build docs developers (and LLMs) love

Get Started

Using GeoViable

Deployment

Data & Database

Troubleshooting

Documentation Index

​Automated Update Schedule

​Manual Trigger

​Update Strategy: TRUNCATE + INSERT

​Update Flow

​Download Strategy

Level 1 — requests + BeautifulSoup

Level 2 — Playwright (fallback)

​Data Source URLs

​Manual Download Fallback

​Hydrographic Basin Filtering

​Checking Update Status

Build docs developers (and LLMs) love

Automated Update Schedule

Manual Trigger

Update Strategy: TRUNCATE + INSERT

Update Flow

Download Strategy

Data Source URLs

Manual Download Fallback

Hydrographic Basin Filtering

Checking Update Status