Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/danizd/urbanviable/llms.txt

Use this file to discover all available pages before exploring further.

The UrbanViable ETL is a local Python pipeline — not a background service, not a container — that runs on a developer workstation whenever the underlying open data needs refreshing. It downloads four public datasets, joins them on Spain’s census section identifier (CUSEC), normalises every scoring variable to [0, 1] across Galicia, and writes a single galicia_scouting.geojson. That file is then handed to Tippecanoe, which converts it into an MBTiles binary that TileServer GL can serve directly. The entire workflow produces one deployable artefact and leaves no moving parts on the server.

​
Pipeline at a glance

The four stages map cleanly onto the three scripts in etl/:
StageStepScript / toolOutput
Extract1 — Download raw sourcesdownload_data.pyRaw files under etl/data/
Transform2 — Load, join, normaliseprocess_data.pygalicia_scouting.geojson
Tile3 — Generate vector tilesgenerate_tiles.sh + Tippecanoegalicia_scouting.mbtiles
Deploy4 — Upload to serverscp (manual)Live map update
Each stage can be re-run independently. If only the rental data changes for a new INE release, you can replace etl/data/renta.csv and re-run steps 2–4 without touching the OSM or Catastro files.

​
Directory layout

etl/
├── download_data.py        ← Checks and downloads raw sources (run first)
├── process_data.py         ← Full pipeline → GeoJSON
├── generate_tiles.sh       ← Tippecanoe → .mbtiles
├── requirements.txt        ← Python dependencies
└── data/
    ├── secciones_censales/ ← CNIG Shapefile (ZIP + extracted SHP)
    ├── renta.csv           ← INE Atlas de Renta (long-format CSV)
    ├── poblacion_ige.csv   ← IGE demographic data by census section
    ├── galicia-260424.osm.pbf
    ├── catastro/
    │   ├── a_coruña/       ← Per-municipality ZIPs
    │   ├── lugo/
    │   ├── ourense/
    │   └── pontevedra/
    └── processed/
        ├── galicia_scouting.geojson   ← ETL output (intermediate)
        ├── galicia_scouting.mbtiles   ← Final deployable artefact
        └── last_update.json           ← Metadata badge for the frontend
The etl/data/ tree is .gitignored. Raw data files (especially the OSM PBF at ~200 MB and Catastro ZIPs) are never committed to the repository.

​
Python dependencies

Install the Python requirements before running any ETL script: 
pip install -r requirements.txt
PackageMin versionRole
geopandas>=0.14Read/write Shapefiles and GeoJSON, spatial joins
pandas>=2.0CSV parsing, joins, aggregations
numpy>=1.26Log normalisation (np.log1p)
pyrosm>=0.6OSM PBF parsing; installs osmium as a dependency
pyproj>=3.6CRS transformations (EPSG:25829/25830 → 4326)
shapely>=2.0Geometry predicates for spatial joins
openpyxl>=3.1Fallback if INE Atlas de Renta is distributed as .xlsx
requests>=2.31HTTP downloads in download_data.py

​
Installing Tippecanoe

Tippecanoe is a standalone CLI tool — it is not a Python package and is not listed in requirements.txt. Install it separately on the workstation where the ETL runs: 
sudo apt-get update
sudo apt-get install tippecanoe
Verify the installation: 
tippecanoe --version

​
The ETL runs on your workstation, not on the server

The production server only serves the .mbtiles file via TileServer GL. It holds no Python runtime, no GeoPandas, and no Tippecanoe. After running the ETL locally you upload two files: 
scp etl/data/processed/galicia_scouting.mbtiles  user@server:~/urbanviable/tiles_data/
scp etl/data/processed/last_update.json          user@server:~/urbanviable/tiles_data/
TileServer GL detects the new MBTiles file without requiring a restart.

​
Output contract: the 11-column GeoJSON

process_data.py guarantees that galicia_scouting.geojson always contains exactly these columns — this is the contract between the ETL and the MapViewer frontend:
ColumnTypeSourceUse
cusecstringCNIG Secciones CensalesSection identifier
NMUNstringCNIG Secciones CensalesTooltip (municipality name)
renta_normfloat [0,1]INE Atlas de RentaGPU scoring slider
renta_absintegerINE Atlas de RentaTooltip (€)
densidad_normfloat [0,1]Padrón / ShapefileGPU scoring slider
jovenes_normfloat [0,1]IGE poblaciónGPU scoring slider
mayores_normfloat [0,1]IGE poblaciónGPU scoring slider
poblacion_absintegerPadrón / ShapefileTooltip (inhabitants)
actividad_normfloat [0,1]OSM (log-normalised)GPU scoring slider
actividad_absintegerOSMTooltip (establishments)
uso_comercial_normfloat [0,1]CatastroGPU scoring slider
antiguedad_normfloat [0,1]CatastroGPU scoring slider
All _norm columns are guaranteed to lie in [0, 1] — enforced by validate_output() before the file is written. Missing source data causes the affected columns to fall back to 0.0 rather than crashing the pipeline.

Data Sources

Where each dataset comes from, how it is structured, and the quirks to watch out for before running the ETL.

Process Data

Step-by-step walkthrough of process_data.py — loading, joining, normalising, and exporting the GeoJSON.

Generate Tiles

Running generate_tiles.sh, every Tippecanoe parameter explained, and how to deploy the resulting MBTiles.

Downloading Source Files

Use python etl/download_data.py --download-osm to fetch the Geofabrik PBF and verify all source paths before processing.

Build docs developers (and LLMs) love

Get started for freeTalk to us