Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/danizd/urbanviable/llms.txt

Use this file to discover all available pages before exploring further.

process_data.py is the core of the UrbanViable ETL. It reads all four raw data sources, joins them on the CUSEC census section identifier, normalises every variable to [0, 1] across Galicia, runs quality assertions, and writes the final galicia_scouting.geojson plus a last_update.json metadata file. Running it takes between a few minutes (sources 1 + 2 only) and several hours (all four sources including full Catastro ingestion).

​
Command

From the project root: 
python etl/process_data.py
No arguments are required. The script auto-detects which data files are present and skips or falls back gracefully for any missing optional sources. Before running, verify all source files are in place: 
python etl/download_data.py

​
Pipeline steps

1
Step 1 — Load and prepare census geometries
2
Function: load_sections()
Input: etl/data/secciones_censales/*.shp
3
The script scans etl/data/secciones_censales/ for the first .shp file, loads it with GeoPandas, and verifies that the CUSEC field is present. It then:
4
  • Zero-pads CUSEC to 10 digits and stores it as lowercase cusec (the join key used throughout).
  • Filters to Galicia only by matching the first two digits of cusec against {"15", "27", "32", "36"}.
  • Reprojects to EPSG:25830 (metric) to compute area_km2, then reprojects to EPSG:4326 for the output.
  • Attempts to find a population column (NPOB, POB_TOT, POBLACION, TOTAL, POB, POPULATION, or pop) to populate poblacion_abs and densidad. If none is found, both default to 0.
  • Sets pct_jovenes and pct_mayores to 0.0 as placeholders (overwritten in step 3 if IGE data is present).
    • 5
    shapefiles = sorted(SECTIONS_DIR.glob("*.shp"))
gdf = gpd.read_file(shapefiles[0])
gdf["CUSEC"] = gdf["CUSEC"].astype(str).str.zfill(10)
gdf = gdf[gdf["CUSEC"].str[:2].isin(GALICIA_PROVINCIAS)].copy()
gdf["cusec"] = gdf["CUSEC"]

projected = gdf.to_crs(epsg=25830)
gdf["area_km2"] = projected.geometry.area / 1_000_000
gdf = gdf.to_crs(epsg=4326)
    6
    Expected console output: Secciones: ~3800
    7
    8
    Step 2 — Join INE Atlas de Renta
    9
    Function: process_renta(gdf)
    Input: etl/data/renta.csv
    10
    The INE CSV is a long-format statistical file — not a flat cusec → income table. The function:
    11
  • Reads the file, trying utf-8-sig first (to strip the BOM), then falling back to latin-1.
  • Uses column positions (not names) to locate the section text column (col 2), indicator column (col 3), period column (col 4), and value column (col 5).
  • Extracts the CUSEC by matching the first 10 consecutive digits at the start of the section text field with parse_cusec_ine().
  • Filters by "Renta neta media por hogar" (with fallback to "Renta neta media por persona") and selects the most recent year.
  • Cleans the Spanish-format number string ("12.345,67" → 12345.67) and coerces suppressed values (".") to 0.
    • 12
    def parse_cusec_ine(text: str) -> Optional[str]:
    """
    Extracts a 10-digit CUSEC from the INE location text field.
    Supported formats:
      - "0100101001 Abegondo sección 01001"
      - "0100101001"  (numeric-only)
    """
    if pd.isna(text):
        return None
    s = str(text).strip()
    match = re.match(r'(\d{10})', s)
    if match:
        return match.group(1)
    return None
    13
    If renta.csv is absent, renta_abs is set to 0 for all sections and the pipeline continues.
    14
    15
    Step 3 — Process IGE population data
    16
    Function: process_poblacion_ige(gdf)
    Input: etl/data/poblacion_ige.csv
    17
    The IGE file provides demographic percentages by census section. The function:
    18
  • Reads the CSV with latin-1 encoding.
  • Identifies rows at census-section level by checking whether the third column contains a 10-digit numeric string (is_cusec_10 predicate).
  • Searches for Galician-language indicator substrings: "menor de 20 anos" for youth and "maior de 64 anos" for elderly population.
  • Merges the resulting pct_jovenes and pct_mayores columns onto the main GeoDataFrame.
    • 19
    If the IGE file is absent or no matching indicators are found, both columns remain at 0.0.
    20
    21
    Step 4 — Process OSM commercial activity
    22
    Function: process_osm(gdf)
    Input: etl/data/galicia-260424.osm.pbf
    23
    This step uses osmium to extract commercial POIs (Points of Interest) from the OSM PBF binary. The POIHandler class iterates over every OSM node and keeps those tagged with any shop key, the specific amenity values listed below, or an office key:
    24
    shop_tags    = {"shop"}
amenity_tags = {
    "restaurant", "cafe", "bar", "fast_food", "bank",
    "pharmacy", "clinic", "school", "supermarket", "marketplace"
}

class POIHandler(osmium.SimpleHandler):
    def __init__(self):
        super().__init__()
        self.points = []

    def node(self, n):
        tags = dict(n.tags)
        is_poi = any(
            key in shop_tags or key in amenity_tags or key == "office"
            for key in tags
        )
        if is_poi:
            self.points.append((n.location.lon, n.location.lat))
    25
    After extraction, the POIs are converted to a GeoDataFrame and joined to the census sections with gpd.sjoin(..., predicate="within"). The count per section becomes actividad_abs; dividing by area_km2 gives densidad_actividad, which feeds the log normalisation in step 5.
    26
    If the OSM file is missing or osmium is not installed, actividad_abs and densidad_actividad default to 0.
    27
    28
    Step 5 — Process Catastro building data
    29
    Function: process_catastro(gdf)
    Input: etl/data/catastro/**/*.ZIP (files whose name contains CONSTRU)
    30
    The function scans CATASTRO_DIR recursively for ZIP files matching the CONSTRU pattern (the Catastro construction layer). For each ZIP it:
    31
  • Reads the Shapefile layer inside with GeoPandas via the zip:// protocol.
  • Auto-detects the use column (CONSTRU, TIPO, currentUse, USO, …) and year column (FECHAALTA, ANO_CONS, beginning, …).
  • Concatenates all buildings into a single GeoDataFrame and spatially joins them to census sections.
  • Aggregates per section: n_edificios, n_comerciales (buildings classified as non-residential), and anio_medio (mean construction year).
  • Derives ratio_comercial = n_comerciales / n_edificios and modernidad = anio_medio (more recent = higher score after Min-Max normalisation).
    • 32
    agg["ratio_comercial"] = agg["n_comerciales"] / agg["n_edificios"].clip(lower=1)
agg["modernidad"]      = agg["anio_medio"]
    33
    Sections with no matching buildings receive ratio_comercial = 0.0 and modernidad = 1970.0 (the fallback year for missing dates).

    ​
    Normalisation

    After all five join steps, the script normalises every raw variable to [0, 1] using Min-Max normalisation over all ~3,800 Galicia sections. The minmax_norm function is defined as: 
    def minmax_norm(series: pd.Series) -> pd.Series:
    values = series.fillna(0)
    vmin = values.min()
    vmax = values.max()
    if vmax == vmin:
        return pd.Series(0.0, index=values.index)
    return ((values - vmin) / (vmax - vmin)).round(4)
    actividad_norm uses a log-normalised variant to prevent urban centres from saturating the colour scale: 
    def log_norm(series: pd.Series) -> pd.Series:
    values = series.fillna(0).clip(lower=0)
    log_vals = np.log1p(values)   # log(1 + x)
    return minmax_norm(log_vals)
    The full normalisation block: 
    gdf["renta_norm"]         = minmax_norm(gdf["renta_abs"])
gdf["densidad_norm"]      = minmax_norm(gdf["densidad"])
gdf["jovenes_norm"]       = minmax_norm(gdf["pct_jovenes"])
gdf["mayores_norm"]       = minmax_norm(gdf["pct_mayores"])
gdf["actividad_norm"]     = log_norm(gdf["densidad_actividad"])
gdf["uso_comercial_norm"] = minmax_norm(gdf["ratio_comercial"])
gdf["antiguedad_norm"]    = minmax_norm(gdf["modernidad"])

    ​
    The COLS_OUTPUT contract

    The script selects exactly 13 items from the GeoDataFrame before writing (11 data columns + geometry): 
    cols_output = [
    "cusec",
    "NMUN",             # municipality name (from Shapefile)
    "renta_norm",
    "renta_abs",
    "densidad_norm",
    "jovenes_norm",
    "mayores_norm",
    "poblacion_abs",
    "actividad_norm",
    "actividad_abs",
    "uso_comercial_norm",
    "antiguedad_norm",
    "geometry",
]
    This list is the ETL–frontend contract. Any column added here must also be declared in the MapViewer source-layer property expressions and in the generate_tiles.sh --include flags.

    ​
    Quality assertions

    validate_output() runs before the GeoJSON is written. It raises ValueError immediately if any assertion fails — the file is not written with invalid data: 
    def validate_output(output: gpd.GeoDataFrame) -> None:
    if len(output) < 1500:
        raise ValueError(f"Numero de secciones inesperado: {len(output)}")
    if output["cusec"].nunique() != len(output):
        raise ValueError("Existen CUSEC duplicados en la salida")

    norm_cols = [c for c in output.columns if c.endswith("_norm")]
    for column in norm_cols:
        if not output[column].between(0, 1).all():
            raise ValueError(f"{column} fuera del rango [0,1]")
    In addition, process_data.py prints the percentage of sections at zero for each _norm column. A figure above 20% triggers a console warning and suggests re-checking that data source.

    ​
    last_update.json

    After writing the GeoJSON, the script writes a small metadata file used by the frontend to display the “last updated” badge: 
    {
  "updated_at": "2024-04-26T14:32:01.123456+00:00",
  "sections_count": 3812,
  "year_data": "2022",
  "sources": {
    "geometries": "CNIG secciones censales",
    "renta": "INE Atlas de Renta (año 2022)",
    "osm": "OpenStreetMap Galicia (Geofabrik)",
    "catastro": "Sede Electronica del Catastro"
  },
  "variables": [
    "renta_norm",
    "densidad_norm",
    "jovenes_norm",
    "mayores_norm",
    "actividad_norm",
    "uso_comercial_norm",
    "antiguedad_norm"
  ]
}
    Both output files land in etl/data/processed/ and both must be uploaded to the server after tile generation.

    ​
    Expected output

    etl/data/processed/
├── galicia_scouting.geojson   ← ~30–80 MB depending on geometry precision
└── last_update.json           ← ~400 bytes
    Each pipeline step is independent and falls back gracefully. If renta.csv is missing, renta_abs and renta_norm are 0 for all sections. If the OSM PBF file is absent, actividad_* columns are 0. If the Catastro directory contains no ZIPs, uso_comercial_norm and antiguedad_norm are 0. The pipeline always completes and always writes a structurally valid GeoJSON — it never crashes due to a missing optional source.

    Build docs developers (and LLMs) love

    Get started for freeTalk to us