Open Data Sources Powering the UrbanViable ETL Pipeline

UrbanViable’s scoring model operates entirely on open public data. Four datasets are downloaded to the developer workstation before each ETL run: the CNIG census section geometries that form the spatial base layer, the INE Atlas de Renta that quantifies purchasing power, the Geofabrik OpenStreetMap extract that captures commercial activity density, and the Catastro vectorial files that characterise the building stock. Together they produce the 11-column galicia_scouting.geojson consumed by the frontend.

Sources summary

#	Source	Provider	Local path	Format	Variables generated
1	Secciones Censales	CNIG / INE	`etl/data/secciones_censales/`	Shapefile (ZIP)	Geometry, `cusec`, `poblacion_abs`, `densidad_norm`
2	Atlas de Renta	INE	`etl/data/renta.csv`	Long-format CSV	`renta_abs`, `renta_norm`
3	OpenStreetMap Galicia	Geofabrik / OSM	`etl/data/galicia-260424.osm.pbf`	PBF (binary)	`actividad_abs`, `actividad_norm`
4	Catastro Vectorial	Sede Electrónica del Catastro	`etl/data/catastro/[provincia]/`	ZIP per municipality	`uso_comercial_norm`, `antiguedad_norm`

Sources 1 and 2 are required for the minimum viable product. Sources 3 and 4 enrich the model with economic activity and urban fabric variables; if their data files are absent the pipeline falls back gracefully to 0.0 for the affected columns.

1. CNIG Secciones Censales

URL: https://www.ine.es/dyngs/DAB/index.htm?cid=1389
Local path: etl/data/secciones_censales/
Format: Shapefile compressed in a ZIP archive This dataset is the geometric backbone of the project. Every other data source is a table of attributes that gets joined onto it. Each row represents one census section (sección censal) — the smallest administrative unit for which INE publishes socio-economic statistics in Spain. Galicia has approximately 3,800 census sections distributed across its four provinces.

Coordinate reference system

Stage	CRS	EPSG code
Original Shapefile	ETRS89 / UTM zone 29N	`25829` or `25830`
Area calculation	ETRS89 / UTM zone 30N	`25830` (metric, for `area_km2`)
Output GeoJSON	WGS 84	`4326` (required by Tippecanoe and MapLibre GL)

The reprojection is handled by GeoPandas:

gdf_metr = gdf.to_crs(epsg=25830)
gdf["area_km2"] = gdf_metr.geometry.area / 1_000_000   # area in km²
gdf = gdf.to_crs(epsg=4326)                            # final output CRS

Area is calculated before reprojecting to WGS 84. Working in a metric CRS (ETRS89 UTM) avoids the angular-unit distortion that would result from computing area in degrees.

The CUSEC identifier

Every census section is uniquely identified by a 10-digit code called CUSEC. The ETL uses cusec (lowercase) as the join key throughout.

Segment	Digits	Meaning	Example
`CCPP`	2	Province code	`15` = A Coruña
`MMM`	3	Municipality code	`001` = Abegondo
`DDD`	3	District code	`001`
`SS`	2	Section code	`01`
Full CUSEC	10	Unique section ID	`1500100101`

Galicia province codes: 15 (A Coruña), 27 (Lugo), 32 (Ourense), 36 (Pontevedra). The ETL filters non-Galicia sections at load time:

GALICIA_PROVINCIAS = {"15", "27", "32", "36"}
gdf = gdf[gdf["CUSEC"].str[:2].isin(GALICIA_PROVINCIAS)].copy()
gdf["cusec"] = gdf["CUSEC"].astype(str).str.zfill(10)

2. INE Atlas de Renta de los Hogares

URL: https://www.ine.es/dynt3/inebase/index.htm?padre=12385&capsel=12384
Local path: etl/data/renta.csv
Format: Long-format CSV (statistical pivot, not a flat table) The INE Atlas de Renta publishes mean household net income at census-section level. It is the primary proxy for purchasing power in the scoring model.

CSV structure

The file is not a direct cusec → income table. It is a long-format statistical export where each row represents a (location × indicator × year) combination:

Municipios y secciones censales;Indicadores de renta media y mediana;Periodo;Total
- A Coruña  Sección:001 01;Renta neta media por hogar;2021;32150
- A Coruña  Sección:001 01;Renta neta media por persona;2021;13200
- A Coruña  Sección:001 01;Renta neta media por hogar;2020;31800

The ETL uses column positions (not names) to locate indicator, period, and value fields, and extracts the CUSEC by parsing the section text field with parse_cusec_ine():

def parse_cusec_ine(text: str) -> Optional[str]:
    """
    Extracts a 10-digit CUSEC from the INE location text field.
    Supported formats:
      - "0100101001 Abegondo sección 01001"
      - "0100101001"  (numeric-only)
    """
    if pd.isna(text):
        return None
    s = str(text).strip()
    match = re.match(r'(\d{10})', s)
    if match:
        return match.group(1)
    return None

The INE CSV format changes between annual releases. The column order, separator character (; or ,), encoding (latin-1 or utf-8-sig), and exact indicator label text ("Renta neta media por hogar") have all varied across editions. Always inspect the downloaded file before running the ETL:

df = pd.read_csv("etl/data/renta.csv", sep=";", encoding="utf-8-sig", dtype=str)
print(df.columns.tolist())
print(df.head(3))
print(df.iloc[:, 1].unique())   # List all available indicators

Known issues

Suppressed data for small sections. The INE suppresses income values for sections with fewer than 100 households. These appear as "." or blank in the CSV. The ETL treats them as NaN and replaces them with 0 before normalisation:

df_renta["renta_abs"] = (
    pd.to_numeric(
        df_renta[col_valor].astype(str)
            .str.replace(".", "", regex=False)
            .str.replace(",", ".", regex=False),
        errors="coerce",
    )
    .fillna(0)
    .astype(int)
)

Sections with suppressed income data receive renta_norm = 0.0. If a high proportion of sections (>20%) fall at zero, the quality check in validate_output() will print a warning.

Variables generated

Column	Type	Description
`renta_abs`	`integer`	Mean household net income in euros (raw value)
`renta_norm`	`float [0,1]`	Min-Max normalised over all Galicia sections

3. OpenStreetMap Galicia (Geofabrik extract)

URL: https://download.geofabrik.de/europe/spain/galicia.html
Local path: etl/data/galicia-260424.osm.pbf
Format: OSM Protocol Buffer Format (compressed binary) The Geofabrik extract provides the geographic distribution of commercial activity across Galicia: shops, restaurants, services, offices, and other economic nodes. This enables calculation of business density per census section — a proxy for commercial vitality. The OSM PBF file (~200 MB) can be downloaded directly using download_data.py:

python etl/download_data.py --download-osm

Processing library: osmium

osmium is installed as a dependency of pyrosm, which is listed in requirements.txt. Running pip install -r requirements.txt is sufficient — no separate install step is needed. The POIHandler class (a subclass of osmium.SimpleHandler) iterates over every OSM node and filters those that carry commercial tags:

shop_tags    = {"shop"}
amenity_tags = {
    "restaurant", "cafe", "bar", "fast_food", "bank",
    "pharmacy", "clinic", "school", "supermarket", "marketplace"
}

class POIHandler(osmium.SimpleHandler):
    def __init__(self):
        super().__init__()
        self.points = []

    def node(self, n):
        tags = dict(n.tags)
        is_poi = any(
            key in shop_tags or key in amenity_tags or key == "office"
            for key in tags
        )
        if is_poi:
            self.points.append((n.location.lon, n.location.lat))

After extraction, a spatial join assigns each POI to its census section, and the result is aggregated to a count per section:

pois_proj      = pois_gdf.to_crs(epsg=25830)
sections_proj  = gdf.to_crs(epsg=25830)

join_osm = gpd.sjoin(
    pois_proj[["geometry"]],
    sections_proj[["cusec", "area_km2", "geometry"]],
    how="left",
    predicate="within",
)

activity = join_osm.groupby("cusec").size().reset_index(name="actividad_abs")

Log normalisation

Because commercial activity follows a power-law distribution (most sections have few establishments, but urban centres have hundreds), the ETL applies log normalisation to prevent saturation in the colour scale:

def log_norm(series: pd.Series) -> pd.Series:
    values = series.fillna(0).clip(lower=0)
    log_vals = np.log1p(values)   # log(1 + x)
    return minmax_norm(log_vals)

gdf["actividad_norm"] = log_norm(gdf["densidad_actividad"])

Variables generated

Column	Type	Description
`actividad_abs`	`integer`	Number of commercial establishments in the section
`actividad_norm`	`float [0,1]`	Business density normalised on a logarithmic scale

4. Catastro Vectorial (Sede Electrónica del Catastro)

URL: https://www.sedecatastro.gob.es/DescargaDatos/SECFormularioDescargas.aspx
Local path: etl/data/catastro/[provincia]/
Format: One ZIP file per municipality, each containing multiple Shapefile layers The Catastro vectorial data characterises the urban fabric of each census section: what proportion of the building stock has commercial or office use, and how modern the buildings are. Galicia has ~300 municipalities, so this source is downloaded and stored as a collection of ZIP files organised by province.

Directory structure

etl/data/catastro/
├── a_coruña/
│   ├── 15001_CONSTRU.ZIP
│   ├── 15002_CONSTRU.ZIP
│   └── ...
├── lugo/
├── ourense/
└── pontevedra/

Building use codes

The currentUse (or CONSTRU) field in the building layer can follow various classification schemes depending on the Catastro file format. The ETL’s is_commercial() function classifies a building as residential (and excludes it from the commercial ratio) only if its code matches one of the explicit residential codes below. Any other value — including mixed-use codes that contain + — is counted as commercial:

Residential code	Meaning
`I`	Residential (single dwelling)
`II`	Residential (collective dwelling)
`III`	Residential (other)
`P`	Parking / garage
`PI`	Parking integral to building

All other values (industrial, office, retail, public services, mixed-use, or unrecognised codes) are treated as commercial when computing ratio_comercial.

Building age

The beginning (or FECHAALTA) field contains a construction date string. The ETL extracts the 4-digit year and stores it directly as modernidad. Because Min-Max normalisation is applied afterwards, higher mean construction years map to higher antiguedad_norm values — more recently built sections score highest.

agg["ratio_comercial"] = agg["n_comerciales"] / agg["n_edificios"].clip(lower=1)
agg["modernidad"]      = agg["anio_medio"]    # higher year → higher score after norm

Data volume: Processing all ~300 municipality ZIPs can take several hours on a standard laptop. For validation runs, process only the 20 largest municipalities first. The ETL iterates over CATASTRO_DIR.rglob("*.ZIP"), so placing only a subset of ZIPs in the directory is a valid strategy.

Variables generated

Column	Type	Description
`uso_comercial_norm`	`float [0,1]`	Proportion of buildings with commercial/office use, normalised
`antiguedad_norm`	`float [0,1]`	Building stock recency normalised — higher = newer buildings

5. IGE Population data (jovenes_norm, mayores_norm)

URL: https://www.ige.gal/igebdt/selector.jsp?COD=6057&idioma=es
Selector: 6057 — Indicadores de poboación. Datos por seccións censais
Local path: etl/data/poblacion_ige.csv
Format: CSV The IGE (Instituto Galego de Estatística) publishes population indicators broken down to census-section level. The ETL uses two demographic percentages:

% jóvenes — share of the population under 20 years old
% mayores — share of the population over 64 years old

These indicators allow retail businesses to weight their site scores towards demographic profiles that match their customer base (e.g. toy stores filter for jovenes_norm; pharmacies filter for mayores_norm). The file is located at etl/data/poblacion_ige.csv. The ETL identifies CUSEC values by looking for 10-digit all-numeric codes in the third column, then searches for Galician-language indicator substrings ("menor de 20 anos", "maior de 64 anos") in the fourth column.

Variables generated

Column	Type	Description
`jovenes_norm`	`float [0,1]`	% population under 20, Min-Max normalised
`mayores_norm`	`float [0,1]`	% population over 64, Min-Max normalised

Data licences

Source	Licence	Required attribution
CNIG / INE — Secciones Censales	CC BY 4.0	”Fuente: IGN / INE”
INE — Atlas de Renta	Free reuse (Ley 37/2007)	“Fuente: INE”
IGE — Población	Public domain (Xunta de Galicia)	“Fonte: IGE — Instituto Galego de Estatística”
OpenStreetMap	ODbL 1.0	”© OpenStreetMap contributors”
Catastro Vectorial	Free reuse (RD 663/2007)	“Fonte: Sede Electrónica do Catastro”

All attributions must appear in the application footer and on the HowToUsePage.

Get Started

ETL Pipeline

Frontend

Deployment

Open Data Sources Powering the UrbanViable ETL Pipeline

Sources summary

1. CNIG Secciones Censales

Coordinate reference system

The CUSEC identifier

2. INE Atlas de Renta de los Hogares

CSV structure

Known issues

Variables generated

3. OpenStreetMap Galicia (Geofabrik extract)

Processing library: osmium

Log normalisation

Variables generated

4. Catastro Vectorial (Sede Electrónica del Catastro)

Directory structure

Building use codes

Building age

Variables generated

5. IGE Population data (jovenes_norm, mayores_norm)

Variables generated

Data licences

Build docs developers (and LLMs) love

Get Started

ETL Pipeline

Frontend

Deployment

Documentation Index

​Sources summary

​1. CNIG Secciones Censales

​Coordinate reference system

​The CUSEC identifier

​2. INE Atlas de Renta de los Hogares

​CSV structure

​Known issues

​Variables generated

​3. OpenStreetMap Galicia (Geofabrik extract)

​Processing library: osmium

​Log normalisation

​Variables generated

​4. Catastro Vectorial (Sede Electrónica del Catastro)

​Directory structure

​Building use codes

​Building age

​Variables generated

​5. IGE Population data (jovenes_norm, mayores_norm)

​Variables generated

​Data licences

Build docs developers (and LLMs) love

Sources summary

1. CNIG Secciones Censales

Coordinate reference system

The CUSEC identifier

2. INE Atlas de Renta de los Hogares

CSV structure

Known issues

Variables generated

3. OpenStreetMap Galicia (Geofabrik extract)

Processing library: osmium

Log normalisation

Variables generated

4. Catastro Vectorial (Sede Electrónica del Catastro)

Directory structure

Building use codes

Building age

Variables generated

5. IGE Population data (jovenes_norm, mayores_norm)

Variables generated

Data licences