GeoSentinel ingests data from six structurally different sources — GDELT, ACLED, FIRMS, USGS, ADS-B, and MarineTraffic — each with its own timestamp format, coordinate convention, event vocabulary, and confidence characteristics. Without a unifying layer, downstream processing (clustering, severity scoring, the API) would need to understand every source independently. The canonical model solves this by requiring every source mapper to emit a singleDocumentation Index
Fetch the complete documentation index at: https://mintlify.com/danizd/GeoSentinel/llms.txt
Use this file to discover all available pages before exploring further.
EventCanonicalCreate object before any data is written to events_canonical. All dates are UTC, all coordinates are WGS84, all categories come from a fixed enum, and all severity and confidence values are normalized to the [0.0, 10.0] range.
EventCanonicalCreate Schema
Every source mapper must return an object that validates against theEventCanonicalCreate Pydantic model defined in backend/schemas/events.py. The validation layer (backend/validation/validator.py) then runs the six rejection rules before the event is persisted.
Validation constraints on
latitude, longitude, severity, confidence, and event_type are intentionally not enforced by Pydantic field validators. This allows malformed events to reach validate_event() in backend/validation/validator.py, where they are routed to events_quarantine rather than raising an unhandled exception.Field Reference
The source-native identifier for this event. Used together with
source to form the composite unique key (source, event_id_source) that drives deduplication. See the Deduplication section for how each source constructs this key.Lowercase source identifier. One of:
gdelt, acled, firms, usgs, adsb, marinetraffic, liveuamap. Must match a row in the sources_metadata table.The moment the real-world event occurred. Always timezone-aware UTC — the mapper is responsible for normalization before populating this field. See UTC Normalization for per-source conversion rules.
Fine-grained classification within the category. Examples:
conflict_battle, conflict_explosion, wildfire_hotspot, earthquake. Mapped from each source’s native vocabulary by the mapper. Events with a null or empty event_type are rejected with NULL_EVENT_TYPE.Coarse-grained event category. One of
conflict, disaster_natural, wildfire, mobility, humanitarian, thermal_anomaly, other. The clustering job processes each category in isolation — see Category Isolation.WGS84 latitude in decimal degrees. Must be in
[-90, 90]; null or out-of-range values are rejected.WGS84 longitude in decimal degrees. Must be in
[-180, 180].Estimated radius of positional uncertainty in kilometres. USGS provides this natively; FIRMS hotspots have a nominal 375 m or 1 km pixel depending on satellite. Used in canonical-point calculation inside the clustering job.
ISO 3166-1 alpha-2 country code, e.g.
UA, SD, US. Populated by the mapper via reverse geocoding or source metadata. Used as a fast filter in the API and AOI matching.First-level administrative division name (state, province, oblast). Optional but populated whenever the source provides it.
List of actor objects involved in the event. Each actor carries a canonical
role (see Actor Roles below), a human-readable name, and an optional cameo_code. Never reject an event because an actor’s code is unrecognized — use role='unknown' and preserve the original name.Reported fatality count.
null means unknown; -1 is the ACLED convention for “unknown” and is explicitly allowed. Values below -1 are rejected with NEGATIVE_FATALITIES.Normalized severity in
[0.0, 10.0]. Computed by each mapper from source-native metrics (magnitude, FRP, fatality count). See Confidence & Severity.Per-event confidence in
[0.0, 10.0]. Reflects the source’s independence class and data quality. See Confidence & Severity.Canonical URL of the source record for human review (ACLED event page, USGS event detail page, etc.).
Array of supplementary references or notes. ACLED populates this with the raw
notes field (truncated to 500 characters) and the originating source publication. FIRMS populates it with satellite, instrument, and brightness values.Set to
true when the source explicitly flags the event as unconfirmed or rumoured. GDELT media events may set this when Goldstein scale or tone indicates low reliability.Event Categories
Thecategory field is enforced by a CHECK constraint on the events_canonical table. Clustering, severity bucketing, and API filters all operate on these values.
| Category | Description | Primary Sources |
|---|---|---|
conflict | Armed conflict, battles, explosions, violence against civilians | ACLED, GDELT, Liveuamap |
disaster_natural | Floods, storms, landslides, volcanic activity | GDELT, ReliefWeb |
wildfire | Forest fires and vegetation fires detected by satellite | FIRMS |
mobility | Aircraft and maritime vessel movements of interest | ADS-B, MarineTraffic |
humanitarian | Displacement, refugee flows, humanitarian crises | ACLED, ReliefWeb |
thermal_anomaly | Industrial heat anomalies distinct from wildfires | FIRMS |
other | Events that do not fit the above categories | Any |
Validation Rules
Before a mapper’s output reachesevents_canonical, it passes through validate_event() in backend/validation/validator.py. Any rule violation routes the raw payload to events_quarantine with the corresponding rejection code and detail string.
| Code | Rejection Condition |
|---|---|
INVALID_COORDS | latitude < -90 or latitude > 90 or longitude < -180 or longitude > 180 |
NULL_COORDS | latitude is None or longitude is None |
FUTURE_DATE | event_time > now() + 1 hour (1-second tolerance applied to account for processing latency) |
NULL_EVENT_TYPE | event_type is None or str(event_type).strip() == "" |
NEGATIVE_FATALITIES | fatalities < -1 (-1 is the ACLED unknown-fatalities sentinel and is permitted) |
SCHEMA_ERROR | Raw payload failed Pydantic parsing before reaching the validator |
backend/validation/validator.py:
raw_payload so an operator or automated job can inspect, correct, and re-enqueue them. A quarantine alert fires when more than 100 unresolved records accumulate within a 30-minute window.
UTC Normalization
Every source uses a different timestamp convention. Normalization to UTC happens inside the mapper, never afterward. The table below shows the conversion for each active source:| Source | Original Field | Original Format | Conversion |
|---|---|---|---|
| ACLED | event_date | YYYY-MM-DD (date only, no time) | Assign 00:00:00 UTC |
| GDELT | event_date | YYYY-MM-DD string | Parse as date, assign 00:00:00 UTC |
| FIRMS | acq_date + acq_time | YYYY-MM-DD + HHMM | Combine and interpret directly as UTC |
| USGS | properties.time | Epoch milliseconds | datetime.fromtimestamp(ms / 1000, tz=UTC) |
| ADS-B | t | Unix epoch seconds | datetime.fromtimestamp(t, tz=UTC) |
| MarineTraffic | TIMESTAMP | ISO 8601 with UTC offset | Convert offset to UTC |
ACLED and GDELT provide only a date, not a time. By convention, GeoSentinel assigns midnight UTC (
00:00:00 UTC). Downstream consumers must treat these timestamps as date-precision only.Deduplication
Theevents_canonical table enforces a UNIQUE(source, event_id_source) constraint. Before inserting, the pipeline checks whether a record with the same composite key already exists:
- Exists → update
ingest_timeif the payload changed; do not create a duplicate. - Does not exist → insert.
DEDUP_WINDOW_DAYS=60) to catch retroactive corrections, which ACLED publishes regularly.
| Source | event_id_source Construction |
|---|---|
| GDELT | id cast to text |
| ACLED | data_id cast to text |
| FIRMS | sha256(lat | lon | acq_date | acq_time | satellite)[:32] — synthetic key because FIRMS has no native ID |
| USGS | properties.ids.split(",")[0].strip(",") — first preferred ID from the comma-separated list |
| ADS-B | hex + ":" + t (ICAO hex code + Unix timestamp) |
| MarineTraffic | mmsi + ":" + TIMESTAMP |
| Liveuamap | id cast to text |
