Resolving Duplicate Incident Reports Across Jurisdictions

At 02:47 a structure fire is called in from a cell phone routed through a county PSAP, dispatched as INC-44021 in Computer-Aided Dispatch (CAD). Ninety seconds later the same column of smoke trips a city traffic-camera analytic and a neighboring mutual-aid agency self-dispatches an engine, writing CTY-9930 to its own CAD with coordinates 110 metres west and a clock skewed four minutes by an un-synced NTP server. The common operating picture now shows two fires on the same block. The resource-allocation model counts two engines committed where one is, the after-action metrics inflate, and an operations chief stares at a map that lies. This page solves exactly that failure: two incident records that describe one real-world event but never match on ID, exact coordinate, or timestamp, and the Python pattern that collapses them without ever silently merging two events that were genuinely distinct. It is the deduplication concern that sits on top of Geopandas vs PyShp for Field Operations, because the library you reach for decides whether this runs in a command-center ETL node or on an offline tablet at the post.

Root Cause and Operational Impact

Cross-boundary duplicates are not a data-entry mistake to be scolded out of existence — they are structural. Overlapping CAD dispatch zones, mutual-aid auto-dispatch, parallel sensor triggers (traffic cameras, gunshot detectors, smoke analytics), and multi-agency radio traffic all generate independent records for one event by design. None of them share a primary key, and three field-specific noise sources guarantee the records never line up exactly: GPS drift and differing Coordinate Reference System (CRS) implementations move the coordinate by tens of metres, un-synced dispatch clocks skew the timestamp by minutes, and free-text incident descriptions diverge entirely between agencies. Exact-match or raw-coordinate-equality filtering catches none of it.

In an office system a duplicate row is an inconvenience you fix at month-end. In an active incident it is operationally dangerous. A double-counted event corrupts real-time situational awareness, so the operations section commits or holds resources against a phantom. It poisons the deployment metrics that drive mutual-aid billing and the FEMA Incident Status Summary (NIMS ICS-209) rollup. And it breaks every downstream spatial join — buffer math, parcel intersection, hydrant assignment — because two geometries sit where one belongs. The fix must therefore be deterministic and conservative: a wrong merge that hides a second real fire is worse than a missed merge that leaves a visible duplicate, so the pattern biases toward flagging the ambiguous case for a human rather than auto-collapsing it.

Spatial-temporal deduplication flow: three reports, one event, three routed outcomes Three independent records for one fire — County CAD INC-44021 (02:47), City CAD CTY-9930 (02:51, 110 m west, clock skewed), and a traffic-camera sensor trigger — enter a normalization stage that reprojects all geometry to a common metric UTM CRS and applies a plus-or-minus ten-minute temporal window. Candidate pairs feed a composite scorer weighting spatial 0.5, temporal 0.3, and attribute 0.2. The score routes each pair into one of three lanes: at or above 0.85 auto-merge with a shared master_id; 0.65 to 0.85 to a human review queue, held out of resource counts; below 0.65 keep both as distinct. All three lanes write to one immutable audit log, making every decision reconstructable and any merge reversible. Three sources, one event Normalize & score Routed by confidence band County CAD INC-44021 · 02:47 PSAP cell-phone call City CAD CTY-9930 · 02:51 · 110 m W mutual-aid self-dispatch Sensor trigger traffic-camera analytic no shared primary key Normalize reproject → metric UTM CRS ±10 min temporal window UTC clocks · bounds-check Composite scorer spatial 0.5 + temporal 0.3 + attribute 0.2 → 0..1 per candidate pair Auto-merge · score ≥ 0.85 assign shared master_id both rows survive · reversible Review · 0.65–0.85 human queue · ops chief held out of resource counts Distinct · score < 0.65 keep both · tag provenance log the rejected near-miss Immutable audit log — every decision recorded master_id · dedup_state · dedup_score · both source IDs · keyed for idempotent replay

Tiered Resolution Strategy

Treat a candidate duplicate the way the CRS resolver treats a missing datum: resolve through an ordered chain from the most definitive evidence to a flagged safe default, and never let the highest-confidence tier be a silent guess. Evaluate three concurrent dimensions — spatial proximity, temporal overlap, and attribute similarity — combine them into one composite score, and route by confidence band:

  1. Definitive identity match. If both records carry a shared correlation key — a CAD-to-CAD exchange ID, a NIEM-XML IncidentTrackingIdentification, or a deduplicated 911 ANI/ALI reference — collapse them immediately. This is the only tier allowed to merge without spatial scoring.
  2. High-confidence spatial-temporal-attribute match. Within a tight time window and a metric proximity buffer, with matching NENA-compliant incident type codes, auto-merge and assign a master_id. Every merge emits an audit record naming both source IDs.
  3. Ambiguous match → review queue. A composite score in the uncertain band (here 0.65–0.85) routes to a human-review queue rather than auto-merging, preserving chain-of-custody for forensic audit. Resource counts treat the pair as possibly one and surface it for the ops chief, not as a settled merge.
  4. Distinct → keep both with audit flag. Below the threshold, retain both records, tag each with the evaluation provenance, and log the near-miss so a post-incident reviewer can see the deduplicator considered and rejected the pair.

Production Python Implementation

The resolver below runs the full path in one place: it harmonizes CRS to a metric projection, windows temporally to keep the comparison out of O(n²), scores each candidate pair on spatial, temporal, and attribute axes, then routes by confidence band and emits a structured audit record for every decision — merge, review, or keep. It uses print-free structured logging, explicit exception handling, and full type hints. It assumes geopandas >= 0.14 (Shapely 2.x / GEOS) and a projected CRS appropriate to the operational area — pick the Universal Transverse Mercator (UTM) zone for the incident, not a continental default, so buffer distances are true metres. The metric reprojection step depends on a correctly resolved input CRS; recovering one when the upstream feed omits it is the job of handling missing CRS in field-collected GPS logs.

python 
import logging
from dataclasses import dataclass
from datetime import timedelta
from difflib import SequenceMatcher
from typing import Optional

import geopandas as gpd
from shapely.strtree import STRtree

logger = logging.getLogger("dedup.incidents")


@dataclass(frozen=True)
class DedupConfig:
    metric_crs: str = "EPSG:32610"     # UTM zone for the operational area
    spatial_threshold_m: float = 150.0  # proximity buffer in true metres
    time_window_min: int = 10           # ± dispatch-to-dispatch skew tolerance
    auto_merge_score: float = 0.85      # >= this: collapse automatically
    review_floor_score: float = 0.65    # [floor, auto): route to human review


def _score_pair(row, cand, cfg: DedupConfig) -> float:
    """Composite 0..1 similarity across space, time, and incident type."""
    dist = row.geometry.distance(cand.geometry)
    spatial = max(0.0, 1.0 - dist / cfg.spatial_threshold_m)

    dt_min = abs((row["dispatch_time"] - cand["dispatch_time"]).total_seconds()) / 60.0
    temporal = max(0.0, 1.0 - dt_min / cfg.time_window_min)

    # NENA-style code equality is a hard signal; fall back to description fuzz.
    if row.get("incident_code") and row["incident_code"] == cand.get("incident_code"):
        attribute = 1.0
    else:
        attribute = SequenceMatcher(
            None,
            str(row.get("description", "")).lower(),
            str(cand.get("description", "")).lower(),
        ).ratio()

    return 0.5 * spatial + 0.3 * temporal + 0.2 * attribute


def resolve_duplicate_incidents(
    incidents: gpd.GeoDataFrame, cfg: Optional[DedupConfig] = None
) -> gpd.GeoDataFrame:
    """Collapse cross-jurisdiction duplicates, emitting an audit record per decision.

    Returns the original frame with three added columns:
      master_id   – id of the surviving record (self for masters/distinct)
      dedup_state – 'master' | 'merged' | 'review' | 'distinct'
      dedup_score – composite score against the assigned master (NaN for masters)
    """
    cfg = cfg or DedupConfig()
    if incidents.empty:
        logger.info("No incidents to deduplicate; returning empty frame")
        return incidents

    try:
        work = incidents.to_crs(cfg.metric_crs)
    except Exception:
        logger.exception("CRS reprojection to %s failed; aborting dedup", cfg.metric_crs)
        raise  # a wrong projection silently corrupts every distance; never proceed

    work = work.sort_values("dispatch_time").copy()
    work["master_id"] = work.index
    work["dedup_state"] = "master"
    work["dedup_score"] = float("nan")

    geoms = list(work.geometry.values)
    tree = STRtree(geoms)
    settled: set = set()

    for idx, row in work.iterrows():
        if idx in settled:
            continue

        # Temporal pre-filter keeps the comparison out of O(n^2).
        lo = row["dispatch_time"] - timedelta(minutes=cfg.time_window_min)
        hi = row["dispatch_time"] + timedelta(minutes=cfg.time_window_min)
        in_window = work[(work["dispatch_time"] >= lo) & (work["dispatch_time"] <= hi)]

        # Spatial pre-filter via the index, then score survivors precisely.
        near_pos = tree.query(row.geometry.buffer(cfg.spatial_threshold_m))
        near_idx = set(work.iloc[near_pos].index) & set(in_window.index)
        near_idx.discard(idx)

        for cand_idx in near_idx:
            if cand_idx in settled:
                continue
            score = _score_pair(row, work.loc[cand_idx], cfg)

            if score >= cfg.auto_merge_score:
                work.loc[cand_idx, ["master_id", "dedup_state", "dedup_score"]] = (
                    idx, "merged", score,
                )
                settled.add(cand_idx)
                logger.info(
                    "MERGE master=%s merged=%s score=%.3f", idx, cand_idx, score
                )
            elif score >= cfg.review_floor_score:
                work.loc[cand_idx, ["master_id", "dedup_state", "dedup_score"]] = (
                    idx, "review", score,
                )
                logger.warning(
                    "REVIEW pair=(%s,%s) score=%.3f routed to manual queue",
                    idx, cand_idx, score,
                )
            else:
                logger.debug(
                    "DISTINCT pair=(%s,%s) score=%.3f kept separate", idx, cand_idx, score
                )

    # Re-attach results to the caller's original CRS frame by index.
    out = incidents.copy()
    out[["master_id", "dedup_state", "dedup_score"]] = work[
        ["master_id", "dedup_state", "dedup_score"]
    ]
    return out

Persist master_id, dedup_state, and dedup_score straight to the audit store; they are the chain-of-custody record that lets a reviewer reconstruct why two records became one. Never overwrite the source IDs in place — masters and merged records must both survive so a mistaken merge is reversible. The same scoring philosophy underpins automated attribute validation rules: normalize the incident-type codes there before they reach the attribute axis here, or the fuzzy fallback will carry the whole decision.

The three weights are not free parameters — each one degrades differently as the incident changes, and knowing which is about to become unreliable is most of the skill in tuning them.

When each scoring component stops being trustworthy The three components of the composite duplicate score, with the conditions that degrade each. Spatial proximity, weighted 0.5, is reliable until reports come from GPS in an urban canyon or from a milepost reference, at which point positions from the same incident can differ by hundreds of metres. Temporal overlap, weighted 0.3, is reliable until dispatch queues back up during a surge, when two reports of one event can be timestamped twenty minutes apart. Attribute similarity, weighted 0.2, is reliable until a mutual-aid partner joins using a different incident-type vocabulary, at which point the same event carries unrelated type codes. Crucially the degradations are not simultaneous: an urban surge breaks spatial and temporal together while leaving attributes intact, which is exactly when a fixed weighting is most wrong. each component fails under different conditions — and rarely at the same time spatial proximity · weight 0.5 degrades with: urban-canyon GPS multipath · milepost-derived positions · a cell-sector centroid two reports of one incident can then sit 300 m apart, scoring as distinct temporal overlap · weight 0.3 degrades with: dispatch queue backlog during surge · a partner CAD bridge batching its pushes two reports of one event can then be timestamped 20 minutes apart attribute similarity · weight 0.2 degrades with: a mutual-aid partner using a different incident-type vocabulary the same event then carries type codes with no lexical relationship at all

The interaction is what makes a fixed weighting fragile. An urban surge degrades spatial and temporal together — multipath from the buildings, backlog from the volume — while leaving attributes untouched, so the 0.8 of the score that has become unreliable outvotes the 0.2 that has not. A mutual-aid influx does the reverse: positions and times stay good and the vocabulary diverges, so a weighting that leans on attributes starts splitting genuine duplicates.

Two responses are worth more than re-tuning the constants. The first is to make each component report its own confidence alongside its score, and to renormalise the weights across the components that are currently trustworthy rather than applying fixed ones. A spatial component that knows its inputs are cell-sector centroids can say so, and the composite can lean on time and attributes instead.

The second is to widen the review band rather than move the auto-merge threshold when conditions degrade. Sending more pairs to a human is a visible, reversible cost; lowering the merge threshold to compensate for noisy inputs is an invisible, irreversible one — it merges incidents that were never the same event, and the audit record will faithfully report a confident score for each.

Validation Checklist

Confirm each item before a deduplication build is cleared for field deployment:

  • The metric CRS in DedupConfig is the UTM zone for the actual incident area, not a continental or web-mercator default — buffer distances must be true metres.
  • Every merged and review record carries master_id and dedup_score written to the audit log alongside both source IDs.
  • auto_merge_score and review_floor_score are tuned against a labeled sample from this jurisdiction, not left at the example defaults.
  • A definitive identity tier (CAD exchange ID / NIEM IncidentTrackingIdentification) short-circuits scoring when a shared correlation key exists.
  • Records in the review band route to a human queue and are excluded from auto-merge resource counts until cleared.
  • Source records are never overwritten in place — both master and merged rows survive so any merge is reversible.
  • Dispatch timestamps are normalized to a single timezone/UTC before windowing, so NTP skew between agency clocks is bounded by time_window_min, not by timezone offset.

Setting the two thresholds is a choice about which error you would rather make, and the two errors are not symmetric.

The operational cost of each error, and why the two thresholds are set differently Two errors are possible. A false merge combines two genuinely distinct incidents into one record: one of them disappears from the resource count, no unit is assigned to it, and because the surviving record looks complete nothing signals the loss — recovery requires someone to notice an incident that is not on the board. A false split leaves one incident as two records: it is double-counted in the resource tally, two units may be assigned where one was needed, and it is discovered within minutes by the second unit arriving on scene. Both are errors; only one of them removes an incident from the response. This asymmetry is why the auto-merge threshold sits high, the review band is wide, and the default when confidence is low is to keep both records. both are errors — only one of them removes an incident from the response false merge — two incidents become one false split — one incident stays two · one incident leaves the resource count · no unit is assigned to it · the surviving record looks complete · nothing signals the loss · the incident is double-counted · two units may be assigned · both records are present and visible · the second unit on scene reports it discovered when somebody notices an absence which during a surge may be never discovered in minutes, on scene costs one wasted response so: merge threshold high, review band wide, and when confidence is low keep both records

The asymmetry is the whole justification for a 0.85 auto-merge threshold that will obviously leave real duplicates unmerged. Those duplicates land in the review queue, where a human clears them in seconds, and in the meantime the worst outcome is a resource tally that is slightly overstated — a condition an operations chief is well equipped to reason about, because it is visible.

A false merge is not visible. The record that survives carries a plausible location, a plausible type and a master identifier; the record that vanished leaves no trace on any board. Discovering it requires somebody to notice that a call which came in forty minutes ago has no unit assigned, which is precisely the kind of attention a surge does not have spare.

This also settles a question that comes up when the review queue grows: whether to raise the auto-merge rate to reduce reviewer load. The answer is no — the correct lever is more reviewers or better upstream normalisation, because trading queue depth for silent merges converts a visible, bounded cost into an invisible, unbounded one.

Edge Cases and Gotchas

  • Axis-order inversion. A neighboring agency exporting GeoJSON as lat/lon while yours is lon/lat transposes a point thousands of kilometres away, so it never enters any candidate set and the duplicate survives unmerged. Enforce always_xy=True on every transform and bounds-check coordinates against the incident extent before scoring.
  • Null-island drift. A (0, 0) fix from a sensor that failed to acquire is within spatial_threshold_m of every other (0, 0) failure, so naive proximity merges unrelated events into one phantom incident at the equator. Filter and quarantine null-island coordinates before the spatial index is built.
  • Clock skew beyond the window. Un-synced agency CAD clocks can exceed time_window_min, pushing a true duplicate outside the temporal pre-filter so it is never compared. Normalize all timestamps to UTC on ingest and widen the window only with a compensating raise in auto_merge_score, or genuine duplicates fall through silently.
  • Agency-specific datum anomalies. A mutual-aid feed still publishing NAD27 or a State Plane grid lands 10–100 m off your WGS 84 basemap, inflating the measured distance so a real duplicate scores below threshold. Reproject with a real datum-transformation grid per the coordinate reference systems for disaster zones workflow before the records ever reach this resolver.
  • Offline device replay. A tablet syncing after an outage replays a batch of already-merged records; without idempotency the resolver merges them a second time and corrupts the master_id chain. Key the audit store on source ID so a re-seen record is recognized as settled, not re-scored.

Up: Geopandas vs PyShp for Field Operations

Other guides in Geopandas vs PyShp for Field Operations: Choosing the Right Spatial Library

Related
Benchmarking GeoPandas vs PyShp Throughput Under Surge Load
→