Automating Address Standardization for 911 Logs

A single mis-standardized address string is enough to misroute an emergency. When a county PSAP exports a batch of Computer-Aided Dispatch (CAD) call logs at 02:00 during a regional storm surge, the raw text routinely arrives as 123 N MAIN ST APT 4, 123 North Main Street #4, and 123 N. MAIN for the same location — plus PO Box artifacts, truncated suffixes, and rural-route placeholders that have no point geometry at all. The narrow failure this page solves is the one where a non-deterministic parser raises an exception on one ugly record and aborts the whole batch, so a dispatcher’s spatial join silently returns the wrong response polygon. The fix is a deterministic normalization stage that always returns a structured result and flags — never drops — anything it cannot resolve.

Root Cause and Operational Impact

The danger is not the messy text itself; it is that the messiness is non-uniform and exception-raising. Libraries such as usaddress and libpostal are probabilistic parsers tuned for well-formed mailing addresses. Fed a legacy CAD string with repeated labels (two StreetName tokens, an embedded apartment, a milepost), usaddress throws RepeatedLabelError. If that exception propagates, every subsequent record in the batch is lost, and the loss is silent: the dispatch console simply shows fewer incidents than were called in.

In Next Generation 911 (NG911) routing this is a life-safety defect, not a data-quality inconvenience. An address that fails to standardize fails to match the Master Street Address Guide (MSAG), so it cannot resolve to an Emergency Service Number, and the call routes to a default or neighbouring PSAP. The same upstream-contract assumptions that govern any Python ETL for Sensor & IoT Data apply here: the stage downstream of you assumes you hand it ordered, structured, in-bounds records, and it has no way to know that 4% of a surge batch quietly vanished into an unhandled exception.

Tiered Resolution Strategy

Resolve each raw string through an ordered fallback chain, descending from the definitive structured parse to a safe default that is never silently trusted:

  1. Definitive fix — primary structured parse. Run usaddress (or libpostal) and accept its components when it returns cleanly. This covers the bulk of standard municipal formats with full confidence.
  2. Deterministic regex extraction. On any parser exception, fall back to a compiled regex that pulls the house number and remaining street name. The result is partial but guaranteed, so the batch continues.
  3. Lookup normalization. Expand directional prefixes (N → NORTH) and street suffixes (ST → STREET) from pre-compiled, jurisdiction-specific maps so MSAG matching is comparing like with like.
  4. Non-routable detection. Flag PO Box, rural-route, and general-delivery artifacts that have no point geometry; these can never satisfy a spatial join and must be diverted, not coerced.
  5. Safe default with audit flag. Anything that survives to here is emitted with a low confidence score and an explicit audit flag, routed to a manual QA queue. The record is preserved and traceable — never dropped, never silently “fixed”.

Production Python Implementation

The routine below implements the full resolution path: chained parse, deterministic fallback, lookup normalization, non-routable flagging, structured logging, and an audit record emitted for every correction. It never raises out of the per-record path, so a single malformed string cannot halt the batch.

python 
import logging
import re
from dataclasses import asdict, dataclass
from typing import Optional

import usaddress

logger = logging.getLogger("ng911.address_standardizer")

# Pre-compiled jurisdictional lookup tables. Tune SUFFIX/DIRECTIONAL maps per county.
DIRECTIONAL_MAP: dict[str, str] = {
    "N": "NORTH", "S": "SOUTH", "E": "EAST", "W": "WEST",
    "NE": "NORTHEAST", "NW": "NORTHWEST", "SE": "SOUTHEAST", "SW": "SOUTHWEST",
}
SUFFIX_MAP: dict[str, str] = {
    "ST": "STREET", "AVE": "AVENUE", "BLVD": "BOULEVARD",
    "RD": "ROAD", "DR": "DRIVE", "LN": "LANE", "CT": "COURT", "HWY": "HIGHWAY",
}
NON_ROUTABLE = re.compile(r"PO\s*BOX|RURAL\s*ROUTE|\bRR\b|GENERAL\s*DELIVERY", re.IGNORECASE)
HOUSE_NUMBER = re.compile(r"^(\d+[\w-]*)")


@dataclass
class StandardizedAddress:
    number: Optional[str]
    prefix: Optional[str]
    name: Optional[str]
    suffix: Optional[str]
    confidence: float          # 1.0 = clean parse, 0.4 = regex fallback, 0.1 = unresolved
    flagged: bool              # True => route to manual QA queue, do not auto-trust
    raw: str                   # original string, retained for audit replay


def standardize(raw: str, record_id: str) -> StandardizedAddress:
    """Resolve one CAD address string. Never raises; always returns a record."""
    prefix: Optional[str] = None
    suffix: Optional[str] = None
    confidence = 1.0
    flagged = False

    try:
        # Tier 1: definitive structured parse.
        parsed = usaddress.parse(raw)
        components = {tag: val for val, tag in parsed}
        number = components.get("AddressNumber")
        name = components.get("StreetName")
        prefix = (components.get("StreetNamePreDirectional") or "").upper() or None
        suffix = (components.get("StreetNamePostType") or "").upper() or None
    except (usaddress.RepeatedLabelError, ValueError) as exc:
        # Tier 2: deterministic regex fallback — partial but guaranteed.
        logger.warning("usaddress fallback id=%s reason=%s", record_id, exc.__class__.__name__)
        m = HOUSE_NUMBER.search(raw)
        number = m.group(1) if m else None
        name = re.sub(r"^\d+[\w-]*\s*", "", raw).strip() if number else raw.strip()
        confidence = 0.4

    # Tier 3: lookup normalization so MSAG compares like with like.
    if prefix:
        prefix = DIRECTIONAL_MAP.get(prefix, prefix)
    if suffix:
        suffix = SUFFIX_MAP.get(suffix, suffix)

    # Tier 4: non-routable detection (no point geometry can satisfy a spatial join).
    if NON_ROUTABLE.search(raw):
        flagged = True
        confidence = min(confidence, 0.1)
        logger.info("non-routable artifact id=%s raw=%r -> manual QA", record_id, raw)

    # Tier 5: anything we could not resolve is flagged, never silently trusted.
    if number is None or name is None:
        flagged = True
        confidence = min(confidence, 0.1)

    result = StandardizedAddress(
        number=number, prefix=prefix, name=name, suffix=suffix,
        confidence=confidence, flagged=flagged, raw=raw,
    )

    # Audit trail: one structured row per record for post-incident review.
    logger.info("standardized id=%s confidence=%.1f flagged=%s out=%s",
                record_id, confidence, flagged, asdict(result))
    return result

Records with flagged=False and full confidence flow straight into MSAG reconciliation; everything else lands in the manual QA queue with its original string intact for replay. Because reconciliation is a spatial-join problem, drive it through the same metric-CRS and library-selection discipline established in Geopandas vs PyShp for Field Operations — match standardized addresses to road centerlines within a 15-metre threshold and reject any candidate that crosses a jurisdictional boundary.

Validation Checklist

Verify each item against a staging copy of a real surge batch before deploying the standardizer to a live PSAP:

  • A batch containing a RepeatedLabelError-triggering string completes without aborting — no records are lost.
  • Every record that hits the regex fallback is emitted with confidence == 0.4 and a warning log line.
  • PO Box, rural-route, and general-delivery strings are flagged and routed to the manual QA queue, not coerced to geometry.
  • Directional and suffix expansion is exercised against the target county’s legacy naming, not just the default map.
  • MSAG reconciliation rejects matches beyond 15 metres and any cross-jurisdiction match.
  • One audit row exists per input record; the original raw string is replayable from the log.
  • Standardized output components are uppercased and trimmed so MSAG joins are case- and whitespace-stable.

Edge Cases and Gotchas

  • Axis-order / null-island drift downstream. Standardization produces text, but the geocoded result enters a spatial pipeline. A lat/lon swap there sends matches to (0, 0); keep pyproj transforms on always_xy=True and bounds-check before the join, exactly as the parent Python ETL for Sensor & IoT Data pattern does.
  • Agency-specific suffix collisions. ST means STREET in most counties but SAINT in place names like ST JOHNS AVE. Order your lookup so a leading ST token followed by a name is not expanded to STREET — false expansion silently breaks the MSAG match.
  • Unit/apartment leakage into the street name. usaddress usually isolates OccupancyIdentifier, but the regex fallback does not. Strip trailing APT, #, UNIT, and STE fragments before MSAG matching or the join rate collapses.
  • Encoding artifacts from legacy CAD exports. Mainframe exports often carry non-UTF-8 bytes (smart quotes, 0xA0 non-breaking spaces) that defeat both the parser and the regex. Normalise encoding on ingest and stage the raw bytes through Offline GIS Data Caching Strategies so a failed batch can be replayed rather than re-pulled.
  • Confidence inflation on partial parses. A regex fallback that finds a number and a plausible name still has no validated suffix. Never let lookup normalization raise its confidence back to 1.0 — the audit flag must survive to the QA queue.
Five-tier address-standardization fallback ladder for 911 CAD logs A raw Computer-Aided Dispatch address string enters a five-tier resolution ladder. Tier 1 runs the usaddress structured parser; a clean parse exits at full confidence 1.0. On a parser exception the record drops to Tier 2, a deterministic regex extraction that always returns a partial result at confidence 0.4. Tier 3 normalizes directional and suffix abbreviations against jurisdictional lookup tables. Tier 4 detects non-routable PO Box, rural-route, and general-delivery artifacts. Tier 5 emits a safe default at confidence 0.1 for anything still unresolved. Records that pass cleanly and are not flagged branch right into MSAG reconciliation with a fifteen-metre proximity threshold; every flagged record branches down into a manual QA queue that retains the original raw string for audit replay. Raw CAD address string 123 N. MAIN ST APT 4 · legacy / truncated / free-text 1 usaddress structured parse number · predir · name · post-type 2 Deterministic regex fallback on parser exception · partial but guaranteed 3 Lookup normalization N→NORTH · ST→STREET (per county) 4 Non-routable detection PO Box · rural route · general delivery 5 Safe default + audit flag unresolved · preserved · never trusted on exception → 0.4 expand & align scan artifacts still unresolved conf 1.0 conf 0.4 conf held conf 0.1 conf 0.1 flagged = False MSAG reconciliation ≤ 15 m to centerline reject cross-jurisdiction flagged = True Manual QA queue raw string retained for audit replay tiers 4 & 5 route here · never dropped

Up: Python ETL for Sensor & IoT Data