Deduplicating CAP Updates Across Alerting Authorities

A public-facing alert map shows two active evacuation orders for the same neighbourhood with different boundaries. Both are real CAP messages, both are current, and one of them was superseded twenty minutes ago — by an update that referenced the county’s original identifier, which the relayed copy never carried.

Root Cause and Operational Impact

CAP is designed to be relayed. A county issues an alert, a state emergency operations centre republishes it, a federal aggregator republishes that, and each hop is entitled to assign its own identifier and sender. That is correct behaviour and it means a consumer aggregating several feeds routinely receives one decision several times, under identifiers that have nothing in common.

One evacuation order arriving four times from three authorities A single evacuation decision reaches an aggregator four times. The county emergency manager issues the original CAP alert. The state emergency operations centre relays it under its own sender identifier with a new message identifier. A federal aggregator republishes the state copy, again renumbering. The county then issues a genuine update extending the boundary, referencing only its own original. To a naive consumer these are four unrelated alerts covering overlapping areas, and the genuine update references an identifier two of the copies never carried. The problem is not duplication for its own sake — it is that the update supersedes one copy and leaves the other two standing, so a handset can end up showing a superseded boundary as current. one decision, four messages, three sender identifiers county · id A · Alert the original decision, 14:02 state relay · id B · Alert · references none same decision, new sender, new identifier, 14:04 federal aggregator · id C · Alert republished from B, renumbered again, 14:07 county · id D · Update references A only · boundary extended, 14:26 B and C are not superseded a handset holding either still shows the old boundary as current The duplication itself is harmless. The update that supersedes one copy and leaves two standing is not. Which is why deduplication has to key on the decision, not on the message.

Duplication on its own is a display nuisance. The real failure is what happens at the update. When the county extends the boundary and issues an update, its references names only its own original identifier — the one the state and federal copies were never issued under. Those copies are not superseded, so a consumer holding them continues to present the old boundary as current, alongside the new one.

That is the operational cost: not two copies of one message, but one message that has been withdrawn and one that has not, describing the same neighbourhood with different instructions.

Tiered Resolution Strategy

  1. Key on the decision, not the message (definitive). Store messages under sender plus identifier, which is their true unique key, but group them under a decision identifier the consumer assigns.
  2. Use the references chain when it is present. It is authoritative and cheap. It is also absent far more often than the specification implies, so it cannot be the only mechanism.
  3. Fall back to a content hash over the fields that carry meaning. Event, area, effective window and instruction. Two byte-equivalent relays hash identically regardless of who sent them.
  4. Fall back again to a spatio-temporal key, and flag whatever it merges (safe default). Overlapping area, same event type, effective windows within a tolerance. This catches relays that reworded the headline, and it can occasionally merge two genuinely distinct nearby incidents — so anything it merges is flagged rather than merged silently.
  5. Apply supersession across the whole decision, never to a single message. This is the step that fixes the failure above, and it is only possible because the copies were kept rather than discarded.
Candidate keys for recognising two messages as one decision Five ways to decide that two CAP messages describe the same decision. The message identifier alone fails immediately, because every relay assigns a new one. Sender plus identifier is the correct uniqueness key for a message but says nothing about equivalence across senders. The references chain works when relays populate it and fails silently when they do not, which is common. A content hash over the event, area, effective time and instruction recognises byte-equivalent relays and misses ones that reworded the headline. A spatio-temporal key — overlapping area, same event type, effective windows within a tolerance — recognises relays that reworded, at the cost of occasionally merging two genuinely distinct nearby incidents. The practical answer is the content hash first, falling back to the spatio-temporal key, with anything merged by the fallback flagged for review. five candidate keys, and where each one breaks identifier every relay assigns a new one — fails immediately useless not even a message key on its own sender + identifier correct message key · says nothing about equivalence across senders necessary use it for storage, not for dedup references chain works when relays populate it · silently absent more often than not when present content hash event + area + effective + instruction · catches byte-equivalent relays primary spatio-temporal overlapping area, same event, effective within tolerance — catches rewording fallback can merge two genuinely distinct nearby incidents — flag whatever it merges

The layering matters because each key is wrong on its own. The references chain is exact and frequently missing. A content hash is exact and defeated by a relay that changed one word of the headline. The spatio-temporal key catches those and is the only one that can produce a false merge, which is why it is last and why its merges are reviewable.

What a decision record holds once copies are collapsed Rather than picking one message and discarding the rest, the consumer keeps a decision record. It carries a decision identifier of its own, the set of message identifiers and senders that have been recognised as that decision, the currently authoritative version chosen by the highest-precedence sender, and the supersession state. When the county issues an update referencing only its own original, the decision record applies the supersession to every member of the set, so the state and federal copies are marked superseded too. Discarding duplicates instead would leave nothing to apply the supersession to, which is exactly how a superseded boundary keeps showing as current. collapse into a decision record — never discard the copies decision d-7f3a members · county/A · state/B · federal/C · county/D authoritative · county/D — highest-precedence sender, latest sent supersedes · A, and by membership B and C as well merged by · references chain for A→D, content hash for B, spatio-temporal for C review flag · set, because C was merged by the fallback key why not just keep the newest and drop the rest? because a later supersession has to be applied to every copy, and a discarded copy is one nothing can be applied to

Keeping the copies is the part that looks wasteful and is load-bearing. A consumer that dedupes by discarding has nothing to apply a later supersession to — the very copies that need withdrawing are the ones it threw away.

Production Python Implementation

python 
from __future__ import annotations

import hashlib
import logging
from dataclasses import dataclass, field
from datetime import datetime, timedelta

from shapely.geometry.base import BaseGeometry

logger = logging.getLogger("incidentgis.cap_dedup")

EFFECTIVE_TOLERANCE = timedelta(minutes=10)
AREA_OVERLAP_MIN = 0.80
# Higher wins when choosing which copy is authoritative for a decision.
SENDER_PRECEDENCE = {"county": 3, "state": 2, "federal": 1}


@dataclass(frozen=True)
class CapMessage:
    sender: str
    identifier: str
    msg_type: str                   # Alert | Update | Cancel | Error
    references: tuple[str, ...]
    event: str
    area: BaseGeometry
    effective: datetime
    instruction: str
    sent: datetime

    @property
    def key(self) -> tuple[str, str]:
        """The message's true unique key — never the identifier alone."""
        return (self.sender, self.identifier)

    @property
    def content_hash(self) -> str:
        """Hash the fields that carry the decision, not the presentation."""
        payload = "|".join([
            self.event,
            self.area.wkt,
            self.effective.isoformat(),
            self.instruction.strip().lower(),
        ])
        return hashlib.sha256(payload.encode()).hexdigest()


@dataclass
class Decision:
    decision_id: str
    members: set[tuple[str, str]] = field(default_factory=set)
    authoritative: CapMessage | None = None
    superseded: bool = False
    needs_review: bool = False
    merged_by: dict[str, str] = field(default_factory=dict)


class DecisionIndex:
    """Group relayed CAP messages into the decisions they describe."""

    def __init__(self) -> None:
        self._by_key: dict[tuple[str, str], str] = {}
        self._by_hash: dict[str, str] = {}
        self._decisions: dict[str, Decision] = {}
        self._counter = 0

    def _new_decision(self) -> Decision:
        self._counter += 1
        d = Decision(decision_id=f"d-{self._counter:06d}")
        self._decisions[d.decision_id] = d
        return d

    def _match(self, msg: CapMessage) -> tuple[Decision | None, str]:
        # Tier 2: the references chain, when the relay bothered to populate it.
        for ref in msg.references:
            for (sender, ident), did in self._by_key.items():
                if ident == ref:
                    return self._decisions[did], "references"
        # Tier 3: byte-equivalent relays.
        did = self._by_hash.get(msg.content_hash)
        if did:
            return self._decisions[did], "content_hash"
        # Tier 4: reworded relays — may over-merge, so it is flagged.
        for d in self._decisions.values():
            other = d.authoritative
            if other is None or other.event != msg.event:
                continue
            if abs(other.effective - msg.effective) > EFFECTIVE_TOLERANCE:
                continue
            inter = other.area.intersection(msg.area).area
            union = other.area.union(msg.area).area
            if union and inter / union >= AREA_OVERLAP_MIN:
                return d, "spatio_temporal"
        return None, "new"

    def ingest(self, msg: CapMessage) -> Decision:
        decision, how = self._match(msg)
        if decision is None:
            decision = self._new_decision()

        decision.members.add(msg.key)
        decision.merged_by[f"{msg.sender}/{msg.identifier}"] = how
        if how == "spatio_temporal":
            # A merge the fallback made is a merge a human should confirm.
            decision.needs_review = True

        self._by_key[msg.key] = decision.decision_id
        self._by_hash.setdefault(msg.content_hash, decision.decision_id)

        if msg.msg_type == "Cancel":
            # Supersession applies to the decision, so every relayed copy is
            # withdrawn — including ones whose identifier was never referenced.
            decision.superseded = True
        elif decision.authoritative is None or _outranks(msg, decision.authoritative):
            decision.authoritative = msg

        logger.info("cap_message_ingested", extra={
            "decision": decision.decision_id, "matched_by": how,
            "members": len(decision.members), "review": decision.needs_review,
        })
        return decision


def _outranks(a: CapMessage, b: CapMessage) -> bool:
    """Higher-precedence sender wins; ties break on the later sent time."""
    pa = SENDER_PRECEDENCE.get(a.sender, 0)
    pb = SENDER_PRECEDENCE.get(b.sender, 0)
    return (pa, a.sent) > (pb, b.sent)

Validation Checklist

  • Messages are stored under sender plus identifier, never under the identifier alone.
  • The references chain is used first and its absence is expected rather than treated as an error.
  • The content hash covers event, area, effective window and instruction — and not the headline.
  • Spatio-temporal merges set a review flag and are visible in the operator interface.
  • Supersession is applied to every member of a decision, not to the referenced message alone.
  • Copies are retained after merging, so a later supersession has something to apply to.
  • Sender precedence is configuration, agreed with the participating authorities in advance.
  • A fixture reproduces the county-state-federal relay chain and asserts one decision with four members.

Edge Cases and Gotchas

  • A relay that alters the polygon slightly. Reprojection round trips can move vertices by centimetres, which changes the WKT and defeats the content hash. Normalise coordinate precision before hashing, exactly as the deterministic artifact rule requires elsewhere.
  • Two real incidents in the same block within ten minutes. The spatio-temporal fallback will merge them. That is the cost of catching reworded relays, and the review flag is what makes it recoverable — do not tighten the tolerance until it stops catching relays.
  • A cancel that arrives before the alert it cancels. Feeds reorder. Create the decision on the cancel and mark it superseded, so the alert is born withdrawn rather than resurrecting the order.
  • Sender precedence disagreements. If two authorities each believe they outrank the other, the authoritative copy flips as messages arrive. Agree precedence in advance and treat an unknown sender as lowest rather than defaulting it into the middle.
  • An update that changes the event type. A wildfire evacuation updated to a flood evacuation is arguably a new decision. Treat an event-type change as a new decision and reference the old one, rather than mutating a decision’s meaning in place.

Frequently Asked Questions

Why does deduplicating CAP alerts by identifier not work? Because every relay is entitled to assign its own identifier and sender. A county issues an alert, a state operations centre republishes it under a new identifier, and a federal aggregator republishes that under another, so one decision arrives three times with nothing in common between the identifiers. Sender plus identifier is the correct unique key for a message, but it says nothing about whether two messages describe the same decision — which is the question a consumer aggregating several feeds actually needs answered.

What actually goes wrong when relayed copies are not linked? The update. When the county extends an evacuation boundary and issues an update, its references field names only its own original identifier, which the relayed copies were never issued under. Those copies are therefore not superseded, and a consumer holding one continues presenting the old boundary as current alongside the new one. The visible failure is two active evacuation orders for the same neighbourhood with different instructions, one of which was withdrawn twenty minutes earlier.

Why keep the duplicate copies instead of discarding them? Because supersession has to be applied to them later. A consumer that dedupes by keeping the newest message and discarding the rest has nothing to withdraw when a cancel or update arrives — the copies that most need withdrawing are precisely the ones it threw away. Collapsing into a decision record that lists every member, names the authoritative copy by sender precedence, and records which key merged each one keeps the supersession applicable and makes any merge made by the fuzzy fallback reviewable.

Up: Public Alerting & CAP Message Pipelines

Other guides in Public Alerting & CAP Message Pipelines

Related
Geo-Targeting Wireless Alerts Without Overshoot
→