Backfilling Missing Lineage on Legacy Spatial Archives

A wildfire jumps a containment line at 02:00 and the incident’s geospatial lead pulls the county’s historical fuels and parcel archive off a decade-old network share to seed the initial map. The archive is thousands of shapefiles and GeoTIFFs in nested folders named after long-departed staff, and almost none carry metadata — no source authority, no stated coordinate reference system, no acquisition date. One parcels.shp could be last year’s assessor extract or a fifteen-year-old snapshot; a burn_severity.tif could be from this fire’s predecessor or an unrelated event three counties over. The analyst has to decide, in minutes, which layers are safe to put in front of an incident commander, and the datasets themselves refuse to say where they came from. This page solves that one narrow, dangerous problem: reconstructing best-effort provenance for undocumented legacy spatial archives, stamping it as machine-readable lineage, and — critically — never letting a reconstructed guess masquerade as a validated fact.

Root Cause and Operational Impact

Missing lineage is rarely negligence; it is the accumulated residue of tools and eras that never enforced metadata. Field exports from older desktop software wrote geometry and attributes but no provenance. Shapefiles lost their sidecars during zip-and-email transfers. Rasters were clipped, reprojected, and re-saved through pipelines that discarded whatever documentation the source once had. The International Organization for Standardization published ISO 19115 (geographic information — metadata) precisely to standardize a LI_Lineage element describing source and process history, but a standard only helps data created under it. A legacy archive predates the discipline, so the lineage element is simply absent.

The operational danger is that undocumented data still looks usable. A layer renders, so it gets trusted. But without provenance an analyst cannot answer the three questions that decide whether a layer is safe: who is the authoritative source, what coordinate reference system are the coordinates in, and when were they acquired. Get the source wrong and you cite a non-authoritative parcel boundary in a legal evacuation order. Get the CRS wrong and every feature shifts — a datum mismatch quietly offsets a fire perimeter by tens of metres, the same class of error the coordinate reference system standard for disaster zones exists to prevent. Get the acquisition time wrong and a stale layer contradicts the live situation. The National Incident Management System (NIMS) and the Federal Emergency Management Agency (FEMA) both expect operational data to be reconstructable for after-action review; a layer that reaches the Common Operating Picture with unknown provenance is an accountability gap. The reconstruction has to be auditable, which is why backfilled lineage belongs inside the discipline described in Emergency Metadata Standards rather than typed into a spreadsheet during the incident.

Reconstructing lineage from archival evidence with per-field confidence and an audit trail Four evidence sources — filename tokens, sidecar files such as a projection or readme file, the folder path, and filesystem timestamps — feed a central inference engine. The engine maps that evidence to three lineage fields: source authority, coordinate reference system, and acquisition time. Each output field carries a confidence badge and is explicitly marked as inferred rather than asserted. Every inferred field, together with its evidence citation, confidence score, and the ruleset version, is written to an immutable audit record so the reconstruction is reproducible and reversible. Evidence sources Filename tokens CALFIRE_NAD83_2019_perimeter Sidecar files .prj · .cpg · readme.txt Folder path /archive/assessor/2019/ Filesystem timestamps mtime bounds acquisition Inference engine scores every field Inferred lineage Source authority confidence 0.9 · inferred Coordinate reference system confidence 0.6 · inferred Acquisition time confidence 0.4 · inferred Immutable audit record field · evidence · confidence · ruleset version · timestamp reproducible and reversible

Tiered Resolution Strategy

Reconstruct lineage in ordered tiers, from strong direct evidence down to a safe default that is always flagged for review. The governing rule: never overwrite a validated field, and never let an inferred value lose its confidence badge.

  1. Adopt authoritative evidence where it exists (definitive). A sidecar projection file states the CRS directly; a readme names the source agency. When evidence is explicit and machine-readable, adopt it at high confidence but still record it as recovered, not asserted.
  2. Infer from structured filename and path tokens. Filenames routinely encode an agency code, a datum, and a year (CALFIRE_NAD83_2019_...). Parse tokens against a controlled vocabulary of known authorities and datums, and score confidence by how unambiguous the match is.
  3. Bound the unknown with filesystem timestamps. When no date token exists, the file modification time gives an upper bound on acquisition and a lower bound only if you trust the archive was never re-saved. Record it as a bounded estimate at low confidence, never as a precise date.
  4. Fall back to an explicit “unknown, flagged” default (safe default). If a field cannot be inferred at all, stamp it as unknown with confidence 0.0 and a review flag rather than leaving it null. A null reads as “not checked”; an explicit flagged unknown reads as “checked, unrecoverable” and routes the dataset to a human.
  5. Emit an audit record for every inferred field. Evidence citation, confidence, the ruleset version, and a timestamp — so any reconstruction is reproducible against the exact rules that produced it and any wrong guess is traceable and reversible.

Production Python Implementation

The routine below carries the full resolution path for a single dataset: evidence harvesting from filename, sidecar, path, and timestamp; per-field inference with confidence scoring; an ISO 19115-shaped lineage record where every backfilled field is explicitly marked inferred; structured logging; explicit exception handling; and an immutable audit entry per inferred field. Thresholds and vocabularies are parameters, not literals, so they can be committed and versioned alongside the archive itself — the reproducibility contract that Version Control for Spatial Workflows provides. Senior-engineer assumptions apply: pyproj is available to normalize a recovered CRS to an authority code, and the caller supplies the controlled vocabularies of agency codes and datum aliases.

python 
from __future__ import annotations

import logging
import re
from dataclasses import dataclass, field, asdict
from datetime import datetime, timezone
from enum import Enum
from pathlib import Path
from typing import Optional

logger = logging.getLogger("incidentgis.lineage")


class Provenance(str, Enum):
    ASSERTED = "asserted"        # dataset carried validated metadata
    INFERRED = "inferred"        # reconstructed from indirect evidence
    UNKNOWN = "unknown_flagged"  # unrecoverable, routed for human review


@dataclass
class LineageField:
    """A single backfilled lineage value with its evidence and confidence."""
    value: Optional[str]
    provenance: str
    confidence: float            # 0.0 unknown -> 1.0 authoritative
    evidence: str                # human-readable citation of the signal used


@dataclass
class AuditEntry:
    """Immutable record of one inferred field, emitted to the audit trail."""
    dataset: str
    field_name: str
    value: Optional[str]
    provenance: str
    confidence: float
    evidence: str
    ruleset_version: str
    recorded_at: str = field(
        default_factory=lambda: datetime.now(timezone.utc).isoformat()
    )


class LineageBackfiller:
    """Reconstruct best-effort ISO 19115 lineage for an undocumented dataset.

    Every inferred field is scored, marked as inferred, and appended to
    ``audit_log`` so the reconstruction is reproducible and reversible.
    """

    # Filename token like CALFIRE_NAD83_2019_perimeter -> agency, datum, year.
    _TOKEN = re.compile(
        r"(?P<agency>[A-Za-z]+)[_-](?P<datum>[A-Za-z0-9]+)[_-](?P<year>\d{4})"
    )

    def __init__(
        self,
        ruleset_version: str,
        agency_vocab: dict[str, str],
        datum_epsg: dict[str, int],
    ) -> None:
        self.ruleset_version = ruleset_version
        self.agency_vocab = {k.upper(): v for k, v in agency_vocab.items()}
        self.datum_epsg = {k.upper(): v for k, v in datum_epsg.items()}
        self.audit_log: list[AuditEntry] = []

    def _emit(self, dataset: str, name: str, lf: LineageField) -> None:
        """Append an audit entry and log any inferred or unknown field."""
        entry = AuditEntry(
            dataset=dataset,
            field_name=name,
            value=lf.value,
            provenance=lf.provenance,
            confidence=lf.confidence,
            evidence=lf.evidence,
            ruleset_version=self.ruleset_version,
        )
        self.audit_log.append(entry)
        if lf.provenance != Provenance.ASSERTED.value:
            logger.info("lineage_backfilled", extra={"audit": asdict(entry)})

    def _infer_crs(self, path: Path, tokens: Optional[re.Match]) -> LineageField:
        """Prefer an authoritative .prj sidecar; fall back to a datum token."""
        prj = path.with_suffix(".prj")
        try:
            if prj.is_file():
                from pyproj import CRS  # local import keeps the parser optional
                crs = CRS.from_wkt(prj.read_text(encoding="utf-8", errors="replace"))
                epsg = crs.to_epsg()
                if epsg is not None:
                    return LineageField(
                        f"EPSG:{epsg}", Provenance.ASSERTED.value, 0.95,
                        f"sidecar {prj.name}",
                    )
        except (OSError, ValueError) as exc:
            # Unreadable or non-WKT .prj: degrade to token inference, never raise.
            logger.warning("prj_unreadable", extra={"path": str(prj)}, exc_info=exc)
        if tokens is not None:
            epsg = self.datum_epsg.get(tokens.group("datum").upper())
            if epsg is not None:
                return LineageField(
                    f"EPSG:{epsg}", Provenance.INFERRED.value, 0.6,
                    f"filename datum token '{tokens.group('datum')}'",
                )
        return LineageField(None, Provenance.UNKNOWN.value, 0.0, "no CRS evidence")

    def _infer_authority(self, tokens: Optional[re.Match]) -> LineageField:
        if tokens is not None:
            agency = self.agency_vocab.get(tokens.group("agency").upper())
            if agency is not None:
                return LineageField(
                    agency, Provenance.INFERRED.value, 0.9,
                    f"filename agency token '{tokens.group('agency')}'",
                )
        return LineageField(None, Provenance.UNKNOWN.value, 0.0, "no authority evidence")

    def _infer_acquired(self, path: Path, tokens: Optional[re.Match]) -> LineageField:
        if tokens is not None:
            year = tokens.group("year")
            return LineageField(
                f"{year}-01-01", Provenance.INFERRED.value, 0.5,
                f"filename year token '{year}'",
            )
        try:
            mtime = datetime.fromtimestamp(path.stat().st_mtime, tz=timezone.utc)
            # mtime only bounds acquisition from above; treat as a weak estimate.
            return LineageField(
                mtime.date().isoformat(), Provenance.INFERRED.value, 0.3,
                "filesystem mtime upper bound",
            )
        except OSError as exc:
            logger.warning("stat_failed", extra={"path": str(path)}, exc_info=exc)
            return LineageField(None, Provenance.UNKNOWN.value, 0.0, "no time evidence")

    def backfill(self, dataset_path: str) -> dict[str, LineageField]:
        """Return a lineage record keyed by field, with an audit entry each."""
        path = Path(dataset_path)
        try:
            tokens = self._TOKEN.search(path.stem)
            record = {
                "source_authority": self._infer_authority(tokens),
                "crs": self._infer_crs(path, tokens),
                "acquisition_time": self._infer_acquired(path, tokens),
            }
        except Exception as exc:  # never let one bad file stall an archive sweep
            logger.error("backfill_failed", extra={"path": dataset_path}, exc_info=exc)
            record = {
                name: LineageField(None, Provenance.UNKNOWN.value, 0.0, "backfill error")
                for name in ("source_authority", "crs", "acquisition_time")
            }
        for name, lf in record.items():
            self._emit(path.name, name, lf)
        return record

The audit_log is the load-bearing output. Persisting it as a committed, content-hashed artifact means a reviewer can replay every inference, see exactly which filename token or sidecar produced each value, and reverse any guess the ruleset got wrong — the same defensibility the wider Emergency Metadata Standards discipline demands before a layer is trusted.

Validation Checklist

Verify every item before backfilled lineage is written into the archive or surfaced to analysts.

  • Agency vocabulary, datum-to-EPSG map, and confidence thresholds are passed as parameters and committed under version control — no literals hard-coded in the sweep.
  • ruleset_version is set from the running release tag so every audit entry is traceable to a specific commit.
  • Every backfilled field is written with an explicit provenance of inferred or unknown_flagged; a validated field is never relabelled or overwritten.
  • Fields that cannot be recovered are stamped unknown with confidence 0.0 and a review flag, never left null.
  • A .prj sidecar that is unreadable or non-WKT degrades to token inference and logs a warning instead of raising.
  • Recovered CRS values are normalized to an authority code (for example EPSG:4269) rather than stored as raw WKT so downstream tools can consume them.
  • Structured logs route to the metadata audit sink, not stdout, and every inferred field appears in audit_log.
  • Downstream consumers read and honour the confidence field, suppressing or flagging low-confidence lineage rather than treating all fields equally.

Edge Cases and Gotchas

  • Filesystem timestamps are not acquisition dates. A file’s modification time reflects the last write, which for a re-projected or copied archive can be years after the data was collected. Treat mtime as an upper bound at low confidence only, and never let it overwrite a filename year token that is closer to the truth.
  • Ambiguous datum tokens. A token like NAD83 maps to many EPSG codes depending on zone and realization; a bare 83 or 27 is worse. Resolve datum tokens against zone evidence from the folder path or a .prj before assigning an EPSG code, and drop confidence when the mapping is one-to-many. This is the reconstruction feeding the coordinate reference system standard for disaster zones that the rest of the pipeline enforces.
  • Encoding-mangled tokens. Legacy filenames and readme sidecars from older systems can carry non-UTF-8 bytes, so a naive read raises or silently mojibakes an agency name. Read with an explicit fallback encoding and a replacement policy — the same failure mode covered when handling shapefile ingestion from mixed-vintage sources.
  • Collisions between asserted and inferred values. If a dataset later gains real metadata, a merge must let the asserted value win and must retain the prior inferred value in the audit trail rather than deleting it. Overwriting history erases the reason a wrong decision was ever made.
  • Confidence inflation on repeat runs. Re-running the backfiller over an archive that already contains inferred lineage must not treat its own earlier guess as fresh evidence. Key inference off original signals only, and skip fields already stamped asserted so the confidence score never bootstraps itself upward.

Frequently Asked Questions

Is inferred lineage safe to treat as authoritative metadata? No. Backfilled lineage is a best-effort reconstruction, not an assertion of fact, and it must never be silently merged with validated metadata. Every inferred field carries a confidence score and an evidence citation, and the lineage record marks it as inferred so an analyst can weight it, verify it, or reject it. Treating a guess as ground truth is exactly the failure this process is designed to prevent.

What evidence can you recover provenance from when metadata is missing? Legacy datasets still carry provenance signals even without formal metadata: filename tokens often encode an agency code, a datum or coordinate system, and a date; sidecar files such as a projection file or a readme record the CRS and sometimes the source; the folder path reflects an archival organization; and filesystem timestamps bound the acquisition or ingest time. None of these is authoritative alone, but combined and scored they reconstruct defensible lineage.

Why does missing lineage matter for emergency response specifically? An emergency dataset with no lineage cannot be trusted, cited, or defended after the fact. An analyst cannot tell whether a flood layer is from this event or a decade old, whether coordinates are in the expected datum, or which agency is accountable for it. NIMS and FEMA both expect operational data to be reconstructable for after-action review, so a layer that reaches the Common Operating Picture with unknown provenance is an accountability gap, not merely an inconvenience.

Up: Emergency Metadata Standards