Version Control for Spatial Workflows

A flood-response team pushes an updated inundation layer at 14:00; by 18:00 a downstream evacuation map disagrees with it by two city blocks, and nobody can say which raster band, which reprojection step, or which analyst’s manual edit introduced the shift. The after-action review stalls because the workspace has no lineage: the GeoTIFF was overwritten in place, the reprojection ran from an un-pinned pyproj, and the script that produced it was edited three times without a commit. Spatial version control exists to make that failure impossible — to guarantee that every incident map is traceable to the exact code, dependencies, and inputs that produced it, and that any past state can be reconstructed on demand. It is the audit-and-reproducibility arm of the broader Python Toolchains for Public Safety GIS, and it carries the same chain-of-custody obligations every node in an incident must satisfy.

Spelled out once for this page: NIMS is the National Incident Management System, ICS is its Incident Command System, FEMA is the Federal Emergency Management Agency, OGC is the Open Geospatial Consortium, and ISO 22320 is the international standard for emergency-management operations.

Prerequisites

This pattern versions logic, data, and the transforms between them; it assumes the contracts below are already established upstream.

• Python packages: dvc >= 3.0 for data versioning and pipeline declaration, geopandas >= 0.14, shapely >= 2.0, and pyproj >= 3.6 for the spatial transforms, and pytest >= 7.4 for the validation gate. The standard-library logging, pathlib, hashlib, and datetime modules carry the audit trail and deterministic output pathing.
• A pinned spatial runtime. GDAL, PROJ, and the spatial Python bindings must be version-locked inside a reproducible image — the contract established when setting up Dockerized GIS environments. Without it, the same commit reprojects differently on two machines and the “reproducible” history is a fiction.
• A declared CRS contract. Field collection arrives in EPSG:4326 (WGS 84) and is normalized to a projected UTM zone (EPSG:326xx / 327xx) before any distance, area, or buffer is computed, consistent with the Coordinate Reference System standard for disaster zones. Every pyproj transform runs with always_xy=True so a lat/lon device never inverts axis order.
• A schema contract. Attribute fields, types, and order are fixed upstream so written GeoPackage layers match what consuming systems expect, and schema drift is caught at ingest rather than at merge.

Architecture: Two Tracks, One Lineage

Conventional Git chokes on large binary payloads and has no concept of geometry, so production incident GIS runs a two-track model over a single commit graph. Git owns the lightweight, diff-friendly artifacts — Python scripts, configuration manifests, dvc.yaml, and the .dvc pointer files. DVC owns the heavy spatial binaries — GeoPackage, GeoTIFF, and LiDAR point clouds — storing each as a content-hashed object in a cache or shared remote and committing only the pointer to Git. The two tracks are stitched together at every commit: a single SHA resolves both the transform logic and the exact data hashes it consumed, so the workspace is auditable, rollback-capable, and aligned with NIMS/ICS documentation expectations.

A reproducible incident workspace follows a fixed shape:

1. Repository structure: scripts/, configs/, data/raw/, data/processed/, tests/, dvc.yaml. Only scripts/, configs/, tests/, and the .dvc pointers are tracked by Git; the data/ payloads are tracked by DVC.
2. Branching model: main (production-ready common operating picture), incident/<id> (active response), hotfix/<id> (critical topology corrections that must merge fast and traceably).
3. Data tracking: DVC tracks large binaries via .dvc metadata; Git tracks only the lightweight pointers and the transformation logic that produced them.

Step-by-Step Implementation

1. Version the ingestion transform

The first thing to put under version control is the step that turns a raw field export into a workspace-ready layer. Field crews deploy tablets, GNSS receivers, and drone payloads that produce heterogeneous vector exports, and without strict ingestion controls, schema drift, CRS mismatches, and attribute-normalization failures corrupt incident basemaps silently. Committing this function alongside the data pointer it produces means every coordinate adjustment is traceable to a specific code release — the same lightweight-shapefile discipline established in Geopandas vs PyShp for Field Operations.

import logging
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

import geopandas as gpd
from pyproj import CRS

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
)
logger = logging.getLogger("vcs.ingest")


class SpatialIngestionError(Exception):
    """Raised when a field export fails schema or topology validation."""


def validate_and_version_field_export(
    raw_path: Path,
    output_dir: Path,
    expected_crs: str = "EPSG:4326",
    required_fields: tuple[str, ...] = ("incident_id", "timestamp", "geometry_type"),
) -> dict[str, Any]:
    """Normalize a raw field export into a versioned GeoPackage layer.

    Returns a manifest dict suitable for emission into the audit trail.
    Raises SpatialIngestionError on a contract violation so the caller can
    quarantine the payload instead of committing a corrupt layer.
    """
    if not raw_path.exists():
        raise SpatialIngestionError(f"Raw field export missing: {raw_path}")

    gdf = gpd.read_file(raw_path)

    # Schema contract: required attributes must be present before merge.
    missing = [field for field in required_fields if field not in gdf.columns]
    if missing:
        raise SpatialIngestionError(f"Missing required attributes: {missing}")

    # CRS contract: enforce the declared frame, log every reprojection.
    target_crs = CRS.from_user_input(expected_crs)
    if gdf.crs != target_crs:
        logger.info("Reprojecting from %s to %s", gdf.crs, target_crs)
        gdf = gdf.to_crs(target_crs)

    # Topology guard: repair invalid geometry deterministically, on the record.
    invalid_count = int((~gdf.is_valid).sum())
    if invalid_count:
        logger.warning("Repairing %d invalid geometries via buffer(0)", invalid_count)
        gdf.geometry = gdf.geometry.buffer(0)

    # Deterministic output pathing so the commit references a stable artifact.
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    out_path = output_dir / f"field_export_{stamp}.gpkg"
    gdf.to_file(out_path, driver="GPKG", layer="incident_assets")

    logger.info("Wrote %d features to %s", len(gdf), out_path)
    return {
        "status": "success",
        "output_path": str(out_path),
        "feature_count": len(gdf),
        "crs": gdf.crs.to_epsg(),
        "repaired_geometries": invalid_count,
    }

Coordinate drift remains a persistent challenge when integrating multi-source GNSS data. When urban-canyon multipath degrades positional accuracy, the correction routine must be applied consistently across every incident layer and itself versioned — the methodology documented in Handling GPS Drift in Urban Canyon Environments should ship as a versioned transformation module so each adjustment maps to a known calibration release.

2. Declare the pipeline in dvc.yaml

Once the transform is committed, the build graph that connects raw inputs to processed outputs becomes a versioned artifact too. A DVC pipeline declares each stage’s dependencies and outputs, so dvc repro rebuilds only what changed and records a content hash for every artifact it produces. The computation itself stays plain Python — DVC tracks inputs and outputs around it.

import hashlib
import logging
from pathlib import Path

import pandas as pd

logger = logging.getLogger("vcs.etl")


def process_sensor_stream(raw_csv: Path, calibration_offset: float = 0.0) -> Path:
    """Versioned ETL stage for incident telemetry aggregation.

    Invoked by a dvc.yaml stage; DVC tracks deps/outs via .dvc metadata.
    The calibration_offset is versioned alongside the data pointer so a
    historical sensor overlay can be reconstructed deterministically.
    """
    try:
        df = pd.read_csv(raw_csv)
    except pd.errors.EmptyDataError as exc:
        raise RuntimeError("Empty telemetry payload; verify sensor connectivity.") from exc

    required_cols = {"timestamp", "lat", "lon", "value"}
    missing = required_cols - set(df.columns)
    if missing:
        raise ValueError(f"Telemetry schema mismatch. Missing: {missing}")

    # Apply the versioned calibration offset and resample to a 5-minute grid.
    df["value"] = df["value"] + calibration_offset
    df["timestamp"] = pd.to_datetime(df["timestamp"], utc=True)
    df = df.set_index("timestamp").resample("5min").mean().dropna()

    # Deterministic output path keyed by inputs so DVC can cache the result.
    digest = hashlib.sha256(
        f"{raw_csv.stem}_{calibration_offset}_{len(df)}".encode()
    ).hexdigest()[:8]
    out_path = Path(f"data/processed/telemetry_agg_{digest}.parquet")
    df.to_parquet(out_path)

    logger.info("Aggregated %d rows → %s", len(df), out_path)
    return out_path

By versioning the calibration_offset alongside the raw-data pointer, an incident commander can reconstruct any historical sensor overlay deterministically: the pipeline state is auditable, and a rollback is dvc checkout <commit> followed by dvc repro.

3. Gate merges on topology validation

Geospatial scripts must not merge into a production incident branch until their geometry has been proven valid. The validation pass below enforces topology rules, CRS presence, and sliver detection, and is wired into both a pre-commit hook on the workstation and the CI gate on the merge — so a malformed layer is rejected before it can feed an operational dashboard.

import logging
from typing import Iterable

import geopandas as gpd
import pytest
from geopandas import GeoDataFrame
from shapely.validation import explain_validity

logger = logging.getLogger("vcs.validate")


def validate_topology(gdf: GeoDataFrame, min_area_threshold: float = 1e-6) -> list[str]:
    """Return a list of topology violations for CI/CD gating (empty == pass)."""
    violations: list[str] = []

    if gdf.crs is None:
        violations.append("Dataset missing CRS definition")

    for idx, geom in gdf.geometry.items():
        if geom is None or geom.is_empty:
            violations.append(f"Row {idx}: null or empty geometry")
            continue
        if not geom.is_valid:
            violations.append(f"Row {idx}: invalid geometry — {explain_validity(geom)}")
        elif geom.area and geom.area < min_area_threshold:
            violations.append(f"Row {idx}: sliver polygon (area < {min_area_threshold})")

    if violations:
        logger.error("Topology gate found %d violation(s)", len(violations))
    return violations


@pytest.fixture
def mock_incident_gdf() -> GeoDataFrame:
    return gpd.GeoDataFrame(
        {"id": [1, 2], "type": ["evac_zone", "hazard"]},
        geometry=gpd.points_from_xy([-122.4, -122.3], [37.8, 37.7]),
        crs="EPSG:4326",
    )


def test_incident_topology(mock_incident_gdf: GeoDataFrame) -> None:
    issues = validate_topology(mock_incident_gdf)
    assert not issues, f"Topology validation failed: {issues}"

Wiring validate_topology into a pre-commit configuration rejects malformed spatial payloads at the developer workstation, reducing the response latency that data corruption would otherwise inject mid-incident.

Configuration Reference

Verification and Smoke Test

Confirm the workspace reproduces deterministically before relying on it in the field. The following sequence rebuilds tracked data, runs the topology gate, and proves a clean working tree.

# Reproduce the pipeline from versioned inputs and pointers.
dvc repro

# Run the topology + CRS gate exactly as CI does.
pytest tests/test_topology.py -q

# Prove nothing drifted: both Git and DVC report a clean tree.
git status --porcelain
dvc status

A passing smoke test means dvc status reports Data and pipelines are up to date, git status --porcelain is empty, and the pytest gate exits zero. To confirm a historical state reproduces, check out a past commit, run dvc checkout to materialize its pointers, and re-run the pipeline inside the pinned container — the regenerated artifact’s hash must match the one recorded at that commit.

Integration With Adjacent Workflows

Version control is the connective tissue between the other toolchain concerns rather than a standalone step. The pinned runtime it depends on is built when setting up Dockerized GIS environments; without an immutable GDAL/PROJ image the “reproducible” history cannot be trusted. The ingestion transform it versions is the same boundary contract enforced across Geospatial Data Ingestion Pipelines, and the CRS normalization it commits is governed by the Coordinate Reference System standard for disaster zones. The library-selection logic that decides whether a stage runs through Geopandas or PyShp is documented in Geopandas vs PyShp for Field Operations, and every coordinate correction this workspace records draws on the edge-case handling in Handling GPS Drift in Urban Canyon Environments.

Troubleshooting

Symptom: dvc pull returns “missing data source” on a field node. The pointer is committed but the binary never reached the shared remote. The author ran git push without dvc push, so the cache object exists only on their workstation. Run dvc push from the node that produced the data, then dvc pull on the field node. Add a CI check that fails the merge if dvc status --cloud reports anything pending.

Symptom: the same commit produces a geometry that differs by metres on two machines. The runtime is not pinned — the two hosts resolved different PROJ pipelines or GDAL builds. Rebuild both from the immutable image, and verify with pyproj.show_versions() that the PROJ data directory matches before trusting any reprojection.

Symptom: the topology gate passes locally but fails in CI. The pre-commit hook is stale or was skipped with --no-verify. Reinstall hooks with pre-commit install, then run pre-commit run --all-files to surface the same violations CI sees. Never bypass the gate to merge a hotfix/<id> faster — an unvalidated correction is the failure mode this workflow exists to prevent.

Symptom: a Git clone takes hours or runs out of disk in the field. A binary was committed to Git directly instead of being tracked by DVC, bloating the pack history. Identify the offending blob, remove it from history, and re-add it with dvc add so only the pointer remains in Git.

Symptom: dvc checkout of an old commit restores stale data into the working tree but the pipeline won’t reproduce it. The dvc.lock was not committed with the code, so DVC has no record of which output hash that commit expected. Always commit dvc.lock alongside dvc.yaml; it is the lineage link that makes a historical rebuild deterministic.

Related

• Setting Up Dockerized GIS Environments — the pinned GDAL/PROJ runtime that makes versioned history reproducible.
• Geopandas vs PyShp for Field Operations — choosing the spatial library each versioned stage runs through.
• Python ETL for Sensor & IoT Data — the real-time ingestion arm whose pipeline stages get versioned here.
• Handling GPS Drift in Urban Canyon Environments — a versioned coordinate-correction module under this workflow.

Up: Python Toolchains for Public Safety GIS