ML Assurance and Data Governance
Last updated: 2026-05-24
Why It Matters
Machine-learning assurance is now a governance problem, not only a model-evaluation problem. ISO/IEC 42001:2023 establishes requirements for an AI management system, including continual improvement and organization-wide controls. ISO/IEC 23894:2023 gives guidance for managing AI-specific risk. ISO/IEC TR 5469:2024 describes risk factors and methods for AI used inside safety-related functions, non-AI safety functions protecting AI-controlled equipment, and AI used to develop safety-related functions. ISO/IEC 5259-5:2025 brings data-quality governance into the management layer for analytics and ML.
For airside autonomy, this matters because perception and prediction failures rarely look like clean component faults. They show up as bad labels, missing night/rain data, apron-specific class imbalance, untracked synthetic data, stale calibration, overconfident detections, or a model retrained without scenario coverage. The safety case should therefore prove that the organization can govern AI risk, data quality, model release, and post-deployment monitoring with the same discipline used for mechanical and functional-safety evidence.
Evidence Model
AI management-system scope. Define which AI/ML components are in scope: obstacle detection, aircraft-stand scene understanding, FOD detection, localization, semantic mapping, intent prediction, remote-assist triage, or fleet optimization. Assign accountable owners, risk acceptance authority, release authority, and escalation paths.
AI risk register. Maintain a risk record for each model and dataset. Include intended use, prohibited use, ODD, known limitations, affected hazards, safety monitors, misuse scenarios, security exposure, human-oversight assumptions, and residual risk. Map the register to NIST AI RMF functions: Govern, Map, Measure, and Manage.
Data-quality governance. Keep a governed inventory for raw data, labels, derived features, simulation data, synthetic data, third-party data, and operational logs. The inventory should track source, consent or license basis, sensor configuration, collection site, weather, lighting, airport geometry, actor classes, label ontology, review status, and retention rules.
Dataset lifecycle controls. Separate training, validation, test, regression, and release-gate datasets. Track dataset version hashes, labeler instructions, inter-annotator agreement, adjudication results, rejected data, known gaps, and leakage checks. Treat ODD slices as explicit coverage units: aircraft stand, service road, gate pushback zone, baggage make-up area, rain, glare, cones, reflective vests, dollies, belt loaders, aircraft shadows, and jetblast/FOD zones.
Model assurance case. For each model release, include architecture, training code version, data version, hyperparameters, calibration, confidence behavior, slice metrics, stress tests, OOD tests, ablation results, robustness tests, and comparison against the prior release. Link each metric to an operational threshold or safety monitor action.
Runtime governance. Monitor drift, confidence distribution, false-positive and false-negative cases, operator interventions, disengagements, near misses, and safety monitor activations. Define retraining triggers and emergency rollback triggers before deployment.
Assurance Controls by MLOps Scale
The MLOps scale ladder in ../../50-cloud-fleet/mlops/mlops-scale-research-scope.md is also an assurance ladder. The evidence threshold is determined by how the model is used, not by team size.
| MLOps scale | Assurance posture | Required governance evidence |
|---|---|---|
| S0 notebook research | Research-only, no safety claim | Run notes, data pointer, limitation statement |
| S1 repeatable prototype | Comparable baseline, no operational authority | Dataset snapshot, validation script, known-gap list |
| S2 production product | Controlled release to one operational workflow | Model card/release note, registry record, dataset manifest, rollback instruction |
| S3 fleet and multi-site | ODD-scoped release with local evidence | Site slices, local holdouts, drift monitors, fleet incident intake, active-learning disposition |
| S4 regulated safety-critical | Safety-case-backed release | Risk register, claim/evidence traceability, approver record, scenario replay, incident retention, rollback drill |
| S5 platform scale | Organization-wide AI management system | Policy-as-code, audit trails, model/data inventory, retention controls, role-based approval, periodic management review |
Evaluation Evidence by MLOps Scale
Assurance evidence should explain why a model is acceptable for its intended authority level, not merely that a metric improved. The same release should trace from data sources to training run, evaluation suite, runtime artifact, monitoring trigger, and rollback target.
| MLOps scale | Dataset independence | Evaluation evidence | Governance control |
|---|---|---|---|
| S0 notebook research | Best-effort split hygiene | Run notes, sampled predictions, known limitations | Mark output as research-only |
| S1 repeatable prototype | Frozen validation snapshot separate from training | Re-runnable metric script, baseline comparison, confidence interval | Require comparable baseline before adoption |
| S2 production product | Independent release holdout and regression clips | Slice metrics, calibration/OOD checks, model card, package compatibility test | Release owner signs model/data/runtime packet |
| S3 fleet and multi-site | Local holdouts by site, route, weather, map release state, and vehicle configuration | Replay suite, shadow report, canary report, delayed-label QA, drift review | Approve or block per ODD cell, not globally |
| S4 regulated safety-critical | Safety-case release gates protected from training influence | Hazard-class tests, stress tests, monitor impact analysis, rollback drill, waiver expiry | Named risk authority accepts residual risk |
| S5 platform scale | Organization-wide inventory with retention and access policy | Automated evidence capture, standardized scorecards, audit trails, policy-as-code checks | AI management system reviews controls periodically |
Evidence should expire. A release packet that passed before a taxonomy migration, sensor calibration change, map publication, runtime upgrade, or airport-layout change should be re-evaluated or explicitly scoped out of the affected ODD cells.
For airside autonomy, S4 controls can be required before S3 fleet size exists. A model that influences aircraft proximity, free-space, FOD response, semantic map publication, or protected-zone behavior needs release evidence, monitoring, and rollback authority even if it runs at one airport on a small fleet.
Acceptance Checks
- AI/ML systems have named owners, approved intended uses, prohibited uses, and risk acceptance records.
- Every production model has a model card or equivalent release note with dataset versions, ODD scope, limitations, metrics by scenario slice, and rollback instructions.
- Data used for release gates is independent from training data and protected from uncontrolled edits.
- Label taxonomy covers airside-specific objects and states, including people in high-visibility clothing, aircraft servicing equipment, tow bars, cones, chocks, carts, belt loaders, fuel trucks, FOD, and temporary works.
- Performance thresholds are defined per safety-relevant slice, not only as aggregate precision/recall or mean average precision.
- Synthetic and simulated data are tagged, justified, and validated against real airside data before they support a safety claim.
- Model updates trigger impact analysis for safety monitors, ODD limits, datasets, calibration, and downstream planning behavior.
- Post-deployment monitoring has clear owners, review cadence, incident intake, triage severity, corrective actions, and field rollback authority.
Failure Modes
- A model passes aggregate metrics while failing at dusk, in rain, near aircraft shadows, or around reflective airport markings.
- Training and validation sets share near-duplicate frames from the same route run, hiding leakage and overstating performance.
- A label ontology changes but old labels remain in the dataset without migration or review.
- Synthetic rare-event data dominates the validation claim without correlation to real sensor artifacts.
- Model confidence is treated as calibrated when no calibration or OOD evidence exists.
- Data rights, retention, or provenance are unclear, making the evidence unusable in an audit.
- A retrain improves one class but degrades pedestrian, cone, or GSE detection in a safety-critical slice.
- AI output silently becomes part of the safety function even though the certified safety layer assumes it is advisory only.
Related Repository Docs
- Knowledge Base Evaluation Protocol
- Perception SLAM Statistical Validity Protocol
- Uncertainty Calibration Perception SLAM Release Gates
- Online Perception Monitoring ODD Enforcement
- Formal Methods Regulatory
- Safety Case Evidence Traceability
Sources
- ISO, ISO/IEC 42001:2023 - AI management systems
- ISO, ISO/IEC 23894:2023 - AI guidance on risk management
- ISO, ISO/IEC TR 5469:2024 - Functional safety and AI systems
- ISO, ISO/IEC 5259-5:2025 - Data quality governance framework
- NIST, AI Risk Management Framework
- NIST AIRC, AI RMF Core
- NIST, AI RMF Playbook