Skip to content

MLOps Reference Architectures by Scale

Last updated: 2026-05-24

This page turns the MLOps scale ladder into concrete architecture choices. Use it after mlops-scale-research-scope.md has identified the required maturity level, and before buying/building tooling. The core decision is not "which MLOps platform is best?" It is which artifacts, interfaces, owners, and promotion gates must be centralized at the current scale.

The architecture should grow by contract first, platform second. A small team can run on scripts, manifests, and a lightweight tracker if the artifact boundaries are disciplined. A large platform still fails if it centralizes dashboards while leaving labels, dataset snapshots, registry aliases, evaluation packs, and rollback evidence ambiguous.

Pair each architecture with the scorecard in mlops-scorecards-and-kpis-by-scale.md. The architecture defines where artifacts and decisions live; the scorecard defines whether those artifacts are reproducible, release-eligible, observable, rollback-ready, and cost-controlled at the current scale. Use ../data-platform/data-catalog-lineage-quality-ops.md to decide whether a data product remains a file manifest, moves into a snapshot-capable lakehouse table, requires a metadata catalog, emits OpenLineage events, or needs quality SLOs and deletion-propagation controls. Use experiment-tracking-reproducibility-by-scale.md to decide which runs need tracker-only notes, baseline authority, candidate output digests, release evidence, or evidence-locked audit records. Use model-registry-artifact-lifecycle-by-scale.md to decide which artifacts need immutable registry identity, alias authority, lifecycle states, artifact-set membership, rollback retention, and S5 registry service boundaries. Use serving-inference-operations-by-scale.md to decide where batch inference, online endpoints, edge runtime packages, serving manifests, traffic routing, autoscaling, shadow/canary, and rollback live in the architecture. Use platform-sre-reliability-by-scale.md to decide which platform services need criticality tiers, SLOs, error budgets, backup/restore, DR, incident lanes, tenant isolation, and bypass controls. Use pipeline-orchestration-release-workflows-by-scale.md to decide which workflow states belong in scripts, DVC, GitHub Actions, Airflow, Argo, Kubeflow, TFX, Ray, Slurm, or managed platforms, and which transitions must remain explicit release or safety gates. Use evaluation-platform-replay-gates-by-scale.md to decide when evaluation remains a local script, when replay/runtime/shadow evidence needs a production lane, and when S5 needs a shared evaluation service with scenario catalogs, policy templates, and service SLOs. Use mlops-migration-checklist-by-scale.md before moving an artifact family from one scale to the next. Pair monitoring decisions with model-monitoring-drift-response-by-scale.md so drift, delayed-label, replay, and incident signals produce controlled actions instead of automatic retraining or dashboard noise. Pair feature and vector-search decisions with feature-embedding-store-ops-by-scale.md so S5 tooling is not introduced before S1-S2 data contracts exist, and pair release-affecting artifacts with secure-artifact-attestation-profile.md so signatures, SBOMs, provenance, and alias policy are added at the right authority level.


Scale Blueprints

ScaleArchitecture shapeLocal componentsShared componentsPromotion authorityMain migration risk
S0 notebook researchSingle-owner experiment loopNotebook/script, local data pointer, config file, result folderGit repository and raw-data locationResearch owner decides whether a result is worth preservingResult becomes tribal knowledge with no rebuild path
S1 repeatable prototypeReproducible baseline laneDocker image, deterministic config, DVC/object snapshot, validation scriptExperiment tracker, baseline metric table, frozen split manifestResearch lead accepts a baselineDemo data leaks into release evaluation later
S2 single-product productionProduct release laneTraining DAG, export script, offline eval, package smoke testModel registry, dataset manifest, release packet, rollback artifactModel/data/runtime/release owners move candidate aliasesTraining completion is mistaken for release approval
S3 fleet and multi-siteSite-sliced fleet flywheelSite holdouts, local trigger queues, edge data triage, site canariesData catalog, active-learning queue, fleet telemetry, release channelsODD-cell release owner expands rolloutOne global champion hides local airport, yard, or warehouse regressions
S4 regulated safety-criticalEvidence-controlled change systemHazard replay packs, safety monitor traces, incident evidence, rollback drillImmutable evidence store, safety-case links, approver records, retention holdSafety owner and release manager approve behavior authorityEvidence exists in tickets but not in immutable release artifacts
S5 platform scaleMulti-tenant ML platformProduct-specific model code, task-specific evals, site adaptersData/model/eval registry, feature/embedding service, GPU scheduler, policy-as-code, cost/SLO dashboardsPlatform policy plus product release authorityPlatform becomes bypassed because it slows product teams or misses autonomy-specific metadata

The same fleet can occupy multiple scales at once. A perception research branch may be S1, the deployed detector may be S3, the semantic-map label export may need S4 evidence, and a foundation-model labeler platform may be S5. Architecture decisions should follow the artifact's authority, not the team org chart.


Minimum Viable Stack

CapabilityS0-S1 minimumS2-S3 minimumS4-S5 minimum
Dataset identityFile manifest, source path, split hashImmutable dataset snapshot, data product ID, approved-use state, quality policyCataloged lineage graph with retention, access class, evidence lock, deletion propagation, and quality SLO
Experiment trackingRun note plus commit/config and authority stateTracker run linked to dataset, split, code, seed, hardware, metric report, output digest, and registry candidateOrganization eval warehouse with lineage graph, audit export, and policy templates
Label operationsManual labels and instructionsVersioned annotation workflow, QA sampling, allowed-use statePolicy-enforced promotion states, expert review, vendor/privacy controls
OrchestrationScript, Makefile, DVC, or CI jobAirflow/Argo/Kubeflow/Ray/managed pipeline for data-train-eval-export-register with explicit release workflow gatesMulti-tenant pipeline with quotas, lineage, policy checks, evidence capture, incident lanes, and audit export
Model and artifact registryCheckpoint path plus release noteImmutable model/package/map/prompt/eval versions, aliases, approval metadata, rollback target, artifact digestRegistry integrated with policy, signing, SBOM, SLSA/in-toto provenance, tenant isolation, artifact-set compatibility, and audit logs
EvaluationValidation script, frozen split, and metric specEvaluation manifest, slice metrics, replay, calibration, runtime smoke, shadow/canary reportShared eval service, safety-case-linked claim/evidence table, scenario catalog, waiver expiry, platform SLOs
Deployment and servingManual batch or offline artifactONNX/TensorRT/container bundle, serving manifest, compatibility manifest, canary channelOTA/SUMS integration, policy-gated serving platform, rollback drill, reportability evidence
MonitoringManual plots and failure notesRuntime health, drift proxies, delayed labels, incident hooks, monitoring event contractFleet SLOs, causal attribution, alert quality, suppression audit, compliance export, platform drift-response policy
Platform reliabilityLocal backup and owner noteService inventory, SLOs for registry/eval/serving, backup/restore smoke, incident laneCriticality tiers, error budgets, DR, immutable audit logs, tenant isolation, platform SRE
Cost and capacityPer-run noteJob owner, GPU queue, timeout, cost tagFinOps allocation, reserved incident capacity, utilization SLO, chargeback

Minimum viable does not mean minimal governance. At S2+, the smallest acceptable production stack is one that can answer: what changed, what data justified it, where is it allowed to run, how will we know it failed, and how do we roll back?


Centralize, Decentralize, or Delay

Decision areaCentralize earlyKeep decentralizedDelay until trigger
Schemas and contractsLabel taxonomy, telemetry schema, dataset manifest fields, model package manifest, map/model/calibration compatibility IDsTask-specific feature engineering and local experiment configsFull schema registry UI until multiple producers/consumers exist
Data storageImmutable raw-data pointer, snapshot identity, retention class, and allowed-use stateSmall research samples and temporary scratch dataFull lakehouse/catalog until release-relevant data products multiply
Data quality and lineageRelease-affecting data-product contract fields, lineage IDs, and quality severity taxonomyExploratory profiling and notebook diagnosticsOrganization catalog, lineage graph, and data-quality platform until several promoted data products have shared consumers
EvaluationRelease metrics, safety slices, scenario IDs, replay package formatExploratory notebooks and model-debug visualizationsShared eval service until teams repeatedly duplicate evaluation logic
Model registryImmutable model ID, alias semantics, rollback target, approval metadataResearch checkpoints not reused outside a branchMulti-tenant registry automation until several product lines share releases
ComputeContainer base images and hardware class tagsLocal prototyping and one-off sweepsGPU scheduler/chargeback until contention or cost becomes a real bottleneck
LabelingTaxonomy, instructions, QA states, accepted/rejected provenanceLocal candidate discovery and failure analysisVendor marketplace and workflow engine until annotation volume requires it
MonitoringCommon model/map/calibration/runtime identifiers in telemetryProduct-specific dashboards and debug panelsCentral alert platform until incidents cross teams or sites
GovernanceOwner fields, evidence IDs, release decision recordsReview meeting format and local triage ritualsPolicy-as-code until manual enforcement becomes the bottleneck

The practical pattern is "centralized identifiers, decentralized iteration." Let teams experiment locally, but force every artifact that can influence release to cross the same typed interfaces.


Interfaces That Must Survive Scale

These interfaces should exist before the platform becomes large. They can begin as JSON or Markdown manifests and later move into services.

InterfaceRequired fieldsConsumed by
Dataset manifestdataset ID, raw sources, split ID, label schema, release-state label schema, access class, retention class, quality reporttraining, evaluation, privacy, safety case
Pipeline run manifestworkflow version, trigger, parameters, owner, input artifact IDs, output artifact IDs, state, retry/failure summaryexperiment tracking, data catalog, registry, audit
Training run manifestrun authority, reproducibility level, code commit, dependency lock, config, seed policy, hardware class, dataset snapshot, split/leakage report, augmentation policy, metric spec, output model hash, registry/release linksmodel registry, experiment tracking, reproducibility review, release packet, audit export
Registry artifact recordartifact ID, type, version, digest, authority state, aliases, producer workflow, inputs, evaluation IDs, compatibility hash, security attestations, retention classdeployment, release review, rollback, safety case, incident response
Model package manifestmodel version, ONNX/TensorRT/container hashes, class order, calibration dependencies, map/schema compatibility, hardware targetruntime deployment, OTA/SUMS, rollback
Inference service manifestservice ID, artifact set, registry alias, input/output contract, runtime target, traffic policy, scaling policy, observability, safety policy, security evidenceserving platform, deployment review, monitoring, incident response
Compatibility manifestmodel, map, calibration, runtime, telemetry, semantic taxonomy, labeler/prompt/evaluator dependencies, MLOps scale, rollback setOTA/SUMS, release review, safety case, incident response
Evaluation manifest/reporteval ID, authority state, evaluator version, artifact-set hash, metric version, aggregate and slice metrics, confidence intervals, replay package IDs, runtime smoke, known failures, waiver staterelease review, safety validation, monitoring thresholds, platform audit
Deployment manifestmodel alias, runtime container, map bundle, calibration package, vehicle/site/ODD scope, rollout cohort, rollback artifactfleet deployment, operations, incident response
Monitoring event schemamodel/map/calibration/runtime IDs, site, route, ODD cell, input quality, output quality, latency, intervention/disagreement fields, action state, owner, suppression expiryfleet observability, active learning, incident triage, drift-response policy
Incident evidence linkevent ID, active artifacts, logs/clips, replay scenario, safety monitor state, containment action, corrective actiongovernance, safety case, post-release learning
Foundation-model artifact manifestprompt/model/checkpoint, retrieval corpus, tool permissions, decoding policy, agent graph, eval pack, trace bundle, reviewer dispositionlabel operations, evaluator governance, agent review, safety review
Artifact attestation manifestsubject digest, artifact type, producer identity, build provenance, SBOM/eval/map-QA predicate, policy result, allowed scope, rollback targetregistry, OTA/SUMS, Kubernetes/admission policy, safety case, audit

If any of these interfaces are missing at S2+, scale will produce hidden coupling. The symptom is familiar: a model passes offline tests, but nobody can prove which data, labeler, map, calibration, runtime, or prompt artifact produced the behavior.


Reference Architecture Patterns

S0-S1: Research and Prototype Lane

  • Local data sample with a manifest and fixed split.
  • Git-tracked config and run script.
  • Docker or locked environment file.
  • Lightweight experiment tracker or committed run notes.
  • Validation script that emits a deterministic metric table.
  • Explicit note that artifacts are not release eligible.

Best use: feasibility, ablations, early dataset discovery, model comparison.

Avoid: production registry aliases, shared feature stores, continuous retraining, and complex orchestration before the baseline can be rerun.

S2: Single-Product Production Lane

  • Curated training and evaluation snapshots with quality reports.
  • Training DAG that produces a candidate checkpoint and provenance.
  • Evaluation lane with slice metrics, replay smoke, calibration/OOD checks, and runtime package smoke.
  • Registry aliases: candidate, shadow, champion, rollback.
  • Release packet that links model, data, runtime, map, calibration, evaluation, and rollback.
  • Canary or shadow deployment channel with explicit exit criteria.

Best use: first deployable model, offline semantic-map segmenter, production batch labeler, or edge detector that affects operations.

Avoid: training job directly moving champion, mutable bucket prefixes feeding release models, and release decisions based only on aggregate metrics.

S3: Fleet and Multi-Site Lane

  • Site/ODD partitions in data catalog and holdout sets.
  • On-vehicle triage and selective upload with trigger budgets.
  • Active-learning queues separated by site, route, weather, vehicle hardware, map state, and safety slice.
  • Replay/scenario mining with promotion from incident to regression case.
  • Canary by ODD cell, not only by fleet percentage.
  • Monitoring joins model, map, calibration, runtime, and telemetry schema IDs.

Best use: learning from operations while controlling blast radius across airports, yards, campuses, warehouses, and logistics districts.

Avoid: one global champion for every site, one undifferentiated label budget, and dashboards that cannot separate map, calibration, runtime, and model regressions.

S4: Regulated Safety-Critical Lane

  • Immutable evidence snapshots with retention hold.
  • Claim/evidence tables tied to safety-case claims.
  • Expert-reviewed labels and hazard scenarios.
  • Waiver owner, expiry, and operational mitigation for every unresolved regression.
  • Rollback drill and incident response runbook linked to the release.
  • Dual approval where behavior authority changes.

Best use: model, map, planner, or labeler changes that can affect people, aircraft, protected zones, false-free-space, or compliance claims.

Avoid: metric-only approval, expired evidence, ticket-only approval records, unversioned prompt packs, and online learning without controlled gates.

S5: Platform Lane

  • Multi-tenant data/model/eval registry with policy-as-code.
  • Shared GPU scheduler with quotas, priority lanes, cache policy, cost attribution, and gpu-queueing-finops-by-scale.md unit economics.
  • Standardized feature/embedding service only where reuse justifies it.
  • Shared evaluation platform with product-specific adapters.
  • Audit API for model, data, prompt, evaluator, map, calibration, deployment artifacts, and digest-bound attestations.
  • Platform SLOs covering usability, latency, queue time, cost, evidence completeness, and alert quality.

Best use: many teams, products, sites, model families, and foundation-model tools sharing data and compute.

Avoid: platform mandates that ignore autonomy-specific metadata, centralized queues that block incident response, and self-service tools that allow unsupported release bypasses.


Migration Sequence

TriggerAdd nextDo not migrate first
A result must be rerun by another engineerExperiment tracker, dataset snapshot, container, validation scriptKubernetes or shared GPU platform
A model reaches an operator, customer, vehicle, or map releaseRegistry aliases, registry artifact record, release packet, runtime compatibility, rollback artifactAutomated continuous deployment
Multiple sites show different failuresSite/ODD slices, local holdouts, canary channels, trigger budgetsOne global model gate
Multiple products duplicate evaluation logicEvaluation manifest contract, shared metric specs, replay service, scenario catalog, platform eval SLOsOne-off dashboards without release authority
Incidents need root-cause across model/map/runtime/calibrationCommon artifact IDs in telemetry and incident evidence linkNew dashboards without schema fixes
Label volume exceeds reviewer capacityLabel workflow states, QA sampling, auto-label provenance, budget metricsMore auto-labeling without allowed-use controls
Many teams contend for GPUsShared queue, image cache, quotas, owner tags, unit-cost metricsMulti-cloud abstraction before capacity policy
Foundation models influence labels, evals, incidents, safety prose, or tool actionsPrompt/model/retrieval/tool registry, trace capture, calibrated judge evals, and llmops-agent-evaluation-by-scale.md gatesLetting assistants write release truth

Architectures scale cleanly when each migration preserves the old interface. For example, a local JSON dataset manifest can become a catalog entry later if the fields are already correct. A local run note can become a registry metadata record if it already names code, data, config, hardware, and metrics.

The checklist companion (mlops-migration-checklist-by-scale.md) turns this table into explicit S0->S1, S1->S2, S2->S3, S3->S4, and S4->S5 gates, including migration evidence packets and 30/60/90-day rollout plans.


Airside and Managed-Site Reference Target

For a 5-20 vehicle airside, port, yard, campus, or warehouse deployment, the practical target is S2-S3 with selective S4 gates:

  • S2 for deployable perception, map-segmentation, labeler, and runtime packages.
  • S3 for site-sliced data mining, active learning, canary by ODD cell, fleet observability, and site-sliced-release-evidence-by-scale.md release manifests.
  • S4 for releases that affect aircraft/personnel proximity, false-free-space, FOD, map publication, semantic-map training exports, or regulatory evidence.
  • S5 only for shared foundation-model tooling, cross-product data catalogs, large GPU scheduling, or multi-team platform services once reuse pressure is real.

The minimum architecture should therefore include registry-backed release packets, immutable dataset snapshots, map/model/calibration compatibility manifests, trigger-based data collection, site holdouts, replay packages, and explicit rollback. Feature stores, federated learning, and multi-tenant platform services are optional until cross-site reuse, data-sovereignty, or team count forces them; use federated-privacy-preserving-training-policy-by-scale.md before adding FL as an architectural lane.


Failure Modes

Failure modeWhat happensArchitecture control
S5 tools before S1 disciplineExpensive platform with irreproducible experimentsRequire dataset/run/config manifest before platform migration
Local flexibility with no central IDsModels, maps, labels, and telemetry cannot be joined after incidentsCentralize artifact IDs and telemetry fields early
Registry without release evidenceAlias movement looks controlled but approvals are emptyMake alias mutation depend on release packet fields
Data catalog without quality statesUsers find bad data fasterPromotion states and quality reports gate training/eval use
Feature store too earlyTeam maintains infrastructure for features that no model reusesUse manifest-backed offline files until reuse threshold is crossed; apply the feature/embedding store scale guide
Global canary by percentageEasy routes pass while target ODD failsCanary by site, route, vehicle, weather, lighting, and map state
Foundation-model outputs treated as truthPrompt drift changes labels, reports, or safety evidence silentlyPrompt/model/retrieval registry plus reviewer disposition and trace bundle
Platform bypassTeams ship bespoke pipelines outside governancePlatform SLOs, self-service templates, fast incident lanes, and policy enforcement at artifact boundaries

  • mlops-scale-research-scope.md - scale ladder and lifecycle controls.
  • experiment-tracking-reproducibility-by-scale.md - run authority states, reproducibility levels, tracker options, and run manifest contract.
  • model-registry-artifact-lifecycle-by-scale.md - artifact registry records, alias authority, lifecycle states, artifact-set membership, and rollback retention.
  • serving-inference-operations-by-scale.md - serving modes, platform choices, service manifests, traffic policy, autoscaling, ODD-cell rollout, and rollback.
  • platform-sre-reliability-by-scale.md - platform criticality tiers, SLOs, error budgets, backup/restore, DR, tenant isolation, and incident lanes.
  • pipeline-orchestration-release-workflows-by-scale.md - orchestrator selection, workflow state machines, and release/evidence gate placement.
  • evaluation-platform-replay-gates-by-scale.md - metric specs, evaluation manifests, replay/runtime gates, shadow/canary evidence, and evaluation-service SLOs.
  • mlops-migration-checklist-by-scale.md - transition gates, tooling triggers, and migration evidence packets.
  • mlops-scorecards-and-kpis-by-scale.md - scale-specific KPIs, release blockers, cadence, and anti-metrics.
  • model-monitoring-drift-response-by-scale.md - monitoring event contract, drift-response architecture, alert routing, and retraining trigger policy.
  • site-sliced-release-evidence-by-scale.md - ODD-cell release manifests, local holdouts, and rollout state machine.
  • feature-embedding-store-ops-by-scale.md - feature, embedding, vector-search, and manifest store architecture by scale.
  • gpu-queueing-finops-by-scale.md - GPU scheduler, queueing, quota, and FinOps architecture by scale.
  • secure-artifact-attestation-profile.md - signing, SBOM, provenance, registry alias policy, and policy-enforced verification by scale.
  • federated-privacy-preserving-training-policy-by-scale.md - trigger policy for federated, hybrid, local, and privacy-preserving training lanes.
  • llmops-agent-evaluation-by-scale.md - prompt, RAG, judge, VLM/VLA, and tool-agent architecture gates by scale.
  • model-governance-release-evidence.md - registry aliases, claims-and-evidence release packets, and rollback evidence.
  • data-flywheel-airside.md - closed-loop fleet learning and active data mining.
  • ../data-platform/fleet-data-pipeline.md - raw logs, ingestion, data product states, and retention.
  • ../data-platform/data-catalog-lineage-quality-ops.md - data-product contracts, catalog/lakehouse/version-control choices, lineage event boundaries, quality gate severity, SLOs, and deletion propagation.
  • ../../40-runtime-systems/ml-deployment/av-cicd-devops-pipeline.md - CI/CD lane architecture across code, ML, maps, configuration, calibration, and deployment.
  • ../../40-runtime-systems/ml-deployment/production-ml-deployment.md - runtime packaging, TensorRT/Triton deployment, and production monitoring.
  • ../ota/perception-slam-artifact-compatibility-matrix.md - model/map/calibration/runtime compatibility.
  • ../../60-safety-validation/standards-certification/ml-assurance-data-governance.md - ML assurance and data governance.

Sources

Public research notes collected from public sources.