Active Labeling and Budget Operations

Last updated: 2026-05-23

Why It Matters

Fleet learning is constrained twice: vehicles cannot upload everything, and humans cannot label everything that reaches the cloud. Active labeling operations decide which samples earn bandwidth, which uploaded samples earn annotation spend, which predictions can be reviewed instead of labeled from scratch, and which labels are good enough to promote into training or safety evidence.

This page covers budgeted labeling operations for perception, prediction, planning replay, and data-quality review.

Operating Model

1. Maintain separate budgets for upload, auto-label inference, human annotation, expert review, and label QA. Do not spend human review on clips that are blocked by privacy, corruption, missing calibration, or duplicate coverage.
2. Score candidates in two stages. On vehicle or edge storage, select clips under bandwidth and retention constraints. In the cloud, select from uploaded data under a global annotation budget.
3. Balance uncertainty, diversity, coverage, and operational risk. DUAL frames this as distributed upload plus active labeling for resource-constrained fleets; the practical lesson is to avoid spending the global label budget on redundant local uploads.
4. Use FiftyOne or equivalent dataset tooling to inspect embeddings, near-duplicates, hard examples, label mistakes, and model predictions before creating annotation tasks.
5. Use Label Studio or equivalent annotation tooling for pre-annotations, ML backend predictions, interactive labeling, and human review. Predictions are not ground truth until reviewed and submitted.
6. Promote labels by state: candidate, pre_labeled, human_labeled, qa_passed, approved_for_training, approved_for_safety_evidence, rejected.
7. For aggregated-map semantic labels, keep open-vocabulary/offboard outputs in a separate review lane. ZOPP, VESPA, UniLiPs, LOSC, SALT, OpenUrban3D, SAM4D, or Grounded-SAM-style predictions can propose candidate_concept and pseudo_labeled regions, but they stay read-only pre-annotations until a reviewer maps them to the controlled taxonomy, requests a taxonomy change, or rejects them. The labeler, prompt pack, thresholds, and reviewer workflow should be registered through ../mlops/offboard-labeler-registry-by-scale.md before S2+ training or release use.

Label Operations by MLOps Scale

Label operations scale from a research convenience into a governed production system. The main transition is S2 to S3: annotation stops being a model-team task and becomes a fleet budget, site-coverage, reviewer-quality, and safety-evidence function.

MLOps scale	Selection policy	Label QA expectation	Promotion boundary
S0 notebook research	Hand-picked examples and small exploratory batches	Manual spot review by the researcher	Labels support exploration only
S1 repeatable prototype	Fixed seed set plus diversity/uncertainty samples	Versioned instructions, schema, and reviewer notes	Labels can define a baseline dataset
S2 production product	Budgeted active-learning queue with dedupe and data-quality filters	QA sampling by class, source, reviewer, and model-disagreement slice	qa_passed labels can enter training; safety evidence needs extra approval
S3 fleet and multi-site	Per-site and per-ODD budgets for rare classes, drift, incidents, and local holdouts	Reviewer agreement, rework rate, defect taxonomy, and local slice coverage are tracked	Promotion is scoped by site/ODD cell and downstream release packet
S4 regulated safety-critical	Safety-weighted labeling for hazards, unknowns, false-free-space, FOD, and incident replay	Expert review and immutable audit trail for evidence-bearing labels	approved_for_safety_evidence requires named data steward and safety owner approval
S5 platform scale	Shared labeling marketplace across products with quotas, policy, privacy, and cost controls	Platform-level quality dashboards, calibration sets, vendor scorecards, and audit APIs	Promotion states are enforced by workflow policy, not convention

Auto-accept thresholds should be conservative and scale-aware. A high-confidence pre-label can reduce reviewer effort at S2, but at S4 it still needs evidence that the auto-labeler, prompt pack, taxonomy, calibration, and reviewer workflow were all the approved versions for the release claim.

Evidence Artifacts

Artifact	Minimum contents	Owner
Budget ledger	Budget type, allocation, spend, remaining quota, owner, period	Label operations
Candidate score record	Source clip, score components, selected/not selected reason, dedupe cluster	Data platform
Upload selection manifest	Vehicle, local model version, storage constraint, selected sample IDs	Fleet data
Annotation batch	Task IDs, label schema, instructions, source data snapshots, pre-label model	Label operations
Pre-annotation record	Model version, prediction score, Label Studio prediction payload, review status	MLOps
Semantic-map candidate batch	Candidate label batch ID, candidate concept, prompt set, source map or sequence hash, projection/calibration hash, offboard model/checkpoint, proposal score, consolidation/voting policy, unknown policy	Label operations
Taxonomy promotion record	Candidate name, alias/parent/new-class decision, reviewed examples, class-frequency evidence, boundary-rule update, taxonomy-change request ID	Data steward
QA-passed semantic patch	Accepted tile/segment IDs, reviewer decisions, taxonomy ID/hash, QA report, semantic-map manifest ID, back-projection export ID	Map operations
QA report	Inter-annotator checks, reviewer decisions, defect taxonomy, rework rate	Label QA
Promotion record	Approved label snapshot, allowed use, expiry, downstream dataset IDs	Data steward

Acceptance Checks

• Selection decisions are reproducible from stored scores, budgets, and source snapshots.
• The annotation batch has a fixed label schema, task instructions, and ODD scope.
• Pre-labels are clearly distinguished from reviewed labels in storage and downstream manifests.
• Open-vocabulary map labels have a reviewed taxonomy mapping before they affect training, replay, release evidence, or a signed semantic-map bundle.
• Label QA samples cover high-risk classes, rare classes, new airports, night/weather slices, and model-disagreement cases.
• Duplicate and near-duplicate samples are controlled before spending annotation budget.
• Labels promoted to safety evidence have stricter QA than labels used only for exploratory training.
• Budget reports expose cost per accepted label, defect rate, rework rate, and downstream model or replay impact.

Failure Modes

Failure mode	Consequence	Control
Label budget follows upload volume	Common routes consume all annotation spend	Global cloud selection with diversity and risk weighting
Unreviewed predictions enter training	Model reinforces its own errors	Separate pre_labeled from qa_passed states
Open-vocabulary prompt names become class IDs directly	Runtime map taxonomy drifts without evidence or backward compatibility	Require taxonomy promotion record and manifest version bump
Active learning chases only uncertainty	Dataset fills with outliers and corrupt samples	Combine uncertainty with quality, diversity, and ODD coverage
Label instructions drift	Annotators create incompatible labels	Version task instructions and schema with each batch
QA samples are random only	Rare safety classes are under-reviewed	Risk-weight QA sampling
Duplicate clips are labeled repeatedly	Budget waste and biased training distribution	Near-duplicate detection before task creation
Promotion has no allowed-use scope	Exploratory labels become safety evidence by accident	Require explicit promotion state and data steward approval

Related Repository Docs

• 50-cloud-fleet/mlops/data-flywheel-airside.md
• 50-cloud-fleet/mlops/offboard-labeler-registry-by-scale.md
• 50-cloud-fleet/data-platform/fleet-data-pipeline.md
• 50-cloud-fleet/data-platform/3d-annotation-tools.md
• 50-cloud-fleet/data-platform/perception-slam-fleet-data-contract.md
• 30-autonomy-stack/perception/datasets-benchmarks/fod-and-airport-apron-detection-datasets.md
• 60-safety-validation/verification-validation/evaluation-benchmarks.md
• 60-safety-validation/verification-validation/knowledge-base-evaluation-protocol.md

Sources

• Akcin, Goel, Zhao, and Chinchali, "Distributed Upload and Active Labeling for Resource-Constrained Fleet Learning," Proceedings of Machine Learning Research 305, 2025. https://proceedings.mlr.press/v305/akcin25a.html
• FiftyOne documentation. https://docs.voxel51.com/
• FiftyOne, "Annotating Datasets." https://docs.voxel51.com/integrations/annotation.html
• Label Studio, "Integrate Label Studio into your machine learning pipeline." https://labelstud.io/guide/ml.html
• Label Studio, "Import pre-annotated data into Label Studio." https://labelstud.io/guide/predictions
• ASAM OpenLABEL. https://www.asam.net/standards/detail/openlabel/
• ZOPP, "A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving." https://arxiv.org/abs/2411.05311
• VESPA, "Towards un(Human)supervised Open-World Pointcloud Labeling for Autonomous Driving." https://arxiv.org/abs/2507.20397
• UniLiPs, "Unified LiDAR Pseudo-Labeling with Geometry-Grounded Dynamic Scene Decomposition." https://arxiv.org/abs/2601.05105
• LOSC, "LiDAR Open-voc Segmentation Consolidator." https://arxiv.org/abs/2507.07605
• SALT, "A Flexible Semi-Automatic Labeling Tool for General LiDAR Point Clouds with Cross-Scene Adaptability and 4D Consistency." https://arxiv.org/abs/2503.23980
• SAM4D, "Segment Anything in Camera and LiDAR Streams." https://openaccess.thecvf.com/content/ICCV2025/html/Xu_SAM4D_Segment_Anything_in_Camera_and_LiDAR_Streams_ICCV_2025_paper.html
• OpenUrban3D, "Annotation-Free Open-Vocabulary Semantic Segmentation of Large-Scale Urban Point Clouds." https://arxiv.org/abs/2509.10842