SpaCeFormer

What It Is

SpaCeFormer is a proposal-free open-vocabulary 3D instance segmentation method.
The paper title is "SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation".
The project page lists the work as accepted to ICML 2026 and the arXiv record was submitted in April 2026.
It predicts 3D instance masks directly from 3D input instead of first generating 2D proposals or class-agnostic 3D proposals.
It is paired with SpaCeFormer-3M, a generated indoor 3D mask-text corpus with 604K instance masks and about 3.0M captions across 7,361 scenes.
It is mainly an indoor 3D scene-understanding method, not a driving-specific LiDAR detector.
The implementation is released through NVIDIA's WarpConvNet project.

Use a sparse-voxel transformer that mixes local 3D spatial windows with space-filling-curve serialization.
Keep local boundaries coherent with fixed-metric 3D window attention.
Use Morton or Hilbert curve attention in deeper stages to get predictable long-range mixing without quadratic full-scene attention.
Add 3D rotary positional embeddings so attention sees relative voxel geometry.
Use learned queries in a RoPE-enhanced decoder to predict masks, CLIP-aligned features, and foreground scores directly.
Avoid external proposal generation so runtime is dominated by one 3D model instead of a multi-stage 2D plus 3D pipeline.
Train from automatically generated multi-view instance masks and captions rather than ground-truth 3D mask labels.

Inputs are 3D scene data represented as sparse voxels or point-derived sparse tensors supported by WarpConvNet.
Training-data inputs include multi-view indoor scenes, camera geometry, generated 2D masks, and VLM captions.
Text supervision comes from multi-view-consistent captions attached to generated 3D instance masks.
Model outputs include 3D instance masks, foreground scores, and CLIP-aligned features for open-vocabulary querying.
It does not output object boxes, occupancy freespace, tracks, scene flow, or safety-certified obstacle absence.
It assumes enough 3D scene coverage for instance geometry; very sparse long-range outdoor LiDAR is a separate transfer problem.

Generate SpaCeFormer-3M by clustering multi-view masks into 3D instances and captioning each instance from several views.
Voxelize or sparsify the 3D scene into the representation expected by WarpConvNet.
Run a hierarchical sparse-voxel U-Net backbone.
Apply Space Attention in shallow stages where local geometric adjacency matters for mask boundaries.
Apply Curve Attention in deeper stages where fixed-length serialized patches make larger-context mixing cheaper.
Encode 3D relative positions with voxel RoPE.
Refine learned instance queries through decoder cross-attention and self-attention, then predict masks and open-vocabulary features.

SpaCeFormer-3M aggregates 604,127 instance masks and 3,020,635 captions from ScanNet, ScanNet++, ARKitScenes, and Matterport3D.
The project page reports 0.14 seconds per scene for inference.
On ScanNet200 zero-shot evaluation, the project page reports 11.1 mAP for the proposal-free, 3D-only SpaCeFormer setting.
On ScanNet++, it reports 22.9 mAP; on Replica, it reports 24.1 mAP.
The arXiv abstract reports SpaCeFormer-3M mask recall at 54.3% versus 2.5% for a prior single-view pipeline at IoU greater than 0.5.
Comparisons should separate matched 3D-only, proposal-free, no-ground-truth-3D-supervision settings from stronger multi-view 2D pipelines or supervised 3D-mask proposal methods.
Runtime claims need hardware, scene size, voxel resolution, and implementation-version context before being used in deployment budgets.

Direct proposal-free 3D masks reduce dependence on slow 2D proposal aggregation pipelines.
The source release through WarpConvNet makes it more reproducible than paper-only frontier items.
SpaCeFormer-3M provides a practical recipe for large mask-text 3D supervision without manual 3D mask annotation.
The method targets both speed and open-vocabulary instance masks, which are often traded off in 3D scene understanding.
It complements Mosaic3D by emphasizing fast proposal-free instance segmentation rather than only broad foundation dataset/model coverage.
It complements OpenVox by providing frame or scene segmentation inputs rather than an online probabilistic voxel map.

The evidence is indoor-scene heavy; road, airside, yard, port, mining, and agricultural scenes are not proven by the reported benchmarks.
Generated captions can be visually plausible while still wrong for operational taxonomies.
Multi-view mask clustering can merge adjacent objects or split one object into several instances.
Sparse or distant LiDAR may not provide enough geometry for small hazards, FOD, hoses, chocks, or low-profile equipment.
Open-vocabulary scores are not calibrated safety evidence and should not drive emergency-stop logic alone.
Dataset and model dependencies need licensing and provenance review before commercial reuse.

Domain	Fit	Note
Indoor robots and AR/VR	strong research fit	The reported datasets, scene scale, and open-vocabulary mask tasks are directly aligned.
Warehouse / logistics facility	conditional	Similar indoor geometry can transfer, but forklifts, pallets, reflective wrap, and dynamic aisles need evaluation.
Road AV	weak direct fit	The method is not trained or evaluated as a road-scene 3D detector, but the mask-text data recipe can inform long-tail labeling.
Airside	conditional for data engines, weak for runtime safety	It can help mine or pre-label terminal interiors, maintenance bays, baggage halls, and dense apron scans, but outdoor sparse-LiDAR and FOD evidence is missing.
Port / mining / construction / agriculture	insufficient evidence	Outdoor scale, dust, rain, vegetation, and machinery geometry differ from indoor benchmark assumptions.

Treat SpaCeFormer first as a data-engine and semantic-map candidate, not as the primary obstacle detector.
Preserve generated caption, source dataset, viewpoint count, and mask-clustering provenance for every training mask.
Evaluate prompt stability with operational synonyms before exposing open-vocabulary labels to reviewers.
Compare against Mosaic3D, OpenVox, Open3DTrack, and 3D-AVS by task boundary: mask-text pretraining, online voxel mapping, tracking, or auto-vocabulary LiDAR segmentation.
Benchmark on the target sensor density and voxel resolution before accepting the reported 0.14-second runtime.
Keep closed-set safety perception, occupancy, and tracking layers separate from open-vocabulary mask discovery.