Skip to content

SalsaNext

What It Is

  • SalsaNext is a real-time range-image LiDAR semantic segmentation network introduced in "SalsaNext: Fast, Uncertainty-Aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving" (Cortinhal, Tzelepi, Aksoy — ISVC 2020, Springer LNCS 12510).
  • It operates on the spherical range image — a compact 2D projection of a full spinning-LiDAR scan (64 × 2048 pixels, 5 channels) — allowing dense 2D convolutional inference rather than sparse 3D operations.
  • Its defining contributions over the predecessor SalsaNet are: residual dilated encoder blocks, pixel-shuffle upsampling, a context module, central dropout regularization, and a dual uncertainty estimation output (aleatoric + epistemic) via Assumed Density Filtering and Monte Carlo dropout.
  • SalsaNext is the canonical projection-family baseline on SemanticKITTI: 59.5% test mIoU at approximately 24 Hz GPU, 6.73 M parameters, 125.68 GFLOPs.
  • See also: cylinder3d.md for the sparse-voxel single-scan alternative; waffleiron.md for the other projection-based method using dense 2D projection of point features rather than a spherical range image.

Core Technical Idea

A spinning automotive LiDAR (e.g., Velodyne HDL-64E) emits 64 vertical beams in a 360° horizontal sweep, producing a structured point cloud whose azimuth and elevation angles are discretized by the sensor geometry. This regularity makes a 2D projection possible without loss of structure: each point has a unique (azimuth, elevation) pair that maps to a pixel in a (width × height) image.

SalsaNext exploits this regularity to convert the 3D segmentation problem into a dense 2D CNN problem. Dense 2D convolution on GPU is the most optimized compute primitive in deep learning — no KD-tree neighbor lookup, no sparse voxel indexing, no graph construction. A 64 × 2048 × 5 tensor is processed exactly as a standard 2D image. This is why SalsaNext achieves approximately 24 Hz with only 6.73 M parameters. By comparison, voxel methods (Cylinder3D) typically run at 6–10 Hz on similar hardware; point methods (PointNet++, RandLA-Net) at 5–15 Hz.

The trade-off is projection artifacts — occlusion, discretization error, and boundary bleeding — mitigated here by dilated convolutions (wider receptive field) and kNN re-projection post-processing. The design thesis: accept the projection approximation and spend the saved compute on uncertainty estimation that quantifies where the approximation fails.

Operator Mechanics

Spherical Range-Image Projection

For a point p = (x, y, z) with range r = sqrt(x^2 + y^2 + z^2), the pixel coordinates are:

u = floor( 0.5 * [1 - atan2(y, x) / pi] * w )
v = floor( [1 - (asin(z / r) + f_down) / f] * h )

Where:

  • u is the column index (azimuth); v is the row index (elevation)
  • h, w = image height and width (for HDL-64E: h = 64, w = 2048)
  • f = |f_down| + |f_up| = total vertical field of view (e.g., 26.8° for HDL-64E)
  • f_down = downward tilt angle (e.g., −24.9° for HDL-64E)

The resulting image has 5 channels per pixel: (x, y, z, intensity, range r). The full tensor is [h × w × 5], fed to the 2D CNN as a standard multi-channel image.

When multiple points map to the same pixel (u, v) — because multiple 3D points lie along the same azimuth-elevation ray at different depths — the standard implementation retains only the nearest (minimum r) and discards the rest. This many-to-one collision is the primary source of the occlusion artifact (see Failure Modes).

kNN Re-Projection (Post-Processing)

CNN inference labels each 2D pixel. Mapping labels back to 3D points:

  1. For each input 3D point p, locate its projected pixel (u, v).
  2. Define a search window W around (u, v) in image space.
  3. Find k nearest neighbors of p in 3D Euclidean space (using range distances from the image as a proxy) from the pool of points in W.
  4. Assign the majority label from the k neighbors to p.

This corrects two artifact classes: discretization error (points at similar angles that competed for a pixel are separated by 3D-distance voting) and boundary bleeding (CNN label smoothing across depth discontinuities in 2D is partially reversed by 3D proximity). The kNN step adds approximately 2.65 ms to total inference time — negligible relative to CNN time. The specific k value follows the RangeNet++ post-processing convention; the SalsaNext paper does not state it explicitly.

Inputs and Outputs

ItemDetail
InputSingle LiDAR scan as (x, y, z, intensity) per point
ProjectionSpherical range image [h × w × 5]; for HDL-64E: 64 × 2048 × 5
Output (raw)Per-pixel class logits, then softmax → per-pixel label + probability
Output (post-kNN)Per-point semantic label (3D), aligned to original point cloud
Optional outputPer-point aleatoric uncertainty sigma_A and epistemic uncertainty sigma_epistemic
Sensor assumptionSingle spinning LiDAR, single ego-centric origin, per-scan

Architecture

Overall Structure

An encoder-decoder CNN operating entirely on the 2D range image:

  1. Context module: A stack of 1×1 and 3×3 convolutions applied to the input before the encoder, providing early global context aggregation across the full image.
  2. Residual dilated encoder: Multiple blocks of dilated convolutions with dilation rates producing effective receptive fields of 3, 5, and 7. Within each block, outputs are concatenated, reduced with a 1×1 convolution, then added via a residual connection. All layers use Leaky-ReLU + Batch Normalization. Spatial downsampling between stages uses average pooling (replacing stride convolution) to reduce information loss during compression.
  3. Central dropout: Spatial dropout inserted after Batch Normalization and before the skip connection, enabling the Monte Carlo dropout uncertainty estimation at inference.
  4. Pixel-shuffle decoder: Replaces transposed convolution. Pixel-shuffle rearranges channels into spatial extent (sub-pixel convolution / depth-to-space), avoiding checkerboard artifacts common in transposed conv. Decoder stages include dilated convolutions and skip connections from the encoder.
  5. Classification head: 1×1 convolution mapping to class logits → softmax.

The five changes from SalsaNet to SalsaNext in summary:

ComponentSalsaNetSalsaNext
DownsamplingStride convolutionAverage pooling
Encoder blocksPlain ResNetResidual dilated conv (rates 3, 5, 7)
Input contextNoneContext module
UpsamplingTransposed convPixel-shuffle
RegularizationStandard dropoutCentral dropout (after BN, before skip)

Parameters: 6.73 M. FLOPs: 125.68 GFLOPs. Encoder-decoder with 16:1 compression at the bottleneck.

Uncertainty Estimation

SalsaNext outputs both aleatoric and epistemic uncertainty per point — uncommon in segmentation methods and practically useful for flagging predictions under sensor noise or distribution shift.

Aleatoric uncertainty (irreducible sensor noise): The known LiDAR sensor noise distribution N(x, v) is propagated through the network using Assumed Density Filtering (ADF). Each activation is modeled as a Gaussian distribution. The network outputs a per-pixel mean prediction mu and per-pixel aleatoric uncertainty sigma_A, quantifying noise from the sensor itself.

Epistemic uncertainty (model uncertainty): Approximated via Monte Carlo (MC) dropout — dropout masks remain active at inference time. The network is run n times (the GitHub implementation defaults to approximately 30 iterations). Epistemic uncertainty per pixel:

sigma_epistemic = (1/n) * sum_{i=1}^{n} (y_i - y_hat)^2

where y_hat is the mean prediction over n trials.

Dropout rate optimization: The optimal dropout rate p* is found via grid search on [0, 1] post-training, minimizing the negative log-likelihood over validation data:

p* = argmin_{p_hat} sum_{d in D} [0.5 * log(sigma_tot^d)
     + (y^d - y_pred^d(p_hat))^2 / (2 * sigma_tot^d)]

Total uncertainty = sigma_A + sigma_epistemic.

Note: MC-dropout uncertainty mode runs approximately 30 forward passes rather than one — significant inference overhead. For standard (non-uncertainty) inference, one forward pass at approximately 24 Hz.

Complexity and Compute

MetricValue
Parameters6.73 M
FLOPs125.68 GFLOPs
GPU throughput (standard inference)~24 Hz
GPU throughput (MC-dropout, ~30 passes)substantially reduced
Jetson AGX Orinnear 10 Hz threshold (approximate; pre-processing 35–83% of runtime)
FPGA (Xilinx Kria KV260, INT8)demonstrated in 2025 (double-head variant)

The pre-processing bottleneck is a critical embedded deployment consideration: spherical projection (typically implemented in C++/Python) constitutes 35–83% of total runtime on Jetson AGX Orin depending on implementation quality (source: arXiv:2410.08365). Optimizing the projection step is as important as model architecture for embedded throughput.

Training Recipe

HyperparameterValue
OptimizerSGD
Initial learning rate0.01
LR decay0.01 per epoch
Momentum0.9
L2 weight decay0.0001
Batch size24
Spatial dropout probability0.2

Loss function:

L = L_wce + L_ls
  • L_wce — weighted cross-entropy with class weights w_i = 1 / sqrt(f_i), where f_i is class frequency. Down-weights frequent classes (road, vegetation); up-weights rare classes (person, motorcyclist).
  • L_ls — Lovász-Softmax: a differentiable surrogate for the Jaccard index (mIoU), directly optimizing the evaluation metric. This is the single largest ablation gain: +6.2 mIoU points (56.6% → 59.5% with kNN on the final ablation configuration).

Data augmentation (each applied independently at 0.5 probability):

  • Random rotation and translation of the point cloud
  • Random flip along the Y-axis
  • Random point dropping (simulates sensor degradation / adverse weather)

Training split: SemanticKITTI sequences 00–10 (~21,000 scans). Validation: sequence 08. Test: sequences 11–21.

Benchmark Results

Ablation — SemanticKITTI Val Set (Sequence 08)

Cumulative contribution of each SalsaNext design choice (mIoU without / with kNN):

Configurationw/o kNN+ kNNParamsFLOPs
SalsaNet baseline43.5%44.8%6.58 M51.60 G
+ Context module44.7%46.0%6.64 M69.20 G
+ Central dropout44.6%46.3%6.64 M69.20 G
+ Average pooling47.7%49.9%5.85 M66.78 G
+ Dilated convolution48.2%50.4%9.25 M161.60 G
+ Pixel-shuffle50.4%53.0%6.73 M125.68 G
+ Lovász-Softmax56.6%59.5%6.73 M125.68 G

Range-Image Family — SemanticKITTI Test Set Comparison

MethodVenue / YearmIoU (test)FPS (GPU)ParamsNotes
RangeNet21++IROS 201947.4%~20 Hz~25 MDarkNet21 backbone
RangeNet53++IROS 201952.2%~12 Hz~50 MDarkNet53; over-parameterized
SalsaNet202045.4%~26 Hz6.58 M19-class retrained
SalsaNextISVC 202059.5%~24 Hz6.73 M+ kNN + Lovász-Softmax
KPRNetarXiv 202063.1%Keypoint post-processing
CENetICME 2022~64.7%Larger kernels, aux heads; community result 67.6% (different training config)
RangeViTCVPR 202364.0%ViT-S/B + conv stem/decoder; Cityscapes pre-train
RangeFormerICCV 202373.3%~6 Hz24.3 MFull-cycle transformer; PQ 64.2%
FRNetarXiv 202373.3%29.1 Hz10.0 MFrustum-range; no kNN post-proc
Fast-FRNetarXiv 202372.5%33.8 Hz7.5 MLighter FRNet variant

FPS figures measured on GPU (RTX/V100-class), single forward pass (no MC-dropout). RangeViT FPS not published; estimated 5–15 Hz given ViT compute cost. CENet published mIoU is ~64.7% (ICME 2022 paper); 67.6% is a community extended result using progressive resolution training — treat separately.

SalsaNext SemanticKITTI Test — Per-Class IoU (%)

carbikemototruckother-vehpersonbicyclistmotorcyclistroadparkingsidewalkother-gndbuildingfencevegtrunkterrainpolesignmIoU
91.948.338.638.931.960.259.019.491.763.775.829.190.264.281.863.666.554.362.159.5

Lowest per-class scores on motorcyclist (19.4%) and other-vehicle (31.9%) — thin, rare objects most vulnerable to boundary bleeding and discretization error.

Variants and Lineage

SalsaNet (Aksoy, Baci, Cavdar — arXiv:1909.08291, September 2019): The predecessor. Encodes road and vehicle only (two classes) using a ResNet encoder-decoder. Explores both BEV and spherical projections. Parameters: 6.58 M; FLOPs: 51.60 G; ~26 Hz GPU; 45.4% mIoU when retrained on 19-class SemanticKITTI. SalsaNext's architectural changes are defined relative to SalsaNet. GitLab: https://gitlab.com/aksoyeren/salsanet

RangeNet++ (Milioto et al., IROS 2019): The founding projection segmenter on SemanticKITTI. DarkNet (21- or 53-layer) backbone on 64 × 2048 range image with 5 channels. Introduces GPU-accelerated kNN post-processing that became standard for the family. RangeNet53++: 52.2% mIoU, ~12 Hz, ~50 M parameters — over-parameterized. GitHub: https://github.com/PRBonn/lidar-bonnetal

CENet (Cheng et al., ICME 2022): Replaces MLP layers with larger-kernel convolutions, selects activation functions (hardswish/mish over ReLU), adds multiple auxiliary segmentation heads with intermediate supervision, and uses progressive resolution training (64×512 → 64×1024 → 64×2048). Code framework derived from SalsaNext. ~64.7% mIoU. Faster and more accurate than SalsaNext; the natural next-step upgrade.

RangeViT (Ando et al., CVPR 2023): First ViT-based range-image segmenter. ViT-S or ViT-B encoder with a convolutional stem and convolutional decoder. Leverages ImageNet/Cityscapes pre-training. 64.0% mIoU (SemanticKITTI), 75.2% mIoU (nuScenes val). Slower than CNN methods.

RangeFormer (Kong et al., ICCV 2023): Full-cycle transformer framework with full-resolution attention, Scalable Training from Range view (STR) augmentation, and range-view semantic coherence loss. First range-view method to surpass point-, voxel-, and fusion-based methods on SemanticKITTI and nuScenes simultaneously. 73.3% mIoU, 64.2% PQ. Trade-off: ~6 Hz GPU — too slow for current embedded real-time.

FRNet (Xu et al., arXiv 2023): Addresses projection artifacts at the architecture level — Frustum Feature Encoder (per-point MLP before projection), Frustum-Point Fusion (bidirectional 2D/3D feature fusion), Head Fusion (multi-level integration), and Range-Interpolation for sparse pixels. Eliminates kNN post-processing. 73.3% mIoU at 29.1 Hz, 10.0 M parameters. Fast-FRNet: 72.5% at 33.8 Hz, 7.5 M. The accuracy-efficiency Pareto frontier for the projection family as of 2023.

Strengths

  • Speed: ~24 Hz GPU at 6.73 M parameters; near real-time on Jetson AGX Orin; FPGA-deployable at INT8. No custom 3D CUDA kernels required.
  • Memory efficiency: Dense 2D tensors require substantially less memory than sparse 3D voxel structures.
  • Mature toolchain: Standard 2D CNN training pipeline; fully reproducible with public code and documented hyperparameters.
  • Uncertainty output: Per-point aleatoric + epistemic uncertainty is rare in segmentation methods and directly actionable for safety margins — flag low-confidence predictions before acting on them or fusing them into a map.
  • Lovász-Softmax training: Directly optimizes mIoU, the evaluation metric, rather than cross-entropy as a proxy — ablation shows +6.2 mIoU points from this change alone.
  • End-to-end differentiable (except the kNN post-processing step, which is non-differentiable).
  • Established baseline with well-characterized performance; straightforward to adapt CENet improvements atop the SalsaNext codebase.

Failure Modes

Occlusion (many-to-one mapping): Multiple 3D points along the same azimuth-elevation ray project to the same pixel. Only the nearest point is retained; far points are silently discarded. The CNN never sees their geometric context. kNN post-processing can recover some labels for discarded points by voting from visible neighbors, but this is a heuristic, not a reconstruction. In airside or dense-infrastructure scenes — a parked jetbridge partially occluded by a terminal wall, an aircraft nose behind a ground vehicle — many points of interest are structurally invisible to the network.

Discretization error: Spherical projection maps continuous 3D coordinates to discrete integer pixel indices. Points near pixel boundaries are quantized to the wrong bin. Angular resolution for a 64 × 2048 image is approximately 0.4° horizontal × 0.4° vertical; points within this angular window compete for the same pixel. The error is proportional to 1/resolution.

Boundary bleeding (label shadow effect): CNNs interpolate across all pixels in the receptive field regardless of 3D depth discontinuities. At the boundary between a near object (e.g., a person at 5 m) and a far background (building at 20 m), adjacent 2D pixels represent geometrically discontinuous 3D regions. CNN predictions smear across this boundary — "person" labels appear on background pixels, or vice versa. kNN post-processing mitigates this by using 3D Euclidean distances, but the correction is imperfect for thin structures (motorcyclist: 19.4% IoU).

Pre-processing bottleneck on embedded hardware: Spherical projection constitutes 35–83% of total runtime on Jetson AGX Orin (approximate, source arXiv:2410.08365). A fast inference model with a slow projection step loses the speed advantage.

Sparse distant regions: At far range (>50 m), points are sparse and pixels are often empty. CNN inference over empty pixels degrades performance. FRNet's Range-Interpolation addresses this; SalsaNext does not.

Rain/fog/dust: Atmospheric scatter produces false near-range returns. The nearest-point-keeper rule retains these noise returns and discards valid background points, corrupting the range image.

High-speed motion distortion: At vehicle speeds above ~50 km/h, the 360° sweep is not contemporaneous — points from the scan start and end are offset by vehicle translation. This introduces geometric distortion that the spherical projection assumes away.

Sensor specificity: Image dimensions (64 × 2048 for HDL-64E) are tied to a specific LiDAR model. A 128-beam sensor or a solid-state LiDAR requires different image dimensions and retraining; solid-state sensors with non-uniform scan patterns may not project to a regular 2D grid at all.

Domain Fit

DomainFitNote
Road AV (on-vehicle, single-scan)strongDesigned for exactly this; the canonical projection baseline for this use case.
Road AV (offline / aggregated maps)not suitableArchitecturally mismatched — see Aggregated-Map Suitability below.
Airside (single-scan, on-vehicle)conditionalViable single-scan baseline once retrained on airside data; thin-object artifacts affect aircraft geometry and GSE at range; occlusion failures matter in high-density ramp areas.
Airside (aggregated / static map)not suitableNo single sensor origin; projection degenerates for multi-viewpoint data.
Warehouse / port / logistics-yardconditionalFits if sensor is a single spinning LiDAR on an ego-vehicle; weak for fused or surveyed clouds; indoor geometry and heavy occlusion increase artifact rates.
Mining / construction / agriculturelimitedRough terrain and vegetation density amplify boundary-bleeding and occlusion artifacts; solid-state sensors common in these settings require retraining.
Per-scan pre-labeling front-endstrongExcellent role: fast per-scan labels from SalsaNext or FRNet feed a separate 3D map integrator as a pre-labeling stage.

Aggregated-Map Suitability

Range-image methods are architecturally mismatched for segmenting aggregated multi-viewpoint LiDAR maps. This is a structural incompatibility, not a solvable fine-tuning problem.

The spherical projection is defined by a single sensor origin. The azimuth and elevation angles of every point are computed relative to the ego-vehicle position at the moment of that scan:

u = f(atan2(y, x))      <- azimuth from THIS sensor position
v = f(asin(z / r))      <- elevation from THIS sensor position

For a single scan, this is exact and lossless (up to discretization). For an aggregated multi-scan map — points from tens or hundreds of sensor poses — there is no single origin. Applying the projection to an aggregated map requires choosing an arbitrary reference origin, after which:

  1. The image becomes vastly sparser. Points from 100 scan positions do not form a dense 64 × 2048 grid from any single viewpoint. Many pixels are empty; others have thousands of competing points from different passes at different distances.
  2. Image size must grow arbitrarily. A 200 m × 200 m urban-district map cannot be represented as a 64 × 2048 spherical sweep — the elevation dimension loses meaning; azimuth density becomes non-uniform.
  3. kNN post-processing assumptions break. kNN relies on the implicit spatial structure of the range image (nearby pixels ≈ nearby 3D points from the same scan). This is violated for multi-origin data.
  4. The pre-training distribution shifts entirely. SalsaNext and all range-image methods are trained on single-scan data. Aggregated maps have completely different point density, overlap patterns, and occlusion structure — not a domain adaptation problem, a format incompatibility.

The correct methods for aggregated multi-viewpoint maps are voxel-based (MinkUNet, Cylinder3D with Cartesian voxels, SphereFormer), point-based (KPConv, PointTransformer), or fusion methods that operate on 3D structure directly. See ../overview/aggregated-map-semantic-segmentation.md §7.3 for the projection family treatment and §7.8 for the model-family comparison.

Where range-image methods ARE relevant to the overall mapping pipeline:

  • Role 1 — Real-time on-vehicle segmenter (single-scan). SalsaNext or FRNet run on the vehicle at 24–34 Hz, segmenting each incoming scan ego-centrically. Outputs feed dynamic object tracking, drivable-space extraction, and candidate labels for the map-building pipeline. The range-image method operates at data-collection time, not at map-segmentation time.
  • Role 2 — Short-horizon temporal accumulation (2–5 scans). Methods like Meta-RangeSeg (arXiv:2202.13377) accumulate a small number of consecutive scans by computing range residual images. The accumulation window is short enough that the sensor origin shift is small, preserving the single-origin approximation. This improves segmentation of slow-moving or partially occluded objects. Categorically different from full map aggregation.
  • Role 3 — Per-scan pre-labeling front-end for map construction. The aggregated map pipeline can use per-scan range-image predictions as initial semantic labels, then merge and regularize them in 3D in a separate step. Fast, high-throughput labeling per scan; a 3D map integrator handles the rest.
  • Role 4 — Uncertainty output as quality signal. SalsaNext's per-point uncertainty estimates can flag unreliable predictions before they are fused into the map database, preventing high-uncertainty labels from corrupting map semantics.

Implementation Notes

  • Use SalsaNext (or its CENet/FRNet successors) for the single-scan, on-vehicle segmentation task. Do not apply to aggregated maps — the projection assumption is the deciding factor.
  • Optimize the spherical projection pre-processing first for embedded deployment. On Jetson AGX Orin it can consume 35–83% of total runtime; a fast CUDA or TensorRT projection kernel is as important as model optimization.
  • The MC-dropout uncertainty mode (~30 forward passes) is expensive; run it selectively — useful for flagging low-confidence predictions before map fusion or for calibrating the safety margin on a new deployment environment, not for routine per-frame inference.
  • Apply the Lovász-Softmax loss term; the ablation is definitive (+6.2 mIoU) and the implementation is a one-line change using the public Lovász-Softmax library.
  • Class-weight the cross-entropy using w_i = 1 / sqrt(f_i) — essential for rare classes (motorcyclist, person) that would otherwise be drowned by road and vegetation gradients.
  • Image dimensions (64 × 2048 for HDL-64E) are sensor-specific. For a different LiDAR model (128-beam, OS2-128, solid-state), compute the correct h × w from the sensor's vertical beam count and horizontal resolution, and retrain from scratch or fine-tune.
  • For a richer baseline at similar compute cost, CENet is the natural upgrade: it derives from the SalsaNext codebase, adds larger-kernel convolutions and auxiliary heads, and gains ~5 mIoU points. For the accuracy-efficiency frontier, FRNet (73.3% at 29.1 Hz) or Fast-FRNet (72.5% at 33.8 Hz) supersede SalsaNext without requiring kNN post-processing.
  • SalsaNext's uncertainty output is a practical differentiator in safety-critical contexts (airside AV, industrial AV) — expose it as a per-point quality score for downstream consumers.

Sources

Public research notes collected from public sources.