Industry Research

Appearance

Sensor Calibration and Time Synchronization Fundamentals ​

Sensor Calibration and Time Synchronization Fundamentals curated visual

Visual: calibration contract diagram linking intrinsics, extrinsics, trigger source, timestamp semantics, clock alignment, validation logs, and fusion failure modes.

Multi-sensor autonomy depends on two promises: every sensor is placed correctly in space, and every measurement is placed correctly in time. Calibration and time synchronization failures often look like model errors, perception false positives, localization drift, or controller instability because the stack is working with a subtly inconsistent world.

This page covers the foundation needed before LiDAR-camera fusion, radar tracking, GNSS/INS localization, online mapping, docking, and incident replay. It deepens the treatment of the world_T_imu * imu_T_lidar * p_lidar composition chain used throughout the autonomy stack, and explains how errors in that chain degrade aggregated-map quality and downstream semantic segmentation.

Why It Matters ​

EffectImpactRisk if ignored
Extrinsic rotation error δθ at lever arm LLateral offset ≈ L · sin(δθ) per scan; summed over 200-scan aggregation systematic bias reaches sub-5 cm thresholdGhost walls, doubled curbs, scan-seam artefacts in the aggregated map
Extrinsic translation error δtShifts every mapped point by δt regardless of aggregation countUniform world-frame offset; double-wall at >1 cm offset
Time-sync error Δτ at speed vLateral smear ≈ v · Δτ; at 5 m/s and Δτ = 20 ms smear = 10 cmThin-class IoU (curbs, markings, runway lights) halved when smear ≈ class width
LiDAR per-beam intrinsic elevation error 0.1°~17 mm lateral offset at 10 m range; fan-pattern rib artefacts in flat surfacesSystematic ring seams corrupt ground-plane fitting and marking detection
Wrong transform direction or stale calibrationFusion appears offset; silent drift as mount flexes over timeSystematic perception errors that look like model failures

Sub-5 cm RMS map error is achievable with good calibration and PTP hardware synchronization. Above ~10 cm combined error, thin-class IoU degrades sharply.

1. The Calibration Transform Chain ​

Every aggregated point in a multi-scan LiDAR map is placed by the composition:

p_world = world_T_imu  *  imu_T_lidar  *  p_lidar

Each factor is a 4x4 homogeneous matrix in SE(3). Errors in any factor propagate into map position. The chain generalizes when a camera or radar is added:

p_camera = camera_T_imu  *  imu_T_world  *  p_world
         = camera_T_imu  *  world_T_imu^{-1}  *  world_T_imu  *  imu_T_lidar  *  p_lidar

Error compounding works as follows:

  • Extrinsic rotation error δθ (degrees) at lever arm L (m) produces lateral offset ≈ L · sin(δθ). For L = 0.5 m and δθ = 0.5°, offset ≈ 4 mm per scan. Negligible for a single scan but summed over a 200-scan aggregation the systematic bias reaches ~4 cm, crossing the 5 cm target.
  • Extrinsic translation error δt adds directly: a 2 cm translation error shifts every point by 2 cm in the sensor frame, which maps directly into the world frame regardless of aggregation count.
  • Time-sync error Δτ at vehicle speed v produces a lateral smear of ≈ v · Δτ. At 5 m/s airside speed and Δτ = 2 ms, smear = 1 cm; at Δτ = 20 ms, smear = 10 cm.

The deskewing step in Rolling Shutter and LiDAR Deskew is the first consumer of this chain: if imu_T_lidar is wrong, deskewing applies an offset trajectory and smears the scan. A 5 ms temporal error at 5 m/s produces 2.5 cm smear — approaching the practical limit for curb-level features. See §5.3 for the per-point deskewing formula.

2. Intrinsic Calibration ​

2.1 LiDAR Per-Beam Model (HDL-64E Style) ​

A spinning multi-beam LiDAR such as the Velodyne HDL-64E has 64 independent laser/detector pairs. Each beam i has up to six factory-calibrated intrinsic parameters:

ParameterSymbolTypical magnitude
Elevation (vertical) anglephi_iVaries ±15° across beams; error ~0.01–0.1°
Azimuth (rotational) offsettheta_i±0.01–0.5° inter-beam
Range scale factorm_i~1 ± 0.001
Range bias (zero-offset)delta_r_i±10–25 mm
Horizontal position offsetdelta_x_imm-level
Vertical position offsetdelta_z_imm-level

Random range measurement noise is ±25 mm; systematic per-beam misalignment (the "layered" error) is the dominant degrader of planar-surface quality if the factory calibration is ignored. The HDL-64E ships with a factory .yaml or .xml file containing all 64 × 6 values. For solid-state or MEMS-based LiDARs (Livox Avia, Ouster OS1), the per-beam model does not apply directly; manufacturers provide factory intrinsics and do not expect field re-calibration of the intrinsic model.

Calibration method: Place a pattern of known planar surfaces at several distances and angles. For each beam, fit a plane to the returned points; deviations from the best-fit plane residuals drive a Levenberg-Marquardt (or equivalent) least-squares solver. The pattern-plane approach uses five mutually non-parallel planes; five unknowns per beam can in principle be solved with five planes.

A 0.1° elevation error at 10 m range produces a ~17 mm lateral offset — the canonical fan-pattern rib artefact visible in flat-surface scans when the factory calibration is stale.

2.2 Camera Intrinsics — Brown-Conrady and Fisheye Models ​

The standard pinhole projection:

[u]   [fx   0  cx] [X/Z]
[v] = [ 0  fy  cy] [Y/Z]
[1]   [ 0   0   1] [ 1 ]

followed by Brown-Conrady radial-tangential distortion:

r_sq = x*x + y*y
x' = x*(1 + k1*r_sq + k2*r_sq^2 + k3*r_sq^3) + 2*p1*x*y + p2*(r_sq + 2*x*x)
y' = y*(1 + k1*r_sq + k2*r_sq^2 + k3*r_sq^3) + p1*(r_sq + 2*y*y) + 2*p2*x*y

For wide-angle / fisheye lenses (common in airside AV for full-perimeter coverage), the Brown model breaks down beyond ~90° FoV. OpenCV's fisheye module implements the Kannala-Brandt equidistant projection model with four distortion terms k1...k4.

Calibration target paradigms:

TargetCamera detectionLiDAR detectionNotes
CheckerboardCorner subpixel (cv2.findChessboardCorners)Plane fit to reflective surfaceClassic; pose ambiguity at symmetry
ChArUcoCorner + ArUco ID for unambiguous posePlane fitHandles partial occlusion; recommended over pure checkerboard
AprilTag / AprilGridTag ID enables pose without flipsRetroreflective backing → bright clusterKalibr default; robust at low resolution
Retroreflective boardArUco overlay on retroreflective surfaceIntensity peak centroidBest for LiDAR-camera joint calibration

Kalibr's camera-IMU pipeline mandates AprilGrid to avoid symmetry-induced pose flips that corrupt the B-spline trajectory estimate. Acceptance criterion: reprojection error < 0.5 px RMS.

3. Extrinsic Calibration ​

3.1 The SE(3) Rigid Transform Problem ​

Each sensor-to-sensor transform is a rigid body transformation T in SE(3):

T = [R | t]    R in SO(3), t in R^3
    [0 | 1]

Six degrees of freedom: 3 rotation (roll, pitch, yaw) + 3 translation (x, y, z). For N sensors, (N-1) pairwise transforms are needed to define a single reference frame. In a typical airside AV: LiDAR_front to IMU (reference), LiDAR_left to IMU, camera_front to IMU, plus optional LiDAR-to-LiDAR for multi-LiDAR rigs.

See Lie Groups SE(3), SO(3), Adjoints, and Jacobians for the Exp/Log maps and Jacobians used in optimization.

3.2 Target-Based Extrinsic Methods ​

Planar board (checkerboard / ChArUco / AprilTag):

  1. Detect calibration target in camera image — extract 3D corner coordinates via PnP.
  2. Fit a plane through the LiDAR points on the target — extract plane centroid and normal n.
  3. Formulate constraint: the plane centroid and normal in LiDAR frame must match the plane in camera frame after transformation T.
  4. Minimize reprojection error over multiple poses (at least 3 non-coplanar board orientations, ideally 10-30) via nonlinear least squares (Ceres Solver or Gauss-Newton).

The plane-normal constraint decouples rotation from translation, improving numerical conditioning. Retroreflective targets allow sub-centimeter placement accuracy in the LiDAR at 20 m range.

3.3 Targetless Extrinsic Methods ​

MethodKey observablePractical requirements
Motion-based (hand-eye AX=XB)Relative motion from odometrySufficient motion excitation (see §4)
Mutual information (MI) maximizationMI between LiDAR intensity and camera grayTexture-rich scene; overlapping FoV
Semantic feature matchingMatching class boundaries in both modalitiesPre-trained segmentation on both
Continuous-time targetlessSfM camera poses + LiDAR voxel mapStructured environment; good initialization

The MI approach maximizes I(LiDAR_intensity; camera_gray) as a function of the extrinsic T using gradient-free or analytic gradient optimization. Convergence basin is narrow (~10 cm / ~5°); good initialization (e.g., from hand measurement) is required.

4. Hand-Eye and Motion-Based Calibration ​

4.1 The AX = XB Problem ​

The hand-eye problem formalizes as:

A_i * X = X * B_i    for i = 1 ... N motion pairs

Where:

  • A_i = relative motion of the "hand" (vehicle odometry) between pose i-1 and i, in SE(3).
  • B_i = relative motion of the "eye" (LiDAR or camera) between the same two times, in SE(3).
  • X = the unknown fixed transform from sensor to reference frame (the extrinsic).

Solution approaches:

  • Tsai-Lenz (1989): closed-form; solves rotation then translation separately; requires simultaneous non-parallel rotations.
  • Daniilidis (1999): dual quaternion formulation; solves rotation and translation simultaneously; better numerical properties.
  • Full SE(3) optimization (Ceres / g2o): iterative; handles sensor noise; the standard approach in modern calibration tools.

4.2 Required Motion Excitation ​

At least two non-collinear rotation axes must be excited to make X unique (rank condition on the Gramian):

  • Translation-only motion leaves rotation unobservable.
  • Rotation-only motion leaves translation partially observable only if rotation axes are linearly independent across motions.
  • Rule of thumb: a "figure-8" trajectory excites all 6 DoF; straight-and-stop trajectories leave yaw-axis translation degenerate.

Degenerate motion patterns:

Motion patternUnobservable DoF
Straight translation onlyAll rotational DoF of T_IL
Rotation in placeTranslation of T_IL (partially)
Planar (ground-robot) motionVertical translation, roll/pitch of T_IL
Single-axis rotationOther two rotation axes

The "all-zeros-residual trap": after a least-squares optimization, zero residual does not imply correct calibration. If the system is degenerate, the optimizer finds a family of solutions with zero residual in the observable subspace. Always inspect the per-DoF covariance from the FIM inverse and verify calibration on a held-out trajectory with known ground truth.

5. LiDAR-IMU Extrinsic and Temporal Calibration ​

5.1 Parameters to Estimate ​

  • Spatial: T_IL in SE(3) (IMU-to-LiDAR rigid transform, 6 DoF).
  • Temporal: delta_t_IL (scalar; the time offset between LiDAR scan timestamps and IMU timestamps; typically -50 ms to +50 ms).
  • IMU intrinsics (often co-estimated): scale/misalignment matrix, bias b_a and b_g (accelerometer and gyroscope biases).
  • Gravity vector g in the IMU frame (3 DoF, constrained to |g| = 9.805 m/s²).

5.2 The LIO-Standard Online Refinement Pipeline ​

LI-Init (HKU-MARS, IROS 2022): Initializes extrinsic and temporal calibration without targets or a prior map. The algorithm aligns LiDAR odometry (from a scan-matching backbone) with IMU integration residuals to recover delta_t_IL and T_IL. After initialization, FAST-LIO2 refines the extrinsic online via its iterated-EKF for an additional 15-30 s of operation. Output is written into a FAST-LIO2 config YAML.

FAST-LIO2 online extrinsic refinement: The iterated-EKF state vector includes T_IL. The update equation re-estimates T_IL at each scan provided the motion is sufficiently exciting (Fisher information condition monitored internally). In practice, online refinement converges within the first 30-60 s of a survey drive.

OA-LICalib (APRIL-ZJU, TRO 2022): Offline batch continuous-time calibration of both LiDAR intrinsics and LiDAR-IMU spatiotemporal extrinsics. Uses an information-theoretic data selection policy (FIM rank monitoring) to pick only the informative trajectory segments, then applies truncated SVD to update only the identifiable directions of the state. This is the most rigorous offline tool currently available. OA-LICalib reports map thickness (Table VI of the paper): 1.9-2.3 cm for their method versus 4.3-5.4 cm for uncalibrated baselines.

5.3 Per-Point Deskewing — Temporal Calibration in Action ​

The per-point deskewing formula assumes linear interpolation of the relative pose between scan start and end:

T_k(t_i) = Exp( (t_i - t_start)/(t_end - t_start) * Log(T_k(t_start)^{-1} * T_k(t_end)) ) * T_k(t_start)

where t_i is the per-point timestamp of point i within the scan (the t channel in the (x, y, z, intensity, t) tuple). If delta_t_IL is wrong, the deskewing uses an offset trajectory, smearing the scan. A 5 ms temporal error at 5 m/s produces 2.5 cm smear — approaching the practical limit for curb-level features.

6. LiDAR-Camera Extrinsic Calibration ​

6.1 Projection-Error Minimization ​

For a checkerboard visible in both camera and LiDAR:

  1. Extract checkerboard corners p_c in image (pixels).
  2. Fit plane to LiDAR points on board; extract 3D corner positions p_L in the LiDAR frame.
  3. Minimize reprojection residual:
min_{R,t}  sum_i  || p_c_i - project(K * [R|t] * p_L_i) ||^2

where project is the perspective (or fisheye) projection and K is the camera intrinsic matrix. Optimized with Ceres; typically needs 10-30 board poses for robust convergence.

6.2 Edge/Line-Based Methods ​

Extract edges from the camera image (Canny) and LiDAR scan (intensity gradient). Penalize distance from projected LiDAR edge points to the camera edge map. More robust to textureless boards; requires a good initial guess within ~5 cm / ~2°.

6.3 Mutual Information — Targetless ​

T* = argmax_T  I( I_camera(u,v) ; f_LiDAR(R*p + t) )

where f is a surface feature (intensity, normal-derived depth, or semantic label). Gradient ascent (or particle swarm for global search) over the 6-DoF space.

6.4 Key Tooling ​

ToolApproachMulti-sensorNotes
KalibrB-spline continuous-time; camera-IMUMulti-cam multi-IMUDoes not natively calibrate LiDAR; needs companion tool
lidar_camera_calibrationTarget-based plane+point correspondencesSingle pairStandard ROS package; uses checkerboard
targetless (arXiv 2302.05094)MI / structuralYesGeneral, single-shot, automatic
OA-LICalibContinuous-time, observability-awareLiDAR+IMUBest offline LiDAR-IMU; no camera
LI-InitMotion-based initializationLiDAR+IMUFor use before FAST-LIO2 / R3LIVE

7. Time Synchronization Architecture ​

7.1 Clock Domains ​

A vehicle commonly contains several clocks:

GNSS receiver clock
  -> PPS / time-of-week
PTP grandmaster
  -> vehicle Ethernet clocks
sensor hardware clocks
  -> camera, LiDAR, radar, IMU
host system clocks
  -> ROS / middleware timestamps

Good systems make the clock domain explicit in every driver and log.

7.2 GPS-PPS ​

The GPS receiver outputs a hardware pulse on its PPS pin every 1000 ms. The LiDAR's sync input latches its internal counter on the PPS rising edge and reads the absolute UTC time from the NMEA GPRMC sentence arriving ~70 ms later at 9600 baud. Result: LiDAR timestamps aligned to UTC within ~1 µs jitter (dominated by cable propagation and UART latency). Ouster, Velodyne, and Livox all support this mode.

7.3 PTP — IEEE 1588v2 Four-Timestamp Math ​

A software-over-Ethernet protocol using the four-timestamp handshake:

  • Sync message (master to slave): t1 (departure), t2 (arrival).
  • Delay_Req / Delay_Resp (slave to master): t3 (departure), t4 (arrival).

Computed quantities:

link_delay   = [(t4 - t1) - (t3 - t2)] / 2
clock_offset = [(t2 - t1) - (t3 - t4)] / 2
             = (t2 - t1) - link_delay

PTP hardware timestamping in the PHY achieves sub-100 ns accuracy on a switched Ethernet LAN. Software-only PTP (no PHY assist) degrades to ~10-100 µs.

7.4 gPTP / IEEE 802.1AS ​

A profile of IEEE 1588v2 optimized for TSN (Time-Sensitive Networking) automotive Ethernet. Removes BMCA and Announce messages; mandates PHY-level timestamping. AUTOSAR Time Sync over Ethernet uses gPTP as the backbone. An Nvidia Orin ECU and a Velodyne LiDAR on the same gPTP domain can achieve < 1 µs alignment.

7.5 ROS Message Timestamping ​

ros::Time::now() at the driver's receive callback has OS-scheduler jitter of 1-10 ms on a non-RT Linux kernel. For precision mapping this is insufficient; hardware-backed timestamps must be used instead (read directly from the UDP packet header where the sensor embeds its PTP-synchronized timestamp).

7.6 Per-Point Timestamp Channel ​

Modern LiDARs encode t (nanoseconds since scan start, or absolute PTP time) in each point's packet. LIO-SAM and FAST-LIO2 consume this channel directly. If the t field is absent (some older drivers strip it), all points in the scan are assigned the scan start time — this is equivalent to assuming zero vehicle motion during the scan and produces a characteristic "fan smear" on fast-moving platforms.

IMU interpolation at a per-point timestamp t_p, given IMU samples at t_k and t_{k+1}:

alpha = (t_p - t_k) / (t_{k+1} - t_k)
omega_p = (1 - alpha) * omega_k + alpha * omega_{k+1}    (linear angular velocity)

For higher fidelity, use SLERP on the integrated rotation quaternion, or the piecewise-linear deskewing formula from §5.3.

7.7 Clock-Skew Model and MEMS Thermal Drift ​

When sensor A and system ECU B use independent oscillators, a linear clock model suffices over short intervals:

t_B = a * t_A + b

where a is the clock ratio (skew; ideally 1.000000 but drifts at ~10-100 ppm for MEMS oscillators) and b is the clock offset. Estimate a and b by cross-correlating a shared signal (e.g., IMU acceleration spike from a vibration event observed in both clocks). Online estimation via Kalman filter tracking [a, b] as a slowly-varying state is the approach used in LIC-Fusion 2.0.

Thermal drift of a MEMS oscillator is 2-10 ppm/°C. Over a 40°C ambient change (cold hangar to sunny apron), the clock skew can shift by ~100-400 ppm, equivalent to 0.1-0.4 ms drift per second of unsynchronized operation — significant for high-speed mapping passes.

7.8 Why Milliseconds Matter ​

Position error from timestamp offset is approximately:

position_error = vehicle_speed * time_offset
Speed2 ms offset20 ms offset50 ms offset
2 m/s low-speed docking0.004 m0.04 m0.10 m
5 m/s airside taxiway0.010 m0.10 m0.25 m
10 m/s yard or campus0.020 m0.20 m0.50 m
25 m/s road AV0.050 m0.50 m1.25 m

Yaw-rate error also matters. A vehicle turning at 20 deg/s with a 50 ms offset has a one-degree angular mismatch before any sensor noise is considered.

8. Observability and Calibration Quality Metrics ​

8.1 Fisher Information Matrix ​

For a calibration problem estimating state theta from measurements z with noise covariance Sigma:

FIM(theta) = J^T * Sigma^{-1} * J

where J = dz/dtheta is the measurement Jacobian. The Cramer-Rao bound gives the minimum achievable covariance: Cov(theta_hat) >= FIM(theta)^{-1}.

Rank deficiency: if the robot trajectory lacks sufficient excitation, some rows/columns of J are linearly dependent → rank(FIM) < dim(theta). The corresponding eigenvectors of FIM indicate unobservable directions. OA-LICalib monitors rank(FIM) during data collection and alerts the operator when calibration segments are uninformative.

8.2 Observability Gramian ​

For continuous-time systems, the observability Gramian integrates the squared Jacobian over the collection window:

W = integral  J(t)^T * J(t) dt

A Gramian with small minimum singular value sigma_min → near-degenerate calibration. GRIL-Calib for ground robots addresses the structural unobservability of the vertical translation DoF under pure planar motion by adding a ground-plane residual constraint that supplements the Gramian.

8.3 Truncated SVD for Degenerate Directions (OA-LICalib) ​

OA-LICalib's state update uses:

delta_theta = V_r * Sigma_r^{-1} * U_r^T * residual

where the subscript r denotes the truncated (rank-r) SVD of the Jacobian — only the identifiable directions are updated. Unidentifiable directions are frozen, preventing the optimizer from fitting noise in degenerate modes.

9. Online vs Offline / Lifelong Calibration ​

9.1 Offline Batch Calibration ​

Run once (e.g., post-installation, after vehicle maintenance). Tools: OA-LICalib, Kalibr + LI-Init. Requires a dedicated calibration drive with scripted motions (figure-8, tilts, stops). Results written to a configuration file consumed by the runtime.

Time to run: 30-120 min including data collection and optimization. Achievable accuracy: < 1 cm translation, < 0.1° rotation (spatial); < 1 ms temporal with hardware sync.

9.2 Online Continuous Refinement ​

The extrinsic state T_IL is added to the SLAM/LIO state vector and jointly estimated during normal operation. FAST-LIO2's iterated-EKF does this by default. LIC-Fusion 2.0 and FAST-LIO-SAM-based systems also track temporal offset online.

Observability caveat: Online calibration only converges if the platform is moving with sufficient excitation. During straight-line driving segments, the calibration state should be frozen (no update) to prevent divergence. Observability-aware online systems monitor the FIM rank per time window and gate updates accordingly.

9.3 Thermal Drift and Vibration ​

Environmental perturbations that shift calibration:

  • Temperature: Mechanical structures expand/contract (~10-20 µm/m/°C for aluminum). A 40 cm lever arm on a mounting bracket changes by ~80-160 µm over a 20°C range — generally below detection, but IMU MEMS scale factor drifts ~0.1% over the same range.
  • Vibration / shock: A minor collision (curb bump at 1-2 m/s) can shift a camera mounting bracket by 1-5 mm and 0.1-0.5°. Post-shock recalibration is mandatory.
  • Production fleet strategy: Monitor the extrinsic residual (mean reprojection error or map wall-thickness metric) in the data pipeline. Trigger a full recalibration when residual exceeds 1.5× the post-calibration baseline. Typical recalibration interval in production AV fleets: every 500-1000 operating hours or after any collision event.

10. Calibration Evidence in Aggregated Maps ​

10.1 How Calibration Errors Manifest in Maps ​

Error typeMap artefactMagnitude threshold for visibility
Extrinsic rotation error (~0.5°, L=1 m)Lateral shear between scan strips~8 mm per scan; visible after 10+ scans
Extrinsic translation error (2 cm)Uniform offset; "double wall" if sign alternatesVisible at ~1 cm offset
Time-sync error (5 ms, 5 m/s)Motion smear; curved walls, ghost points near dynamic objectsVisible at ~2 cm smear
LiDAR intrinsic per-beam elevation error (0.1°)Fan-pattern rib artefacts in flat surfacesVisible as ~17 mm offset at 10 m range

Double walls are the canonical extrinsic calibration artefact: two nearly-parallel thin surfaces appear where only one physical wall exists, caused by two sensors (or two scan epochs) placing the same wall surface at slightly different world positions.

Dynamic ghosting on slow-moving objects (taxiing aircraft, ground-support vehicles) looks like calibration error but is actually a time-sync error: the object moved between the LiDAR scan timestamp and the IMU-pose timestamp, so deskewing applies the wrong motion correction.

10.2 How the Segmentation Pipeline Detects Bad Calibration ​

  • Seam artefacts at scan boundaries: If the network is trained on clean aggregated maps, it will misclassify "phantom" points between scan strips as object surfaces. Seam artefacts appear as repeating thin parallel structures at scan-overlap distances.
  • Class confusion at calibration-error scale: When calibration error is comparable to the physical width of a class (e.g., 10 cm error vs. 15 cm curb), the model sees a smeared gradient rather than a sharp curb edge, causing ground/curb confusion. Per-class IoU for thin structures (curbs, markings, runway lights) is the most sensitive diagnostic for calibration quality.
  • Map-entropy metric: The entropy (or "thickness") of flat surfaces (ground plane, walls) in the aggregated map is a calibration-quality proxy: lower entropy → sharper surfaces → better calibration. OA-LICalib reports map thickness: 1.9-2.3 cm for their method vs. 4.3-5.4 cm for uncalibrated baselines.

See Aggregated-Map Semantic Segmentation for the full treatment of how map quality gates segmentation performance.

11. Production Checklist ​

11.1 Order of Calibration Operations ​

  1. Camera intrinsics — static, in a controlled environment (low-light variation, static target). Tools: OpenCV calibrateCamera or Kalibr with AprilGrid. Verify: reprojection error < 0.5 px RMS.
  2. LiDAR intrinsics (if factory calibration is unavailable or suspect) — five-plane pattern at 5-20 m range. Verify: planar residual < 15 mm at 20 m.
  3. Camera-to-camera extrinsics (if stereo / multi-camera) — Kalibr with simultaneous AprilGrid views. Verify: stereo reprojection error < 1 px; epipolar error < 1 px.
  4. Camera-to-LiDAR extrinsics — target-based (checkerboard / retroreflective board at 5-15 m, at least 10 poses covering full angular range). Or targetless MI method initialized from rough hand-measurement. Verify: projected LiDAR edges align with image edges within 3 px.
  5. LiDAR-to-IMU spatiotemporal extrinsics — run LI-Init with a figure-8 + stop sequence (minimum 60 s). Verify: temporal offset delta_t < 0.5 ms; spatial residual < 2 cm.
  6. System-level time sync — enable PTP or GPS-PPS before the survey drive; verify sync status via sensor API. Verify: per-point timestamp monotonicity; no > 2 ms discontinuities.
  7. End-to-end map check — collect a 5-minute closed-loop survey drive; build a dense map; measure wall thickness and ground-plane flatness. Accept if wall thickness < 3 cm and ground-plane RMS < 2 cm.

11.2 Recalibration Triggers and Frequency ​

EventAction
New sensor installationFull calibration sequence (steps 1-7)
Sensor repositioning / bracket replacementSteps 4-7
Vehicle collision (any severity)Steps 4-7; check intrinsics if sensor hit directly
Temperature change >30°C from calibration baselineSteps 5-6 (spatial/temporal drift check)
Wall-thickness metric exceeds 5 cm in production mapsSteps 4-7
Operating-hours interval (production fleet)Full calibration every 500-1000 h

11.3 Logging Schema ​

Every calibration artifact must record: software version and tool; date, ambient temperature, platform ID, and sensor serial numbers; all extrinsic transforms (SE(3) matrix + covariance diagonal); temporal offset per sensor pair (delta_t ± sigma); post-calibration quality metrics (reprojection error, wall thickness, ground-plane flatness); FIM minimum singular value per DoF; time-sync mode and verified offset; approval status and rollback target. Distribute static transforms through versioned URDF/YAML — the format matters less than consistent versioning and review.

11.4 Survey Drive Protocol Tie-In ​

The calibration survey drive is distinct from the operational survey drive but informs its design:

  • The calibration drive should include tight figure-8 turns (radius ≤ 5 m), stops, ramp traversals (to excite pitch/roll), and a straight baseline return for consistency check.
  • Operational survey drives that begin without hardware time-sync validation (e.g., PTP lock not confirmed) should be flagged; all data collected without confirmed sync must be post-processed with software offset estimation before map ingestion.
  • The data pipeline should auto-compute the map-thickness metric for every survey leg and reject legs exceeding the 5 cm threshold before feeding into the segmentation training set.

12. Implementation Notes ​

  • Apply LiDAR-IMU extrinsic before integrating IMU increments; the lever arm amplifies angular velocity error if not applied at the IMU output stage.
  • Use T_target_source naming consistently and add projection tests in CI to catch inversion bugs.
  • Verify PTP phase locking is active in multi-LiDAR setups; unsynchronized overlapping beams produce cross-talk returns in the merged cloud.
  • For targetless MI methods, initialize from a hand-measured approximate transform; convergence basin is narrow (~10 cm / ~5°).
  • Use map wall-thickness as the primary end-to-end acceptance criterion; it integrates all error sources simultaneously.
  • Freeze the online calibration state during straight-line segments to prevent divergence from degenerate geometry.
  • Include the FIM minimum singular value in calibration logs; a small sigma_min is a leading indicator of poor calibration even when residuals look acceptable.
  • Preserve calibration covariance; correlated sensors treated as independent create overconfident fusion.
  • Runtime monitors: LiDAR-camera reprojection residuals; multi-LiDAR overlap ICP residuals; PTP offset and grandmaster identity; IMU bias residuals; message age. Feed these into degraded-mode policy (slow down, disable fusion path, request service, or stop). Log raw sensor timestamps, PTP/PPS lock status, static transform tree, and calibration artifact IDs for incident replay.

13. Failure Modes ​

SymptomCauseDiagnostic
Double walls in aggregated mapExtrinsic rotation or translation error between sensors or scan epochsMeasure wall thickness metric; compare with and without per-sensor calibration applied
Fan-pattern rib artefacts in flat surfacesLiDAR per-beam elevation error; stale factory calibrationRun per-beam plane-fit residuals; compare planar std-dev by ring against spec
Silent extrinsic driftMount flex, thermal change, or vibration shifts calibration graduallyMonitor map wall-thickness; require recalibration when thickness exceeds 1.5× post-cal baseline
Deskewing worsens accuracyLiDAR-IMU time offset or extrinsic error; IMU biasCheck residuals during rotation vs. translation segments; validate at known surveyed structure
Motion smear despite deskewTemporal calibration error larger than expected; missing t channelVerify per-point timestamps are present; check delta_t_IL vs. independent measurement
Wrong transform directionSensor fusion appears offset or mirroredUse T_target_source naming and projection tests in CI
Unsynchronized clocksMoving objects smear; estimator innovation grows during turnsUse hardware timestamps and PTP/PPS where available
Timestamping at arrival timeLatency changes with CPU/network loadTimestamp at acquisition in the sensor or driver boundary
PTP grandmaster failoverClock jumps or offset ramps during operationMonitor grandmaster identity and holdover state; define degraded policy
Bad calibration datasetOptimizer finds a plausible but unobservable solutionUse calibration motions and targets that excite all unknowns; inspect FIM eigenvalues
Temperature or vibration sensitivityCalibration correct in depot, wrong in serviceValidate across operating temperature and vibration envelope; recalibrate after collisions
Correlated sensors treated as independentFusion becomes overconfidentPreserve calibration covariance; avoid double-counting measurements from shared sources
Dynamic ghosting mistaken for calibration errorTime-sync error: object moved between LiDAR scan and IMU-pose timestampsCompare static vs. dynamic scene calibration residuals; check delta_t_IL specifically
"All-zeros residual" trapDegenerate calibration: optimizer fits noise in unobservable subspaceInspect per-DoF covariance from FIM inverse; validate on held-out trajectory
Camera-LiDAR projection misalignment at edgesBrown-Conrady model insufficient for fisheye lensSwitch to Kannala-Brandt / OpenCV fisheye model for >90° FoV cameras

14. Related Repository Documents ​

15. Sources ​

Public research notes collected from public sources.

Copyright © 2026 kvynlim