Cyclops is proposed, a framework that translates sparse Non-Repetitive Scanning LiDAR intensity into RGB video, enabling camera-free inference for all-day perception tasks and mitigating inter-frame flickering.
Abstract
Conventionally, robotic perception relies heavily on cameras due to the rich semantic texture they provide. However, their performance degrades significantly in low-light or high-dynamic-range environments. Conversely, while Light Detection and Ranging (LiDAR) captures illumination-invariant geometric and intensity properties, the resulting data are typically single-channel and sparse, creating a significant modality gap when applying vision models pre-trained on RGB datasets. In this paper, we propose Cyclops, a framework that translates sparse Non-Repetitive Scanning LiDAR (NRS-LiDAR) intensity into RGB video, enabling camera-free inference for all-day perception tasks. Our approach first converts sparse LiDAR intensity projections into dense representations via a frozen pre-trained densification module, serving as a geometrically rich source condition. The dense intensity latent is then transported toward the target RGB distribution through Latent Bridge Matching (LBM) with a learned velocity field in a few ODE integration steps. To mitigate inter-frame flickering, we inject prior-frame context via temporal attention layers and further formulate the velocity field as a policy optimized by a differentiable terminal reward that encourages terminal fidelity through backpropagation along the ODE trajectory. Extensive experiments demonstrate that the synthesized RGB, including those generated under near-dark conditions, enable standard RGB-based perception models to substantially outperform both LiDAR baselines and conventional cameras on semantic segmentation, lane detection, and point cloud colorization across diverse lighting conditions.
In autonomous driving perception, the fusion of LiDAR and camera modalities has become the dominant paradigm for 3D object detection. However, current multi-modal frameworks heavily rely on massive visual backbones pretrained on 2D semantic tasks. This reliance introduces substantial parameter redundancy and a structural misalignment, as 2D priors are ill-equipped to handle the extreme sparsity of LiDAR projections required for Bird's-Eye-View geometry. To address this, we present DeGuNet, an ultra-compact and plug-and-play image backbone explicitly designed for depth-guided representation learning. By incorporating sparsity-aware feature extraction mechanisms, DeGuNet effectively aligns multi-view images with unstructured LiDAR depth while strictly preventing invalid-region contamination. Extensive experiments on the nuScenes dataset demonstrate DeGuNet's broad plug-and-play applicability and superior efficiency. When integrated into established baselines, it fundamentally eliminates architectural redundancy, reducing GPU memory consumption by up to 66.5% and achieving a 1.16x inference speedup. Concurrently, DeGuNet delivers up to a 6.20 absolute mAP gain, establishing a new paradigm for parameter-efficient multi-modal 3D perception.
Haifan Zhang, Yijing Wang, Peixi Peng et al.· 0 citations
Precise spatial alignment between cameras and LiDAR is a prerequisite for robust multi-modal perception, yet this alignment is often disrupted by mechanical vibrations or drifts during operation. To recalibrate sensors on the fly using naturally collected scene data, 3D Gaussian Splatting-based methods have emerged as promising differentiable solutions. However, current frameworks generally focus on RGB and depth consistency, and have not fully leveraged the reflectance information provided by LiDAR intensity, which limits their ability to fully bridge the modality gap. Bridging this gap, we propose a novel online calibration framework that incorporates LiDAR intensity as a learnable Gaussian attribute, enabling the generation of dense, differentiable intensity maps from sparse LiDAR point clouds. To effectively align these rendered intensity maps with camera grayscale images, we adopt a Modality Independent Neighborhood Descriptor loss that captures local self-similarity patterns rather than absolute pixel values. Experiments on public driving benchmarks validate that leveraging differentiable LiDAR intensity improves online LiDAR–camera calibration.
Daeho Kim, Jeong-Yeun Lee, Kyoleen Kwak et al.· IEEE Robotics and Automation...· 0 citations
Accurate depth perception is a cornerstone of autonomous driving, yet LiDAR sensors—the primary source of metric depth—remain costly and operationally complex. In this paper, we propose a cross-modal supervision framework that uses sparse LiDAR depth maps solely during training, enabling camera-only dense depth inference at test time. A ConvNeXt-base encoder with an FPN neck and a lightweight depth decoding head is trained with a log-scale L1 loss, gradient consistency term applied exclusively at valid LiDAR pixels (~0.7% pixel density), and an image-guided edge-aware smoothness loss operating on all pixels, alongside a two-phase backbone freeze-then-unfreeze strategy to stabilize early convergence. Evaluated on a large-scale Korean highway dataset of 64,840 frames, our model achieves AbsRel of 0.0675, RMSE of 3.907 m, and $\delta \lt 1.25$ accuracy of 0.943, demonstrating that ultra-sparse LiDAR supervision is sufficient to train competitive monocular depth estimators.
H. Park, Young Hoon Suh, Sun-Kug Jung et al.· International Conference on...· 0 citations
LiDAR sensors provide reliable physical distance measurements using laser signals, enabling accurate acquisition of 3D information for various optical systems. However, they are costly, require significant weight and space, and their reliability and accuracy degrade under adverse environmental and weather conditions. In contrast, thermal cameras operating in the infrared spectrum can capture stable visual information even in challenging scenarios such as nighttime, low-light, and rain. However, they cannot directly provide the physical 3D depth information that LiDAR offers. To design efficient optical systems, there is a growing need for techniques that transform thermal image data into LiDAR-like depth information. While deep learning models can theoretically learn direct mappings between thermal and LiDAR modalities, the scarcity of acquiring paired thermal–LiDAR datasets and the difficulty of acquiring them make this task challenging. In this paper, we propose a thermal image-to-LiDAR depth transformation framework. Our method leverages large-scale pretrained visual models for depth estimation to generate initial depth predictions from thermal inputs. Since pretrained RGB-based models face a modality gap when applied to thermal data, we introduce a two-stage depth refinement. Stage 1 corrects global scale inconsistencies, and Stage 2 refines local structural details. Experiments on the MS2 dataset demonstrate that the proposed framework consistently improves the initial DepthPro outputs across day, night, and rainy conditions. Both quantitative metrics and qualitative comparisons show that RGB-pretrained depth predictions can provide useful structural cues for thermal depth estimation when their global scale and local structural errors are explicitly refined.
Low-light image enhancement is crucial in situations where visible sensors might suffer from severe noise and information loss ( e.g., nighttime surveillance). Recent approaches investigate auxiliary modalities invariant to illumination to improve the performance, such as thermal infrared imaging. We propose a Multimodal Intrinsics-Guided Framework that integrates RGB and thermal data to reconstruct well-lit images. Our method utilizes a two-stage pipeline: first, we employ an intrinsic decomposition strategy to separate re-flectance and shading components through knowledge distillation, where a teacher network guides a student model in re-constructing consistent intrinsic components; then, a refine-ment stage restores fine structures and visual details. We train the proposed model on synthetic data from HDRT dataset and demonstrate strong generalization to real-world benchmarks such as LLVIP and V-TIEE, outperforming state-of-the-art methods in most evaluation metrics. Code is available at : https://github.com/simonemelc/TIRGlow
S. Melcarne, J. Dugelay· International Conference on...· 0 citations