It is indicated that LiDAR-conditioned diffusion models can induce structured 3D representations from 2D supervision alone, with a modality-dependent manifold that locally unifies near a shared bottleneck.
Abstract
Transferring the rich priors of large 2D foundation models to sparse 3D LiDAR remains challenging, as training native 3D foundation models at comparable scale is limited by data and annotation scarcity. We introduce a LiDAR-conditioned diffusion model trained on pseudo-labels from off-the-shelf 2D foundation models. The model supports multiple output modalities, including depth, semantic segmentation and instance prediction, selectable via a textual task prompt. Because the model is conditioned on LiDAR, both its outputs and its intermediate UNet features can be projected back onto the input point cloud, enabling analysis of a 3D representation learned entirely under 2D supervision. We study this representation directly in point-cloud space, explicitly excluding raw spatial coordinates to isolate feature content from projection geometry. Linear probes recover up to ~23% Mean Intersection over Union (MIoU) on 3D semantic classes, compared to ~3.5% for a matched Gaussian-noise control, indicating substantial non-trivial structure. Pairwise cosine similarity across modality-specific feature streams reveals a layered organization. Early encoder layers remain weakly aligned across modalities while individually decodable, intermediate layers converge toward a shared representation, and decoder layers re-specialize toward task-specific outputs. These findings indicate that LiDAR-conditioned diffusion models can induce structured 3D representations from 2D supervision alone, with a modality-dependent manifold that locally unifies near a shared bottleneck. This positions diffusion as a viable mechanism for transferring large-scale 2D priors into sparse 3D domains.
LiDAR-SAM2 is introduced, a framework that turns a 2D video foundation model, SAM2, into a scalable source of supervision for the 4D LiDAR domain and produces semantic and panoptic labels on SemanticKITTI that approach the quality of full human annotation from only a few points.
Jihun Kim, Hyun-Kurl Jang, Hyemin Yang et al.· 0 citations
Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it...
Cigdem Kokenoz, Amir Salarpour, Alkim Domeke et al.· 0 citations
While 2D Vision Foundation Models offer a pathway to automate 3D semantic pseudo-labelling, translating these priors into robust 3D representations typically requires complex heuristics or multi-model ensembles. We introduce SplatLabel, an automated pipeline that leverages a 4D Gaussian representation to extract LiDAR...
Nitya Nanvani, Andras Palffy, H. Caesar· 0 citations
This work studies the effect of extrinsic reliability and pose conditioning on feed-forward 3DGS, and proposes a calibration-aware pipeline that anchors reconstructed scenes to the robot's metric workspace and presents a renderable, metric-pose-anchored dataset with scene-level reliability information for robot manipul...
This work investigates whether a frozen, self-supervised point transformer already contains the structural information required to isolate object instances without any handcrafted geometric prior, and develops a training-free segmenter that groups points via connected components on a key-similarity graph, using neither...
Ted Lentsch, Santiago Montiel-Mar'in, Holger Caesar et al.· 0 citations
GhostPoint is proposed, an SSL framework that hallucinates latent features in local neighborhoods around discovered instances, generated via a novel instance voxel dilation, and introduces a predictor-level supervision scheme on sampled voxels from generated neighborhoods.
Mohamed Abdelsamad, Bin Yang, Michael Ulrich et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.