Skip to content
Preprint

M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesis

Sep 2026 · 0 citations · 74 references
Computer Science

TL;DR

M3GD is presented, a Camera--LiDAR multimodal representation for generative NVS that composes independently pretrained 2D image and 3D point-cloud foundation models without separately pretraining a cross-modal translator.

Abstract

Robotic novel view synthesis (NVS) must recover both visual appearance and metric 3D structure, yet most generative NVS methods rely only on images, overlooking LiDAR, a complementary sensor common on robotic platforms. We present M3GD, a Camera--LiDAR multimodal representation for generative NVS that composes independently pretrained 2D image and 3D point-cloud foundation models without separately pretraining a cross-modal translator. We show that, after camera projection, frozen LiDAR and image features exhibit substantial shared spatial structure, providing a natural cross-modal representation. M3GD conditions generation on LiDAR through this structure: it combines explicit geometry statistics with learned point-cloud descriptors into view-aligned packets on the image-latent grid, injected through a lightweight residual adapter into a multi-view flow-matching generator whose latent space, decoders, and training objective remain intact. On the GrandTour dataset, M3GD improves target-view RGB and depth synthesis over an image-only version of the same backbone. Ablations show that the gains come from pixel-aligned LiDAR content and that target-view LiDAR acts as a geometric query linking the requested view to source observations. Deployment on a ground robot demonstrates practical real-world operation, with a configurable quality--cost trade-off controlled by the number of Euler integration steps.

View source

Similar papers

Preprint Sep 2026

DXPR: Depth-Based Vision-LiDAR Cross-Modal Place Recognition Using Vision Foundation Models

We present DXPR, a depth-based cross-modal place recognition (CMPR) framework that uses vision foundation models (VFMs) to match monocular camera queries against a LiDAR map without modality-specific encoders. This enables robots and autonomous vehicles to robustly localize using only cameras within pre-built LiDAR map...

Yu-Hang Han, Youngseok Jang, Seungwon Roh et al. · 0 citations
Sep 2026

View Translation: Geometry-Guided Latent Diffusion for Cross-View Image Synthesis on Ground Vehicles

View Translation, a geometry-guided latent diffusion framework that generates a target camera view from a source image, relative camera pose, and an available target-view depth prior, achieves competitive FID while improving LPIPS and PSNR over baseline approaches, supporting cross-view synthesis for ground vehicle per...

O. Mayekar, M. Aiyetigbo, Ameya Salvi et al. · 0 citations
Preprint Aug 2026

Cyclops: LiDAR as a Camera That Dreams in Color

Cyclops is proposed, a framework that translates sparse Non-Repetitive Scanning LiDAR intensity into RGB video, enabling camera-free inference for all-day perception tasks and mitigating inter-frame flickering.

Wei Gao, Jian Shu, Ming-Le Zhao et al. · 0 citations
Preprint Aug 2026

Geometry-Grounded Unified 3D Perception for Autonomous Driving

A Geometry-grounded Unified 3D Perception (GeoUP) framework that adapts the reconstruction-oriented latent of VGGT to calibrated, streaming multi-camera driving scenes and achieves SOTA performance across detection, occupancy, and depth estimation is presented.

Longfei Xu, Xiao-Hui Wang, Ze-Hao Huang et al. · 2 citations
Open access Sep 2026

Cross-modal point cloud completion for robust non-contact 3D measurement under occlusion

UBCnet, a cross-modal point cloud completion framework intended to improve the robustness of image-assisted 3D measurement under incomplete observations, is presented and shows more consistent reconstruction of edges, curved transitions and grasp-relevant structures.

Yu-Hao Yang, Gun Li, Jia-Cheng Luo et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.