Skip to content
Preprint

LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation

Oct 2026 · 0 citations · 118 references
Computer Science

TL;DR

LucidLyra is a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents, and enables downstream applications including 3D scene editing, 3D asset mining, and robotic simulation.

Abstract

We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into a unified model to jointly infer 3D Gaussians encoding geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization, while training requires neither manual 3D annotations nor captured multi-view data. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, while preserving reconstruction quality. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, 3D asset mining, and robotic simulation.

View source

Similar papers

Review Oct 2026

GATOR: Generative and Agentic 3D Object Reconstruction From Casual Images

Reconstructing complete, scene-aligned 3D objects from casual images requires integrating sparse, uncertain observations and inferring surfaces hidden by occlusions. We present GATOR, a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images. Our local...

Qi-Rui Wu, Stanley T. Birchfield, Hesam Rabeti et al. · 0 citations
Preprint Oct 2026

PixGR: Pixel-Space Generative Decoding for Fast Feed-Forward 3D Reconstruction

PixGR is an efficient framework that integrates lightweight pixel-space generative modeling into feed-forward 3D Gaussian reconstruction to model scene uncertainty and maintains sub-second inference and remains competitive with substantially heavier pretrained generative models.

Ran-Ran Huang, Wen-Jing Bian, Wei-Xun Luo et al. · 0 citations
#machine learning Preprint Oct 2026

Zebra3D: Efficient Hybrid Foundation Models For Scalable 3D Reconstruction

Zebra3D, a layerwise sparse-linear hybrid attention framework that improves the efficiency of pretrained 3D foundation models without training from scratch, achieves strong reconstruction and camera-pose accuracy at high attention sparsity, with training-free variants remaining close to their dense counterparts.

Utkarsh Saxena, Rui-Jing Yang, Mehdi Rezagholizadeh et al. · 0 citations
Preprint Oct 2026

Holi-Spatial 360: Generative Reconstruction from Sparse Panoramas for Spatial Rendering and Intelligence

Spatial intelligence requires scalable 3D environments that are both photorealistic and semantically grounded, yet existing indoor datasets are dominated by room-scale captures and costly LiDAR acquisition. Online real-estate platforms offer abundant panoramas of diverse multi-room homes, but sparse viewpoints and text...

Yuan-Yuan Gao, Zhi-Rui Liu, Yu-Ning Gong et al. · 0 citations
Preprint Oct 2026

GenIA: Generative Reconstruction with Test-Time Input Alignment

GenIA is introduced, a framework for test-time input-aligned generation that grounds SAM3D's generative prior in geometric and photometric observations without retraining the foundation model.

Stefano Esposito, Naama Pearl, Polina Karpikova et al. · 0 citations
#artificial intelligence Preprint Oct 2026

RelationVGGT: Visual Geometry Transformers for 3D Spatial Relation Segmentation

Recent advances in 3D reconstruction have progressed from per-scene optimization to feed-forward inference, and semantic scene understanding has followed suit -- yet existing methods remain confined to object-centric perception, neglecting spatial relations between objects. We formulate 3D spatial relation segmentation...

Minsu Kim, Jaesung Choe, Ji-Woo Lee et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.