Skip to content
Preprint

Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them

Sep 2026 · 0 citations · 54 references
Computer Science

TL;DR

Proxy2World is presented, a controllable world model that learns depth-conditioned RGB generation and joint RGBD generation through cross-modal flow matching from ordinary posed RGBD videos, without training on authored proxy-video pairs.

Abstract

Lightweight scene proxies let creators control scene layout and motion while leaving room for imagination in appearance, lighting, and visual effects. However, a suitable proxy is not uniquely defined, making paired proxy-video data difficult to construct automatically at scale. We present Proxy2World, a controllable world model that learns these complementary capabilities from ordinary posed RGBD videos, without training on authored proxy-video pairs. The model jointly learns depth-conditioned RGB generation and joint RGBD generation through cross-modal flow matching. Learning both tasks enables proxy-camera hybrid denoising at inference to follow the proxy structure while producing natural, detailed visuals. We further introduce ProxyBench to evaluate this capability across a diverse set of scenes, camera trajectories, and subject motions. Experiments on ProxyBench show that Proxy2World achieves a better balance between structural adherence and visual quality than camera-controlled and geometry-conditioned methods, supported by quantitative metrics, VLM assessments, human evaluations and diverse qualitative results.

View source

Similar papers

Preprint Aug 2026

SpatialCrafter: Single Image World Modeling with Generative 3D Proxies

SpatialCrafter is presented, a novel two-stage framework that addresses explorable image-to-scene generation issues by introducing a global 3D proxy for high-fidelity image-to-scene generation and appearance refinement and introduces Parallel Geometry Injection and Proxy-Aware Corruption training strategies.

Chuan Fang, Lingteng Qiu, Yixun Liang et al. · 3 citations
#artificial intelligence Preprint Sep 2026

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

WorldCrafter is a video world model that learns a camera-queryable implicit 3D-aware memory that enables streaming scene exploration from a single input image or text prompt and shows substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploratio...

Wang-Bo Yu, Kun-Hao Liu, Wen-Bo Hu et al. · 3 citations
#artificial intelligence Preprint Oct 2026

Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis

This paper examines the role of Novel View Synthesis (NVS) in geometric representation learning. In principle, NVS should reason about 3D scene structure, thereby enabling transferable multi-view geometric representations. Yet, existing encoder-based NVS methods yield poor representations. This is not because of a lack...

Keerthi Kaashyap, D. Anthony, Akshay Krishnan et al. · 0 citations
Preprint Sep 2026

W2Rep: Learning Visual Representations by Watching the World Change

Images capture the world at one moment, whereas video reveals how it changes. Image self-supervision learns spatial structure from a single moment, while video methods commonly learn temporal relationships inside a representation computed jointly from several frames. We ask whether watching a scene change can instead i...

Wen Huang, Hang Guo, Jia-Rui Yang et al. · 0 citations
Preprint Aug 2026

GenRec: Knowing Where to Reconstruct and Where to Generate

GenRec is introduced, a multi-view flow matching model that builds the reconstruction--generation split directly into its architecture, supervision, and gradient flow, and attains the best reconstruction fidelity in observed regions while also surpassing purely generative baselines on perceptual quality in unobserved o...

Ata Çelen, Jaewoo Jung, Federico Tombari et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.