Skip to content

WildHSR: Metric Feed-Forward 4D People-Scene Reconstruction from a 3D Foundation Model

Sep 2026 · 0 citations · 43 references
Computer Science

TL;DR

WildHSR combines both readouts to reconstruct metric cameras, scene and people from monocular video, and is the first feed-forward method in the published comparison to beat the best optimization-based WA-MPJPE and RTE while leading feed-forward methods on all three world-frame metrics.

Abstract

3D foundation models recover video cameras and geometry in one forward pass, but some of the strongest are up to scale. Joint people-scene reconstruction then requires two missing outputs: metric scale and persistent person identity. We ask whether one up-to-scale foundation representation can support both through lightweight adaptation. Exact metric labels are scarce, but unlabeled in-the-wild video is abundant. We use people in curated web video to initialise the solution: a posed metric body and 2D keypoints give an approximate, closed-form scale pseudo-label. These pseudo-labels pretrain a Scale Readout, which is then fine-tuned together with a lightweight adapter using exact metric supervision from standard real-video training splits. At inference the head predicts metric scale from foundation-model tokens, without the ruler or its teachers. For person identity, we probe the pretrained foundation model alone and find evidence that its intermediate query-key features encode person correspondence across frames. In most evaluated moving-person clips, a mid-layer token prefers that person over the vacated location and other people. A tiny projection reads this correspondence; together with metric pelvis motion and proposal confidence, it drives dustbin-aware Sinkhorn association of per-frame bodies. WildHSR combines both readouts to reconstruct metric cameras, scene and people from monocular video. Each window is predicted feed-forward; analytic association and Sim(3) composition connect windows. On EMDB-2, WildHSR is the first feed-forward method in the published comparison to beat the best optimization-based WA-MPJPE and RTE while leading feed-forward methods on all three world-frame metrics. On RICH, it leads feed-forward people-and-scene methods on WA-MPJPE and W-MPJPE. The complete pipeline runs at 10.1 fps on one GPU.

View source

Similar papers

Preprint Sep 2026

VI3: Grounding Pretrained 3D Foundation Models with Inertial Cues

VI3, a model-agnostic framework that metrically anchors a pretrained 3DFM using only IMU readings, is introduced, acting as a fine refinement under well-conditioned motion and as a strong prior when motion is less informative.

E. Lozano, Alberto Jaenal, Javier Civera · 2 citations
#machine learning Preprint Oct 2026

Zebra3D: Efficient Hybrid Foundation Models For Scalable 3D Reconstruction

Zebra3D, a layerwise sparse-linear hybrid attention framework that improves the efficiency of pretrained 3D foundation models without training from scratch, achieves strong reconstruction and camera-pose accuracy at high attention sparsity, with training-free variants remaining close to their dense counterparts.

Utkarsh Saxena, Rui-Jing Yang, Mehdi Rezagholizadeh et al. · 0 citations
Sep 2026

OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute

Feed-forward 3D reconstruction models have achieved impressive performance by scaling model and dataset size, but their cost excludes most research groups and precludes edge deployment. Additionally, generating 3D supervision without sensors still relies on slow, unreliable Structure-from-Motion, as the community lacks...

Brandon Leblanc, Charalambos Poullis · 0 citations
Preprint Aug 2026

UniQuery4R: Unified 4D Scene Reconstruction from a Single Query

UniQuery4R is presented, a query-conditioned framework that encodes a multi-frame clip once and selects the source view, target view, and continuous source-image coordinate only at decoding time via source-to-target cross-attention, and introduces a direction-magnitude parameterization of scene flow with separate super...

Tiancheng Chen, Sheng Tang, Wenhua Jin et al. · 1 citation
Preprint Oct 2026

3DROID: A Renderable 3D Gaussian Dataset with Measured Per-Scene Reliability

This work studies the effect of extrinsic reliability and pose conditioning on feed-forward 3DGS, and proposes a calibration-aware pipeline that anchors reconstructed scenes to the robot's metric workspace and presents a renderable, metric-pose-anchored dataset with scene-level reliability information for robot manipul...

W. Cho, Junhoo Lee, N. Kwak · 0 citations
Preprint Oct 2026

PAGER: Partial-to-global Alignment via Geometric and Relational Distillation

Pretrained 3D encoders are typically developed on globally reconstructed scenes expressed in a consistent world coordinate frame, whereas embodied systems must reason from partial, viewpoint-dependent observations in camera coordinates. We show that this shift from globally learned 3D feature spaces to realistic partia...

Akira-Miranda Adeyomi Adeniran-Lowe, B. Singh, Lars Arnold Dethlefsen et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.