ViBe, a post-training framework for adapting motion trackers to perceptive control tasks, uses pre-trained visual encoders with a multi-query extractor module to learn task-relevant perceptive feedback and solves a goal-oriented Repose Cube task with a deliberately simple planner, demonstrating the efficacy of perceptive controllers, adapted by this approach.
Abstract
Motion tracking provides a scalable recipe for humanoid whole-body control. By design, the resulting trackers lack exteroceptive feedback hence reacting to the environment remains the responsibility of a higher-level planner. Existing perceptive controllers train geometry-only encoders from scratch, trading semantics for sim-to-real ease, and typically rely on teacher-student distillation for a task of interest. We present ViBe, a post-training framework for adapting motion trackers to perceptive control tasks. We leverage pre-trained visual encoders with a multi-query extractor module to learn task-relevant perceptive feedback. This feedback is grafted onto the tracker's input via low-rank adapters, enabling parameter-efficient fine-tuning. Given a task reward and a reference dataset, this modular controller can be adapted directly via policy optimization. Across four tasks, ViBe shows zero-shot sim-to-real transfer spanning perceptive walking on curbs and parkour, Repose Cube, omni-object loco-manipulation, and dodgeball, with visually robust performance across outdoor, low-light, and RGB distractor conditions. Finally, we solve a goal-oriented Repose Cube task with a deliberately simple planner, demonstrating the efficacy of perceptive controllers, adapted by our approach.
TRACE first injects kinematic priors into input and output projection layers and aligns nominal joint dynamics to reduce embodiment mismatch and support stable fine-tuning, then combines encoder distillation with low-rank reinforcement learning on a compact, quality-screened motion set.
Qing-Yu Lan, Zhi-Hao Tian, Zi-Fan Wang et al.· 0 citations
PASSAGE, a perception-conditioned planner--tracker framework for humanoid traversal, using virtual reality and inertial motion capture to collect 100 h of scene-aligned human motion across 1,500 cluttered scenes and tests across 50 unseen physical layouts demonstrate traversal without prebuilt maps or offboard computat...
General purpose humanoids require locomotion controllers that are multi-skill, perceptive, dynamic, and robust enough to go anywhere humans can. In this work, we present a two layer locomotion architecture: (1) a perceptive flow matching motion generator plans whole body trajectories from raw depth images while a (2) p...
Zachary Olkin, William D. Compton, Aaron D. Ames· 0 citations
EgoAlign, a data-construction framework that converts these demonstrations into action and state supervision compatible with a general-purpose, continuous whole-body controller, without collecting physical-robot demonstrations, is presented.
Yi-Ming Jiang, Jin Chen, Chong-Yang Xu et al.· arXiv.org· 0 citations
To the knowledge, this work provides the first demonstration of real-time cross-terrain whole-body teleoperation on a humanoid robot, alongside object interaction, stable responses to missing supports, and robust recovery from falls.
Zi-Yang Cheng, Tian-Shu Tang, Jin-Xi Lan et al.· 0 citations
A unified four-step post-training pipeline comprising a learned temporal hand-action codec, supervised fine-tuning, DAgger, and real-world residual reinforcement learning provides a practical path for adapting VLA foundation models to reliable real-world dexterous manipulation.
Jun-Lei Zhu, Shen-Zhe Yao, Chao-Gui Huang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.