This work proposes a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control and introduces a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rate.
Abstract
Humanoid loco-manipulation requires robots to interpret task instructions and scene semantics while executing coordinated whole-body motions. We propose a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control. Each keyframe specifies a target whole-body robot pose and, when applicable, an object pose. Given a language instruction, scene observations, and execution feedback, the VLM selects successive task-relevant keyframes from a predefined library. The selected keyframes are retargeted to the current scene to account for object poses and dimensions. A keyframe-conditioned whole-body policy then generates joint-level actions to reach these goals. We introduce a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rate from 44% to 92% when using sparse VLM keyframes. We evaluate our framework on object pickup, transport, and placement tasks in simulation and on a Unitree G1 humanoid. The system successfully performs both one- and two-handed manipulation and generalises to placement locations beyond the training reference data.
This paper describes the original point-based formulation of the framework and its relationship to the subsequent GeneralVLA extension, and describes the original point-based formulation of the framework's relationship to the subsequent GeneralVLA extension.
Guo-Qing Ma, Ming-Qi Yuan, Chen Gao et al.· 0 citations
H-VLA is proposed, a hierarchical VLA framework that decouples high-level key-action reasoning from low-level motion generation and adopts a two-stage training strategy that emphasizes key-action reasoning during pre-training and dense motion generation during fine-tuning.
Xiong-Feng Peng, Lu Xu, Yan-Dong Wang et al.· arXiv.org· 0 citations
EgoHumanoid-V2 is presented, the first egocentric human-to-humanoid skill transfer framework for coordinated whole-body loco-manipulation, which improves end-effector pose accuracy while preserving whole-body coordination and uses robot-arm rendering and training-time image augmentation to reduce the visual embodiment...
Jin Chen, Yi-Ming Jiang, Chong-Yang Xu et al.· arXiv.org· 2 citations
Opt2VLA is presented, a force-aware VLA framework that introduces explicit force commands at the VLA-to-control interface for humanoid whole-body manipulation and shows that explicit force conditioning enables more accurate and consistent force regulation than motion-only control, while physically grounded torque super...
Fu-Kang Liu, Yipu Chen, Jaehwi Jang et al.· arXiv.org· 1 citation
HAF (Humanoid Adaptation Framework), a two-part framework consisting of HAF-VLA and HAF-Steer that transfers off-the-shelf generalist VLA foundation models to humanoid whole-body loco-manipulation, surpasses vanilla single-stage VLA baselines and improves whole-body coordination and task performance.
Lang-Zhe Gu, Cheng-Kai Hou, Meng Li et al.· 3 citations
Vision-language-action (VLA) models using discrete action tokens have proven effective for controling robotic arms on manipulation tasks. For a humanoid, however, the whole-body action space -- legs, torso, arms, and hands -- is far higher-dimensional and heterogeneous, raising tokenization, training, and real-time inf...
Wen-Xin Shao, Si-Qi Chai, Kun Li et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.