Skip to content
Preprint

KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation

Sep 2026 · 1 citation · 39 references
Computer Science

TL;DR

This work proposes a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control and introduces a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rate.

Abstract

Humanoid loco-manipulation requires robots to interpret task instructions and scene semantics while executing coordinated whole-body motions. We propose a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control. Each keyframe specifies a target whole-body robot pose and, when applicable, an object pose. Given a language instruction, scene observations, and execution feedback, the VLM selects successive task-relevant keyframes from a predefined library. The selected keyframes are retargeted to the current scene to account for object poses and dimensions. A keyframe-conditioned whole-body policy then generates joint-level actions to reach these goals. We introduce a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rate from 44% to 92% when using sparse VLM keyframes. We evaluate our framework on object pickup, transport, and placement tasks in simulation and on a Unitree G1 humanoid. The system successfully performs both one- and two-handed manipulation and generalises to placement locations beyond the training reference data.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

HiWE: Hierarchical World Knowledge Model with Visual Keypoint Enhancement for Zero-Shot 3D Path Planning

This paper describes the original point-based formulation of the framework and its relationship to the subsequent GeneralVLA extension, and describes the original point-based formulation of the framework's relationship to the subsequent GeneralVLA extension.

Guo-Qing Ma, Ming-Qi Yuan, Chen Gao et al. · 0 citations
Sep 2026

H-VLA: Hierarchical Vision-Language-Action Model with Key-Action Reasoning and Motion Planning in a Unified Action Space

H-VLA is proposed, a hierarchical VLA framework that decouples high-level key-action reasoning from low-level motion generation and adopts a two-stage training strategy that emphasizes key-action reasoning during pre-training and dense motion generation during fine-tuning.

Xiong-Feng Peng, Lu Xu, Yan-Dong Wang et al. · 0 citations

EgoHumanoid-V2: Human-to-Humanoid Transfer of Coordinated Whole-Body Skills for Loco-Manipulation

EgoHumanoid-V2 is presented, the first egocentric human-to-humanoid skill transfer framework for coordinated whole-body loco-manipulation, which improves end-effector pose accuracy while preserving whole-body coordination and uses robot-arm rendering and training-time image augmentation to reduce the visual embodiment...

Jin Chen, Yi-Ming Jiang, Chong-Yang Xu et al. · 2 citations
Sep 2026

Opt2VLA: Force-Aware Vision-Language-Action for Contact-Rich Humanoid Whole-Body Manipulation

Opt2VLA is presented, a force-aware VLA framework that introduces explicit force commands at the VLA-to-control interface for humanoid whole-body manipulation and shows that explicit force conditioning enables more accurate and consistent force regulation than motion-only control, while physically grounded torque super...

Fu-Kang Liu, Yipu Chen, Jaehwi Jang et al. · 1 citation
Preprint Aug 2026

HAF: Adapting Generalist VLAs to Humanoid Whole-Body Loco-manipulation via Hierarchical Action Flow and Spectral Latent RL

HAF (Humanoid Adaptation Framework), a two-part framework consisting of HAF-VLA and HAF-Steer that transfers off-the-shelf generalist VLA foundation models to humanoid whole-body loco-manipulation, surpasses vanilla single-stage VLA baselines and improves whole-body coordination and task performance.

Lang-Zhe Gu, Cheng-Kai Hou, Meng Li et al. · 3 citations
Preprint Sep 2026

Humanoid Loco-Manipulation With Discrete VLA Model

Vision-language-action (VLA) models using discrete action tokens have proven effective for controling robotic arms on manipulation tasks. For a humanoid, however, the whole-body action space -- legs, torso, arms, and hands -- is far higher-dimensional and heterogeneous, raising tokenization, training, and real-time inf...

Wen-Xin Shao, Si-Qi Chai, Kun Li et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.