Skip to content

HiWE: Hierarchical World Knowledge Model with Visual Keypoint Enhancement for Zero-Shot 3D Path Planning

Sep 2026 · 0 citations · 21 references
Computer Science

TL;DR

This paper describes the original point-based formulation of the framework and its relationship to the subsequent GeneralVLA extension, and describes the original point-based formulation of the framework's relationship to the subsequent GeneralVLA extension.

Abstract

Robot demonstration generation requires a system to identify where an interaction should occur, plan a feasible motion, and execute the required contact. HiWE connects these decisions through a point-based interface between visual grounding and language-based planning. PointVLM is instruction-tuned to associate task-relevant objects with image coordinates using a mixture of point annotations, segmentation-derived samples, robot observations, and visual question answering data. Depth measurements lift these predictions into a semantic 3D representation. A language planner, 3DLLM, uses this representation to specify end-effector waypoints and gripper commands, while a hybrid grasping module resolves local grasp poses. The evaluation covers 14 simulated manipulation tasks and four physical-robot tasks, together with ablations of the visual training data, spatial inputs, and grasp selection. Here, zero-shot execution refers to deployment without task-specific demonstration training; the visual model uses existing robot data during fine-tuning. This paper describes the original point-based formulation of the framework; its relationship to the subsequent GeneralVLA extension is detailed in the introduction.

View source

Similar papers

Preprint Sep 2026

KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation

This work proposes a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control and introduces a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rat...

Si-Tong Chen, Fatemeh Zargarbashi, Jin Cheng et al. · 1 citation
Preprint Oct 2026

GRASP: Global-Local Representation Alignment and Spatial-Physical Grounding for Bimanual Robotic Manipulation

Vision-Language-Action (VLA) models that directly regress joint-space actions avoid the deployment-time cost of inverse kinematics. However, their training objectives remain misaligned with task success, as joint-space losses ignore the unequal physical consequences of joint errors and visual conditioning lacks explici...

Ze-Yuan Ren, Guan-Xun Zhao, Qi Li et al. · 0 citations
#small language model Preprint Sep 2026

FRAM: Trajectory-Guided Visual Feature Selection for Compact Language-Conditioned Robot Manipulation

This work proposes the Future Representation Action Model (FRAM), a small policy that explicitly links the future end-effector trajectory to the current visual input and selects visual information based on future motion to obtain both high performance and robustness in a small robot policy.

Hiroshi Ito, Hyogo Hiruma, Yoshiki Kanai et al. · 0 citations
Nov 2026

CAD2Real: CAD-Grounded Skill Primitives and Vision-Language Planning for Precision Assembly

Putting together a multi-part assembly feels routine in human hands: we visually align parts, feel first contact, correct tiny misalignments, and decide what to do next from the evolving scene. For a robot, precision assembly is a long-horizon, contact-rich problem where millimeter-level pose errors or a mistimed actio...

Wen-Long Ma, Cheng-Long Li, Lin Wang et al. · 0 citations
Preprint Sep 2026

Grounded Action Model: 3D Grounding as a Foundation for Robotics

Grounded Action Models (GAMs), a new paradigm of robot foundation models built with 3D grounding, are proposed, which can be run autonomously and serve as a low-level controller that a high-level planner controls using its various input modalities, allowing for long-horizon and memory-dependent manipulation.

Ge-Hao Zhang, Weikai Huang, S. Shailesh et al. · 0 citations
Preprint Oct 2026

UNITAS: A 3D-Native World Action Model for Embodied Manipulation

World action models (WAMs) aim to answer a coupled physical question: given a task instruction, what motion should the robot execute, and how will that motion change the surrounding world? Most existing WAMs build on pretrained video generators and represent world evolution through images or visual latents. Robotic int...

Rui-Xiang Wang, Yong-Yi Su, Wen-Lve Zhou et al. · 2 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.