Skip to content

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation

Sep 2026 · arXiv.org · Vol abs/2609.28955 · 0 citations · 40 references
Computer Science

TL;DR

ActGaze is a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements, and consistently outperforms the base VLA policy and other visual-grounding approaches.

Abstract

Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.

View source

Similar papers

Sep 2026

PSR: Predictive Sensorimotor Representation Learning for Contact-Rich Manipulation

Predictive Sensorimotor Representation learning is introduced, a framework that learns a hierarchy of predictive representations from multimodal sensorimotor signals and integrates them into the action stream of a visuomotor policy, and is instantiate within a Vision-Language-Action model, resulting in PSR-VLA.

Sheng-Bao Li, Peng Xu, Chao Tang et al. · 0 citations
Preprint Sep 2026

From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation

This work introduces the first training-free Gaze Target Agent (GTA) for gaze-guided reasoning across tasks such as gaze target prediction, attention localization, and object identification by leveraging pretrained vision-language models, augmenting them with visually guided prompts, and employing a memory-based retrie...

Shayan Nasiriboukani, Sara Atito, Mohammad Nezamipour et al. · 0 citations

Learning to Act under Visual Interruptions with Vision-Language-Action Models

MINT is proposed, which first trains VLA policies to remain functional under missing visual inputs, and selectively supplements missing observations using optical-flow extrapolation or an action-conditioned world model, and withdraws predicted views when they become unreliable.

Ming Jiang, Rui Xu, Yun-Ke Wang et al. · 0 citations
Preprint Sep 2026

HINT: Human-Intent Inception for Long-Horizon Robot Manipulation

HINT (Human-INTent INcepTion), an agentic framework inspired by the human manipulation principles: semantic intent changes sparsely at manipulation-pattern transitions, whereas continuous control primarily depends on the evolving object-hand relationship.

Ming-Yu Mei, Haojie Xu, Shi-Hao Jin et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.