ActGaze is a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements, and consistently outperforms the base VLA policy and other visual-grounding approaches.
Abstract
Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.
eye-tracker-supervised gaze prompting is introduced, which uses gaze recorded during VR teleoperation to provide frame-level visual guidance for VLA fine-tuning.
Yi-Han Zhou, Rui Yan, Ming-Cong Li et al.· arXiv.org· 0 citations
Predictive Sensorimotor Representation learning is introduced, a framework that learns a hierarchy of predictive representations from multimodal sensorimotor signals and integrates them into the action stream of a visuomotor policy, and is instantiate within a Vision-Language-Action model, resulting in PSR-VLA.
This work introduces the first training-free Gaze Target Agent (GTA) for gaze-guided reasoning across tasks such as gaze target prediction, attention localization, and object identification by leveraging pretrained vision-language models, augmenting them with visually guided prompts, and employing a memory-based retrie...
Shayan Nasiriboukani, Sara Atito, Mohammad Nezamipour et al.· 0 citations
MINT is proposed, which first trains VLA policies to remain functional under missing visual inputs, and selectively supplements missing observations using optical-flow extrapolation or an action-conditioned world model, and withdraws predicted views when they become unreliable.
Ming Jiang, Rui Xu, Yun-Ke Wang et al.· arXiv.org· 0 citations
HINT (Human-INTent INcepTion), an agentic framework inspired by the human manipulation principles: semantic intent changes sparsely at manipulation-pattern transitions, whereas continuous control primarily depends on the evolving object-hand relationship.
Ming-Yu Mei, Haojie Xu, Shi-Hao Jin et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.