Skip to content
Preprint

RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents

Sep 2026 · 0 citations · 50 references
Computer Science

TL;DR

This work introduces RoboFollow, a diagnostic benchmark with three principles, which exposes genuine instruction following as a critical, overlooked bottleneck in embodied agents' ability to follow instructions.

Abstract

Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest. We trace this illusion to a structural property we term low scene entropy: when a visual scene admits only one valid task, language becomes redundant and a policy can score highly while barely using it. We introduce RoboFollow, a diagnostic benchmark with three principles: (1) High Scene Entropy: each training scene supports multiple kinematically distinct task branches, making vision alone insufficient and forcing reliance on language. (2) Hierarchical Diagnostic Protocol: a four-level protocol (L0--L3) progressively perturbs visual layout and semantics, probing whether equivalent instructions yield consistent behavior and distinct ones yield discriminable behavior across spatial relations, attributes, trajectory constraints, and logic. (3) Confound-Controlled Diagnosis: we simplify interaction objects, restrict actions to the trained repertoire and report stage-wise Intent and Execution scores, isolating comprehension from motor execution. Evaluation of nine VLA and WAM policies shows that strong L0 performance, where attained, does not reliably transfer to L1--L3 under our fine-tuning setup. Representative mitigations, including stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance, all fail to close this gap. RoboFollow exposes genuine instruction following as a critical, overlooked bottleneck. Code and dataset are available at https://github.com/AutoLab-SAI-SJTU/RoboFollow and https://huggingface.co/datasets/AutoLab-SJTU/robofollow-data.

View source

Similar papers

#machine learning Preprint Oct 2026

Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies

Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through sc...

Shao-Han Jiang, Jia-Hang Cao, Qi-Duo He et al. · 1 citation
#artificial intelligence Preprint Aug 2026

Neurosymbolic Embodied Agents

A neurosymbolic agent that factors long-horizon household tasks into task-directed visual exploration and constrained symbolic planning and evaluates executable continuations using a domain-independent planning heuristic is presented.

Mohammad Albinhassan, Yuming Feng, Alessandra Russo et al. · 0 citations

When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models

The analysis of the imitation objective shows how narrow conditional action support can leave grounded and instruction-keyed solutions indistinguishable on the demonstrations, and motivates Equivariant Counterfactual Training (ECT), which acts at two levels.

Hung-Jen Chen, Yue-Ling Hou, Yan-Hong Chen et al. · 1 citation
Preprint Sep 2026

RoboChrono: A Real Robot Benchmark for Streaming Task Understanding

Understanding ongoing robot manipulation requires models to interpret visual observations in relation to interaction history and task progress. We introduce RoboChrono, a benchmark for streaming task understanding comprising 39 scenarios and 34,713 evaluation instances, constructed from real robot executions and comple...

Yu-Zhou Wu, Long-Teng Fan, Zi-Meng Li et al. · 0 citations
Preprint Aug 2026

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

It is argued that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction.

Mengao Zhao, Ziang Li, Chaodong Huang et al. · 2 citations
Preprint Sep 2026

In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks

We study robotic in-context learning (ICL), an emerging paradigm that enables robots to infer and execute tasks from visual demonstrations. Despite its growing promise, the problem itself remains under-defined: a visual demonstration simultaneously conveys action trajectories, object semantics, manipulation affordances...

Min-Xing Li, Ming-Hao Han, Wei-Zhi Zhao et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.