Ask-to-Act: Learning when Robots Should Clarify Ambiguous Grounded Instructions

Abstract

Robots that follow natural-language instructions must decide not only what action to execute, but also whether the current evidence is sufficient to act at all. We present Askto-Act, a framework for grounded robot instruction following that treats clarification as a cost-sensitive decision under uncertainty. The system either executes immediately or requests one minimal grounding cue before acting. We evaluate Ask-to-Act on SCOUT++, a grounded HRI benchmark with commandlabel-image examples from situated human-robot interaction. In a text-only setting, Ask-to-Act improves grounded action prediction from 0.8225 to 0.8308 accuracy while asking in 0.2350 of episodes. In a multimodal setting, command-time visual grounding improves the always-act baseline to 0.8503 accuracy, and Ask-to-Act further improves accuracy to 0.8545 while asking in 0.1277 of episodes. These results show that Askto-Act can adapt its interaction rate to the available evidence. It asks more often when language alone leaves action choices underdetermined and asks less often when visual grounding strengthens the base action predictor. To analyze this behavior, we define ask-beneficial ambiguity as cases where the gold action is in the model's top-k candidates but not ranked first, and evaluate uncertainty-based ambiguity detectors with AUROC and AUPRC. Overall, Ask-to-Act positions clarification as an adaptive component of grounded robot learning rather than a fixed requirement of language interaction.