SAGE: Execution-Grounded Adaptive Planning for Egocentric Interactive Agents
TL;DR
This work introduces \method, a Status-Aware Grounded Execution framework, a Status-Aware Grounded Execution framework that maintains a unified task state, executes currently supported steps grounded in visual and tool evidence, and replans unresolved requirements before branch actions.
Abstract
Egocentric interactive agents must translate first-person observations and constrained user requests into reliable tool-mediated actions. ReAct acquires branch-specific evidence but may over-rely on local observations; Plan-and-Solve decomposes tasks globally but can linearize mutually exclusive branches. We introduce \method, a Status-Aware Grounded Execution framework. \method maintains a unified task state, executes currently supported steps grounded in visual and tool evidence, and replans unresolved requirements before branch actions. The tool branch uses compact planner views, step-specific schemas, and Reporter summaries, while the visual branch applies Staggered Pair Grounding to pair hand-indicated targets with less-occluded temporal evidence. Without task-specific training, \method achieves 78.13% joint success (25/32) on retail1, outperforming the official Qwen3.5-397B-A17B baseline by 34.38 percentage points. It reaches 81.01% ground-truth tool-call recall, compared with 75.95% for ReAct and 69.62% for Plan-and-Solve.