Skip to content
Book Open access

SAGE: Execution-Grounded Adaptive Planning for Egocentric Interactive Agents

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 67 references

TL;DR

This work introduces \method, a Status-Aware Grounded Execution framework, a Status-Aware Grounded Execution framework that maintains a unified task state, executes currently supported steps grounded in visual and tool evidence, and replans unresolved requirements before branch actions.

Abstract

Egocentric interactive agents must translate first-person observations and constrained user requests into reliable tool-mediated actions. ReAct acquires branch-specific evidence but may over-rely on local observations; Plan-and-Solve decomposes tasks globally but can linearize mutually exclusive branches. We introduce \method, a Status-Aware Grounded Execution framework. \method maintains a unified task state, executes currently supported steps grounded in visual and tool evidence, and replans unresolved requirements before branch actions. The tool branch uses compact planner views, step-specific schemas, and Reporter summaries, while the visual branch applies Staggered Pair Grounding to pair hand-indicated targets with less-occluded temporal evidence. Without task-specific training, \method achieves 78.13% joint success (25/32) on retail1, outperforming the official Qwen3.5-397B-A17B baseline by 34.38 percentage points. It reaches 81.01% ground-truth tool-call recall, compared with 75.95% for ReAct and 69.62% for Plan-and-Solve.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.