An event-driven framework that monitors workspace state changes with an event monitor and, upon event completion, extracts stabilized pre/post snapshots that characterize the resulting state transition achieves performance comparable to variants given user instructions.
Abstract
Assistance in collaborative manipulation is often initiated by user instructions, making high-level reasoning request-driven. In fluent human teamwork, however, partners often infer the next helpful step from the observed outcome of an action rather than waiting for instructions. Motivated by this, we investigate an event-driven formulation of proactive assistance, where human--object interaction outcomes initiate assistive reasoning without user-provided task specifications at inference time. To this end, we propose an event-driven framework that monitors workspace state changes with an event monitor and, upon event completion, extracts stabilized pre/post snapshots that characterize the resulting state transition. A frozen pretrained Vision-Language Model (VLM) then uses its semantic priors to infer the task context, decide whether assistance is appropriate, and, when needed, generate a sequence of assistive actions from the observed transition. To make outputs executable and verifiable, we restrict actions to a set of action primitives and reference objects via integer IDs. We evaluate the same framework across three distinct real world tabletop collaboration tasks without task-specific training or fine-tuning. The event-driven framework achieves performance comparable to variants given user instructions.
A memory-based agentic framework in which a VLM task predictor reads a Procedural Assistance Memory alongside the current observation to decide when to assist and what task to perform, and a VLA executor performs the predicted task.
Dohyun Lee, Minho Park, Jeonghoon Park et al.· 1 citation
History-dependent robotic manipulation requires task context and interaction history, as the current observation alone is insufficient to determine the next action. Hierarchical frameworks use task memory for high-level subgoal reasoning and low-level policies for execution. However, memory updates require accurate com...
Jing-Jiao Zhao, Dong Jing, Ming-Yu Cao et al.· 0 citations
Robots operating in human-centered environments are typically designed to execute explicit instructions, and most robot-learning datasets likewise pair observations with task instructions or low-level actions. Although recent work has begun to explore proactive embodied assistance, existing resources target different s...
This work introduces a unified formalism for proactive robot assistance, organize it into three levels, and provides a framework to address the highest level of unprompted proactive assistance, and presents a method, GAP, that instantiates the framework, learning from passive observation to anticipate user goals and ac...
Adaptive behavior is a fundamental requirement for social robots operating in real-world environments, where interactions must dynamically respond to both observable actions and inferred user states. In this work, we propose a novel framework that models human–robot interaction as a planning problem, where adaptive con...
Giulia Berettieri, Anna Allegra Bixio, Lucrezia Grassi et al.· Companion Publication of the...· 0 citations
Robot policies increasingly incorporate semantic reasoning and future-world prediction, yet combining these capabilities does not guarantee that local predictions and actions remain aligned with task progress. We introduce CogWAM, a cognition-guided world-action model that establishes an explicit semantic interface bet...
Sen Wang, Liu Liu, Xin-Jiang Wang et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.