"Great! The Next Step Is...": In Pursuit of Proactive Assistance with Multimodal Foundation Models

Abstract

Large language and multimodal models have been largely employed to reactively generate content and assistance. In contrast, we explore how models can be employed to provide proactive assistance that fluidly supports ongoing human activities. We identify and tackle challenges in such streaming scenarios—where models must act continuously and responsively in real time, rather than waiting for explicit user requests. We explore solutions and evaluation metrics in the context of a mixed-reality AI agent designed to assist people in carrying out procedural physical tasks. Specifically, we focus on the challenge of identifying when each step of a procedure has been completed so that an agent can appropriately move on to the next step. Using a publicly available multimodal dataset of task-centric human-agent interactions, we study an array of prompting strategies across several frontier LLMs and VLMs, considering both the content and timing of model queries. Our findings demonstrate the criticality of developing application-driven metrics that enable meaningful comparisons and highlight key gaps in current model capabilities for proactive assistance. This work underscores the importance of temporally-aware evaluation in designing fluid real-time human-AI collaboration.