LiveProBench is introduced, which evaluates models at one-second stream intervals without an explicit response cue, revealing a substantial gap in the temporal decision-making required for human-like interaction.
Abstract
Streaming video understanding requires models to process continuous multimodal input while maintaining temporal context. Existing evaluations are predominantly reactive: they query a model at a selected timestamp and therefore do not assess when it should respond. Proactive interaction instead requires monitoring a standing request, responding within an appropriate interval after the target event, and otherwise remaining silent. We introduce LiveProBench, which evaluates models at one-second stream intervals without an explicit response cue. Its six subtasks vary trigger ambiguity and timing tolerance. Event Sensitivity geometrically combines response and silence rates on the same recording; four window-based subtasks distinguish early, in-window, and missed responses; and Duplicate Counting penalizes omissions and repetitions. Premature responses outnumber missed responses for half of the evaluated models, revealing a substantial gap in the temporal decision-making required for human-like interaction.
StreamMind is introduced, a two-tier architecture that assigns latency-critical interaction and proactive monitoring to independently scheduled frontend workers, while backend workers asynchronously construct persistent multimodal memory and perform historical recall and external search.
Xichen Zhang, Guankai Li, Yinghao Zhu et al.· 1 citation
VT-Contrast supervises selected late-layer last-frame video tokens, where temporal information is expected to be integrated before language generation, and contrasts order-preserving views with same-video reordered counterfactuals graded by Kendall tau distance.
Yu-Meng Shi, Quan-Yu Long, Yin Wu et al.· 0 citations
InteracVid is introduced, the first open-source large-scale dataset that addresses this missing supervision, so that every sample couples a preceding audio-visual context and an external stimulus with the realinteractive response that follows.
Chi Zhang, Hao-Yan Shi, Yue-Yi Liu et al.· 2 citations
MotionBlind, a contrastive benchmark of self-recorded video for physically grounded motion, the variables a world model must predict, is introduced, a pair of near-identical clips that differ only in motion.
Dhairya Bhatia, Bishoy M. Galoaa, Oliver Fritsche et al.· 0 citations
Temporal reasoning benchmarks for Video-LLMs are often mediated by language, leaving room for linguistic shortcuts from option wording, answer correlations, or language priors, so TempCloze is introduced, a video cloze benchmark for evaluating visual temporal reasoning in Video-LLMs.
Wen-Qi Pei, Henry Hengyuan Zhao, Yi-Lai Liu et al.· 0 citations
Streaming video understanding requires answering questions that arrive at arbitrary moments over an unbounded video stream. Existing systems primarily focus on what to retain in a bounded memory, yet access that memory using the same fixed-cost procedure for every query, despite substantial variation in the evidence re...
Ce Zhang, Jing Bi, Jinxi He et al.· 2 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026