Referring video segmentation with heterogeneous multimodal queries---spanning text, audio, and reference images---demands both robust cross-modal understanding and precise spatio-temporal reasoning. We propose OPERA (Omnimodal Progressive spatio-tEmporal Reasoning Agent), a unified reasoning agent built on a single MLLM that performs dual-axis progressive reasoning via three specialized stages. Along the temporal axis, a Temporal Reasoning Agent narrows the frame search space through coarse-to-fine filtering to identify the most informative key frame. Along the spatial axis, a Distillation Agent establishes what to locate via cross-modal semantic distillation, and a Grounding Agent enhanced with GRPO determines where the target appears, with dense mask propagation completing the pixel-level output. OPERA sets a new state of the art on OmniAVS and Ref-AVS and transfers zero-shot to standard referring video segmentation benchmarks.
Embodied agents must reason about 3D space while the video is still arriving, answering questions as soon as they have observed enough of the scene. VLMs that incorporate 3D geometric priors achieve strong spatial reasoning, but they operate offline, i.e., the full video must be available before they produce an answer....
Hai-Rong Yin, Huang-Ying Zhan, Shin-Fang Chng et al.· 0 citations
Long-video understanding places substantial demands on memory, as answering questions often requires retrieving information distributed across extended temporal spans. Existing approaches broadly follow two paradigms: query-driven exploration, which is sensitive to localization errors, and query-independent memory cons...
Yu-Cheng Liu, Yu-Fei Yin, Ming-Xiao Feng et al.· 0 citations
VCN-Bench is introduced, a VCN-Bench benchmark for probing closed-loop spatial reasoning over prior visual experience in MLLMs, and MV-DualVLN is proposed, a planning-oriented baseline that jointly leverages prior video and in-episode observations.
Si-Qi Zhang, Meng Wei, Chen-Yang Wan et al.· 0 citations
In the era of big data, precise event retrieval from large video collections has become increasingly important and challenging. In particular, locating multiple events from the same video in their temporal order is difficult when different events are described by different types of evidence, such as visual content, on-...
COMET is a temporally grounded framework that systematically strengthens video MLLMs through explicit temporal representation, appearance-motion fusion, and direction-aware optimization and achieves consistent overall improvements with a pronounced motion-temporal bias.
Cheng-Hua Zhu, Zhaolu Kang, Qifan Shi et al.· 2 citations
StreamPI is proposed, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters and seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference.
Zhe Liu, Jing-Hua Hou, Yuxiang Lu et al.· 4 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
