Skip to content

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

Jun 2026 · arXiv.org · Vol abs/2606.14777 · 10 citations · 53 references
Computer Science

TL;DR

Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin, and the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.

Abstract

Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.

View source

Similar papers

#generative ai Book Open access Oct 2026

Can You See Me, GenAI? Characterizing Video-Based Conversational Interaction with Generative AI

The authors' measurements reveal several fundamental bottlenecks: conversational latency remains above natural turn-taking thresholds even in unconstrained networks; protocol choices significantly affect system robustness under packet loss; and conventional video streaming practices, such as prioritizing stable frame r...

Rui-Zhi Cheng, Guo-Wu Xie, Bo Han · 1 citation
Preprint Oct 2026

InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems

A video assistant must speak when its instruction warrants a response and stay silent otherwise. We introduce a benchmark that evaluates this decision for the complete system of model, memory, and response controller. InteractionBench covers query responses, event triggers, and ongoing updates in 1,060 interactions ove...

En-Xin Song, Su-Hao Yu, Yi-Fei Xu et al. · 0 citations
#machine learning Preprint Sep 2026

LiveProBench: Can Streaming Video Models Really Interact Like Humans?

LiveProBench is introduced, which evaluates models at one-second stream intervals without an explicit response cue, revealing a substantial gap in the temporal decision-making required for human-like interaction.

Kai-Xuan Du, Xin Wan, Hang Zhang et al. · 3 citations · ⚡1

AVTR-1: Open Stack for Real-Time Interactive Avatars

Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live system must synchronize the model's output with speech from an external voice agent, schedule video frames for playback, and handle interruptions. We introduce AVTR-1, an...

A. Kravtsov, Dmitrii Ziganshin, V. Poletaev et al. · 1 citation
#generative ai Book Open access Oct 2026

Huni: A Networked Testbed for Real-time Human and Generative AI Interaction

Hanni, a networked testbed for real-time human-GenAI interaction, is presented and the usability of Huni is demonstrated through two case studies: comparing call setup time across different WebRTC and QUIC-based schemes, and evaluating the performance of multiple congestion control algorithms under diverse network cond...

Rui-Zhi Cheng, Mohamed Mohsin Shaik, Guo-Wu Xie et al. · 1 citation
Preprint Sep 2026

MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs

MotionBlind, a contrastive benchmark of self-recorded video for physically grounded motion, the variables a world model must predict, is introduced, a pair of near-identical clips that differ only in motion.

Dhairya Bhatia, Bishoy M. Galoaa, Oliver Fritsche et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.