Can You See Me, GenAI? Characterizing Video-Based Conversational Interaction with Generative AI
TL;DR
The authors' measurements reveal several fundamental bottlenecks: conversational latency remains above natural turn-taking thresholds even in unconstrained networks; protocol choices significantly affect system robustness under packet loss; and conventional video streaming practices, such as prioritizing stable frame rates, often degrade VLM inference accuracy.
Abstract
Video-based conversational interaction between humans and generative artificial intelligence (GenAI) represents a rapidly emerging paradigm at the intersection of networking, multimedia communication, and vision language models (VLMs). Applications such as ChatGPT and Gemini enable users to point their smartphones at objects/scenes and receive contextual responses, opening new possibilities for on-the-go visual understanding. However, despite their growing adoption, the underlying performance dynamics and system behavior of these applications remain poorly understood. In this paper, we present the first systematic measurement study of video-based human-GenAI interaction across five representative applications with 11 distinct variants. We develop a custom toolset to characterize these applications across diverse network conditions and interaction scenarios, exposing how design decisions in video delivery and transport protocols shape user-perceived experience. Our measurements reveal several fundamental bottlenecks: conversational latency remains above natural turn-taking thresholds even in unconstrained networks; protocol choices significantly affect system robustness under packet loss; and conventional video streaming practices, such as prioritizing stable frame rates, often degrade VLM inference accuracy. Our findings provide an empirical foundation for designing and optimizing next-generation real-time human-AI interaction.
