The Model in the Middle: Toward AI-Native Real-Time Communication
TL;DR
The model is a stateful computational middlebox inside a human-centered feedback loop, with network transport, model serving, and user playback jointly shaping how the interaction evolves, and is called for an AI-native real-time communication stack that resolve the joint control problem spanning communication, computation, and playback.
Abstract
Full-duplex omni models are transforming human--AI interaction from turn-based exchanges into continuous multimodal conversations in which speaking, listening, and reasoning unfold concurrently. Rather than viewing the model as a replacement for a human endpoint, we argue for a new perspective: the model is a stateful computational middlebox inside a human-centered feedback loop, with network transport, model serving, and user playback jointly shaping how the interaction evolves. This perspective breaks the traditional boundaries among stages designed around local objectives. Rather than optimizing them in isolation, an AI-native real-time stack should allow the state of each stage to shape the actions of the others. We explore three cross-stage coordination opportunities: network-aware inference scheduling, execution-aware transport prioritization, and playback control that accounts for both network and model variability. We are building Conflux to explore these ideas, and preliminary results show substantial improvements in response latency and playback deadline adherence under network degradation. More broadly, we call for an AI-native real-time communication stack that resolve the joint control problem spanning communication, computation, and playback.