Skip to content
Preprint

LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds

Nov 2026 · 0 citations · 61 references
Computer Science

TL;DR

A Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining and a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining is introduced.

Abstract

Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/

View source

Similar papers

#artificial intelligence Preprint Oct 2026

WorldSonus: Bringing Sound to Worlds

WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models, generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment.

Peng-Jun Fang, Jing-Yi Fa, Kam-Man Wu et al. · 1 citation · ⚡1
Preprint Nov 2026

LiveView: Real-Time Camera-Controlled Re-filming of Streaming Videos

Real-time camera-controlled video re-filming aims to synthesize target views from a streaming source video under online user-specified camera trajectories. This task poses two key challenges: maintaining synchronization and content consistency across views, and achieving low-latency rendering with long-horizon stabilit...

You-Can Xu, Xin-Jie Huang, Jia-Xin Shi et al. · 0 citations
Sep 2026

HelixWorld: A Real-time Interactive Audio-Visual World Model

World simulation is inherently multisensory, demanding synchronized visual and acoustic dynamics in real time. Yet prevailing interactive world models remain strictly silent, focusing exclusively on visual rendering and control while overlooking the acoustic dimension. We present HelixWorld, a real-time interactive aud...

Lei Ke, Jia-Hao Pan, Ze-Yue Tian et al. · 1 citation
Preprint Oct 2026

StreamErase: Towards Ultra-Fast Streaming Video Erasing

Current video object removal methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present StreamErase, enabling sub-second latency with up to 26 FPS streaming generation on a single GPU. Our approach first augments a text-to-video model with mask...

Zhuo-Bai Dong, Xiao-Bing Tu, Xian-Tao Zhang et al. · 0 citations
Open access Oct 2026

Hear What You See: Depth-Aware Spatial Audio Rendering for Immersive Panoramic Video.

Panoramic video has emerged as a core medium for virtual reality (VR) and immersive media, yet its audio typically remains stereo, lacking spatially coherent soundfield reproduction consistent with the visual content and severely compromising immersion. Although multi-loudspeaker spatial audio systems can reconstruct r...

Di Zhang, Zi-Shan Peng, Yi-Nuo Du et al. · 0 citations
Sep 2026

Enabling Immersive Audio-Visual Experience from Any Video

Most videos capture only a narrow field of view and provide no spatial audio, limiting the sense of immersion they can provide. Recent video generation models can expand perspective videos into panoramic ones, but do not provide the corresponding spatial soundscape. Without spatially consistent audio, these expanded vi...

Zitong Lan, Mu-Tian Tong, Jia-Tao Gu et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.