LiteCASS is presented, to the authors' knowledge the first lightweight end-to-end network for real-time stereo CASS, which combines deterministic STFT subband rearrangement with two jointly trained compact U-Nets and achieves the highest averaged SI-SDR among the compared CASS baselines.
Abstract
Cinematic audio source separation (CASS) decomposes a soundtrack into dialogue, music, and sound-effects (SFX) stems. Existing CASS methods, however, suffer from two critical limitations: they rely on heavily parameterized network architectures and GPU-class hardware, limiting their use in real-time and resource-constrained scenarios, and they are overwhelmingly designed for monaural signals, leaving the stereo scenario largely unexplored. We present LiteCASS, to our knowledge the first lightweight end-to-end network for real-time stereo CASS. LiteCASS combines deterministic STFT subband rearrangement with two jointly trained compact U-Nets: the first extracts dialogue, and the second separates music and SFX from the predicted non-speech component. A multi-task waveform-domain L1 loss supervises all stems. On a spatialized stereo extension of DnR v3, LiteCASS-K8 uses only 1.06M parameters and 0.72G MACs per second, while achieving the highest averaged SI-SDR among the compared CASS baselines.
This work presents TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU, and is designed primarily for English and German, with additional multilingual support.
While the stereo format is the most common representation for audio in general, it cannot provide a complete immersive audio experience due to its lack of surround sound. For this reason, multi-channel configurations such as 5.1 or 7.1 are used in cinema or home studios, offering an improved sonic sensation. However, c...
Marian Negru, Bogdan Moroșanu, A. Neacsu· IEEE Transactions on Audio,...· 0 citations
SpectralBridge is presented, a lightweight extension that gives a large audio-language model access to the 8-16 kHz band by joining low- and upper-band representations with time-aligned paired cross-frequency attention and improves both zero-shot transfer and downstream fine-tuning.
Sergei Polezhaev, Saba Anjum, Michael Gormish· 0 citations
Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals. However, existing conversational systems have mainly focused on text or speech, leaving out the visual feedback that is essential for natural interaction. In this paper, we propose...
Junseok Ahn, Jaehyun Park, Jeongsoo Choi et al.· 0 citations
Real-time conversational agents require speaker diarization that streams and runs on a CPU. Most systems apply an utterance-level speaker encoder to short, heavily overlapping chunks, which wastes computation and leaves the model optimized for the wrong task. We instead turn a state-of-the-art speaker recognition archi...
We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. It clones voices from multiple references, follows natural-language direction, revises a take when the next tur...
Da-Wei Han, Shi-Dong Li, Shi-Jia Liao et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.