This work introduces TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio, and extends it with expressive speech attributes to obtain TAC2-S, a unified model that yields strong spoken audio understanding across diverse reasoning backends.
Abstract
Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.
Large audio-language models (LALMs) excel at understanding and reasoning tasks over atomic sound events, yet their ability to infer higher-level human activities from such fine-grained events remains largely unexamined. Everyday human actions and activities, such as setting a table, cleaning the house, or preparing a b...
Michel Olvera, Paraskevas Stamatiadis, Chang-Hong Wang et al.· Conference on Empirical Meth...· 0 citations
We present Qwen-Audio-3.1-ASR, a family of audio-language models that builds upon its predecesor Qwen-Audio-3.0-ASR with improved speech recognition and expanded capabilities in speaker-diarized transcription and general audio understanding. The family retains support for 30 languages and 16 Chinese dialects, together...
This work augments the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query, and shows that the grounding model can provide temporal evidence to support downstream reasoning.
Yan Shi, Yan Song, Jun-Hui Li et al.· arXiv.org· 0 citations
Audio captioning benchmarks typically evaluate a single unstructured free-text description of an audio scene, providing limited insight into how a model fails to understand polyphonic audio. A plausible caption may omit a source, or conflate multiple sources into one, or assign an attribute of one source to another, an...
Sripathi Sridhar, Yujin Kim, Giovana Morais et al.· 0 citations
Extensive evaluations show that R2T2 brings streaming recognition close to the base model's full-context quality while delivering stable text with low emission lag, and its gains extend beyond the post-training languages, demonstrating a practical path from full-context ASR to stable streaming.
Daniel Huang, Qian-Yue He, Shao-Xiong Duan et al.· 0 citations
TiTok is presented, an audio-visual large language model (AV-LLM) that localizes an arbitrary number of temporal event segments for each query, and the Time Token Interleaving (TTI) method is introduced, which explicitly injects special time tokens into the audio-visual stream to align input-side temporal perception wi...
Eunji Shin, Dahyun Choi, Seungyeon Jo et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.