Skip to content

MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing

Sep 2026 · arXiv.org · Vol abs/2609.23416 · 2 citations · 11 references
Computer Science

TL;DR

MuLA-Bench exposes conditional failure patterns that a single long-context score does not capture, and evaluates ten audio-language models and conducts pooled diagnostics on a fixed eight-model cohort.

Abstract

Long-form audio performance is often summarized by context length and aggregate accuracy, obscuring how language, evidence, and task jointly shape difficulty. We introduce MuLA-Bench: 5,038 open-ended questions over 1,769 in-the-wild recordings totaling 1,377.9 hours, covering 16 languages and eight domains. A balanced Language x Domain semantic track supports controlled comparisons, while a complementary acoustic track preserves naturally occurring non-speech evidence. Evidence-grounded generation, shortcut checks, and language-expert review provide auditable questions without translating a shared source set or injecting target sounds. We evaluate ten audio-language models and conduct pooled diagnostics on a fixed eight-model cohort. Language rankings change across domains and tasks; acoustic-semantic performance gaps vary with the requested operation; and temporal errors can persist after the correct event is identified. Long-range retrieval is comparatively strong, while precise clock alignment and factual grounding of natural acoustic events remain fragile. MuLA-Bench thus exposes conditional failure patterns that a single long-context score does not capture.

View source

Similar papers

Preprint Sep 2026

AudioICL-Bench: A Benchmark for Large Audio Language Model In-Context Learning

AudioICL-Bench is introduced, a diagnostic benchmark whose per-episode rules are resampled so that no correct answer is recoverable from prior knowledge, and its nine tasks are organized along two axes that separate what must be learned from demonstrations from what must be perceived in the signal, enabling failures to...

Jia-Hung Chen, Yi-Cheng Lin, Kai-Wei Chang et al. · 0 citations
#machine learning Preprint Sep 2026

FuseAlign: Forced Alignment in the Wild

FuseAlign, a transformer-based aligner trained on large-scale pseudo-labeled speech with online label correction, is introduced, a transformer-based aligner trained on large-scale pseudo-labeled speech with online label correction that substantially outperforms all baselines and remains robust under real ASR transcript...

Mithilesh Vaidya, Stephen W. Bailey, Sumukh Badam et al. · 0 citations

Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models

Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient information. This issue is even more pronounced in the speech domain, where audio, as a low-compression modality, requires substantially more embeddings than text to prese...

Xi-Ze Cheng, Wen-Xu Jia, Chenyuhao Wen et al. · 4 citations · ⚡1
#natural language process... Preprint Sep 2026

SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

SEA-SpeechBench is introduced, the first large-scale multitask benchmark that evaluates speech understanding in 11 SEA languages through 97,194 samples across 99 evaluation sets and 597 hours of curated audio data, exposing critical model limitations and underscore the need for inclusive model development.

Jingyi Liao, Wenyu Zhang, Zhuo-Han Liu et al. · 1 citation
Preprint Sep 2026

TAG-Bench: Benchmarking Temporal Audio Grounding in Large Audio Language Models

Large audio language models (LALMs) can describe what is heard, but their ability to localize when queried content occurs remains less systematically evaluated. We present TAG-Bench, a benchmark for temporal audio grounding in which a model returns every time interval that matches a natural-language query. TAG-Bench co...

Yu-Hang Dai, Xin Shu, Zeng-Xi Li et al. · 2 citations

Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding

Mizar, a 159.3M-parameter ALM, is introduced, a recipe that brings together architecture, data, and three-stage training to build Mizar, a 159.3M-parameter ALM that surpasses the previous best-performing ALM below 200M parameters on all three benchmarks.

Kai-Yang Li, Shaobo Han, Yue Tian et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.