As large audio language models (LALMs) advance, robust evaluation frameworks have become essential. In this context, Spanish speech understanding under realistic acoustic conditions has received particularly little attention. We introduce ESCUCHA, the first Spanish speech understanding benchmark designed to evaluate LALMs across heterogeneous acoustic conditions and reasoning abilities. ESCUCHA comprises 1,000 human-curated questions paired with audio, totaling 162.9 hours sourced directly ``from the wild''rather than drawn from existing datasets, with durations ranging from a few seconds to over 80 minutes. The benchmark emphasizes reasoning, spanning 9 perceptual and 10 reasoning categories, and it captures linguistic diversity through multiple Spanish accents and non-normative speech. ESCUCHA further includes multi-audio questions, spoken questions, and audio instructions, and it flags which questions support open-ended evaluation. Benchmarking several state-of-the-art multimodal and speech models reveals substantial performance gaps relative to trained humans.
MuLA-Bench exposes conditional failure patterns that a single long-context score does not capture, and evaluates ten audio-language models and conducts pooled diagnostics on a fixed eight-model cohort.
Ze-Yu Yang, Xin-Yu Zhang, Zi-Bo Bi et al.· 2 citations
This work introduces MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs, and finds that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably.
Yi-Ze Li, Ning-Yuan Yang, Sile Yin et al.· 2 citations
SEA-SpeechBench is introduced, the first large-scale multitask benchmark that evaluates speech understanding in 11 SEA languages through 97,194 samples across 99 evaluation sets and 597 hours of curated audio data, exposing critical model limitations and underscore the need for inclusive model development.
Jingyi Liao, Wenyu Zhang, Zhuo-Han Liu et al.· 1 citation
It is argued that principled memory evaluation requires jointly characterizing the acoustic evidence to be retained and the operations applied to it, and introduced a taxonomy along these two axes, which aims to provide a foundation to measure and drive progress on the full scope of spoken conversational memory.
Xiao Yang, Vidhyasaharan Sethu, Eun-Jung Holden et al.· 0 citations
This work develops ParA-LLM, a framework of 22 paralinguistic characteristics that surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench and releases ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories.
Nishit Anand, Jia-Qi Su, Ke Chen et al.· 1 citation
Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient information. This issue is even more pronounced in the speech domain, where audio, as a low-compression modality, requires substantially more embeddings than text to prese...
