RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark
TL;DR
RAIL is introduced, a human-centric benchmark grounded in the Cattell--Horn--Carroll (CHC) framework, which formalises auditory cognition into five core capabilities, and develops them into structured evaluation tasks with principled data curation and human-aligned evaluation protocols.
Abstract
Humans process rich auditory environments through various cognitive capabilities such as audio perception, audio reasoning and memory. Despite recent progress in large audio-language models (LALMs), current evaluation remains largely task- or domain-centric, focusing on end performance while ignoring underlying auditory cognitive behaviors. This gap between how auditory cognition is understood in humans and how it is evaluated in LALMs hinders the interpretation of model behaviour in underlying capabilities and limits alignment with human auditory objectives. We introduce RAIL, a human-centric benchmark grounded in the Cattell--Horn--Carroll (CHC) framework, which formalises auditory cognition into five core capabilities, and develops them into structured evaluation tasks with principled data curation and human-aligned evaluation protocols. Evaluating 26 LALMs reveals strong performance in knowledge-related tasks but weak auditory perception and memory, reflecting reliance on language-based pretraining. Limited reasoning under auditory settings further suggests a mismatch between existing training paradigms and human-like auditory reasoning, as well as over-reasoning at the cost of efficiency. Six LALMs exceed human performance in general, while auditory processing all lags behind human. Overall, RAIL establishes a new framework for human-aligned evaluation of auditory intelligence.
