Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
TL;DR
RQEval is proposed, a unified multi-dimensional framework for measuring LLM reasoning quality from a behavioral perspective that operationalizes six theoretically grounded dimensions rooted in cognitive science and introduces deployment-aware aggregation, enabling context-specific model selection beyond accuracy-based leaderboards.
Abstract
Despite remarkable progress on reasoning benchmarks, current LLM evaluation practice remains anchored to final-answer correctness. This provides limited insight into how models reason, how reliably they behave under contextual variation, or how efficiently they reach conclusions. This paper proposes RQEval, a unified multi-dimensional framework for measuring LLM reasoning quality from a behavioral perspective. The framework operationalizes six theoretically grounded dimensions rooted in cognitive science: Correctness (CQ), Consistency (CS), Robustness (RS), Local Logical Coherence (LS), Efficiency (ES), and Stability (SS). It also introduces deployment-aware aggregation, enabling context-specific model selection beyond accuracy-based leaderboards. Applying RQEval to seven LLMs across four benchmarks reveals an outcome-level cluster (CQ, CS, RS, and ES) characterized by strong intercorrelations and a trace-level layer in which LS is empirically distinct from the outcome-level metrics, whereas SS retains moderate associations with several of them. Across the 28 model–dataset observations, LS showed no statistically significant correlation with any other dimension. Efficiency-weighted deployment scenarios, nevertheless, produced limited ranking inversions among models that were otherwise ranked consistently across weighting schemes. The resulting pipeline provides a foundation for diagnosing LLM reasoning behavior across deployment contexts, while highlighting domain-specific validation as an important direction for future work.
