It is observed that reasoning models consistently exhibit increased robustness to prompt variations and task perturbations, which is read as evidence for a robustness-based account rather than for a new ToM-specific ability.
Abstract
Large language models (LLMs) have recently shown strong performance on Theory of Mind (ToM) tests, prompting debate about the nature and validity of the underlying capabilities. At the same time, reasoning-oriented LLMs trained via reinforcement learning with verifiable rewards have demonstrated notable improvements across a range of benchmarks. In this work, we examine the behavior of such reasoning models in ToM tasks using novel adaptations of machine psychological experiments together with results from established benchmarks. We observe that reasoning models consistently exhibit increased robustness to prompt variations and task perturbations. Our analysis suggests these gains come at least partly from models being more robust at reaching the correct answer under prompt and task variation. We read this as evidence for a robustness-based account rather than for a new ToM-specific ability.
A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept. Do reasoning models robustly exhibit such systematicity? If so, we would expect consistent performance on structurally equivalent variants of the same t...
The results reveal that, although LLMs occasionally succeed in decoding communicative intentions, their performance is not attributable to human-like ToM reasoning, and offers insight into their interpretive biases, contributing to a deeper understanding of their linguistic capabilities.
A. Lombardi, Alessandro Lenci· Transactions of the Associat...· 0 citations
This study systematically investigates how prompt structure, model architecture, and environment representation affect the quality of Explainable RL (XRL) explanations, and conducts a multi-model evaluation using quantitative metrics to assess explanation coherence, helpfulness, and stability.
Nivetha Suntharamoorthy, Jørn Eirik Betten, Dennis Gross et al.· 1 citation
Different capacities for mentalization across LLMs are demonstrated, and cognitive computational modeling is highlighted as a formal method for assessing comparative intelligence across humans and machines.
Aamir Sohail, Xintong Zhong, Arkady Konovalov et al.· 0 citations
Understanding how humans reason -- and how reasoning responses vary across tasks and individuals -- remains a core challenge for modeling and explanation in cognitive science. We investigate the stability of response patterns within reasoners and whether variation in these patterns can be used to predict learning effec...
Clemens Bombach, Rajmadan Lakshmanan, Marco Ragni· 0 citations
This work isolates abductive reasoning from deductive reasoning: unlike deductive tasks, its difficulty cannot be inferred from formal structure and offers no shortcuts a model could exploit to mimic effort without genuine search, providing firmer ground for empirical claims of shared effort.
Henry Arthur· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
