This work introduces Fuse, a multi-agent simulation framework for studying user-mediated social reasoning, and applies Fuse to 12 LLMs and demonstrates its analytical utility by systematically isolating key factors.
Abstract
LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation settings remains challenging since (i) it requires setups where the assistant learns about social situations from subjective user narratives, and (ii) social properties, such as others'intentions, typically lack verifiable ground truth. To address these challenges, we introduce Fuse, a multi-agent simulation framework for studying user-mediated social reasoning. In Fuse, a target agent with a hidden motive interacts with other agents including one representing the user, who then consults the evaluated assistant to infer the target's motive, providing verifiable ground truth by construction. Simulation faithfulness is validated through a human study with 24k annotations. We apply Fuse to 12 LLMs and demonstrate its analytical utility by systematically isolating key factors, showing that (i) user mediation compounds the inherent difficulty of social reasoning; (ii) LLMs exhibit systematic sensitivity to biased user framing; (iii) models can require more details than humans need to reach a correct prediction; and (iv) longer conversations do not always improve performance despite providing opportunities for clarifying questions. We open-source Fuse and a dataset with 21k examples.
ReAdapt is proposed, which augments the ReAct loop with an explicit structured social state z = (G, B, R, N, D) capturing goal, belief, relationship, norm, and disclosure, and which helps LLM agents turn retrieved social evidence into revised decisions.
Jian-Zhe Lin, Xiao-Lin Li, Yu Liu et al.· arXiv.org· 0 citations
This work presents Collaborative Reasoner, a framework to evaluate and improve the collaborative reasoning abilities of language models, and proposes a self-play method to generate synthetic multi-turn preference data and further train the language models to be better collaborators.
Ansong Ni, Ruta Desai, Yang Li et al.· Neural Information Processin...· 7 citations
A strategic game arena is introduced that situates LLM agents in social deduction scenarios inspired by Among Us, requiring theory of mind, deception detection, and cooperative deliberation under uncertainty, and motivates architectural changes for virtual agents that must cooperate with humans.
Kevin Kurian, Kevin Scroggins, Emmanuel Dorley et al.· Proceedings of the 26th ACM...· 0 citations
This paper implements a text-based, multi-agent version of the classic board game Clue as an environment to evaluate multi-step, agentic deductive reasoning, and introduces a tool-augmented approach in which a structured possibility matrix converts implicit game state from generated reasoning logs into an explicit repr...
This work conducts a controlled human study of six reasoning formats across tasks of varying complexity, supported by a web-based framework that randomizes task domains, problem instances, and representation order.
It is found that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when agents align with the target preference.
Dong-Chan Shin, Xing Han Lù, Jiaqi Deng et al.· arXiv.org· 0 citations