Skip to content

Verifiable Social Reasoning for LLM Assistants

Sep 2026 · 2 citations · 43 references
Computer Science

TL;DR

This work introduces Fuse, a multi-agent simulation framework for studying user-mediated social reasoning, and applies Fuse to 12 LLMs and demonstrates its analytical utility by systematically isolating key factors.

Abstract

LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation settings remains challenging since (i) it requires setups where the assistant learns about social situations from subjective user narratives, and (ii) social properties, such as others'intentions, typically lack verifiable ground truth. To address these challenges, we introduce Fuse, a multi-agent simulation framework for studying user-mediated social reasoning. In Fuse, a target agent with a hidden motive interacts with other agents including one representing the user, who then consults the evaluated assistant to infer the target's motive, providing verifiable ground truth by construction. Simulation faithfulness is validated through a human study with 24k annotations. We apply Fuse to 12 LLMs and demonstrate its analytical utility by systematically isolating key factors, showing that (i) user mediation compounds the inherent difficulty of social reasoning; (ii) LLMs exhibit systematic sensitivity to biased user framing; (iii) models can require more details than humans need to reach a correct prediction; and (iv) longer conversations do not always improve performance despite providing opportunities for clarifying questions. We open-source Fuse and a dataset with 21k examples.

View source

Similar papers

When LLM Agents Fail to Read the Room: ReAdapt for Relational Social Reasoning

ReAdapt is proposed, which augments the ReAct loop with an explicit structured social state z = (G, B, R, N, D) capturing goal, belief, relationship, norm, and disclosure, and which helps LLM agents turn retrieved social evidence into revised decisions.

Jian-Zhe Lin, Xiao-Lin Li, Yu Liu et al. · 0 citations
2025

Collaborative Reasoner: Self-Improving Social Agents with Synthetic Conversations

This work presents Collaborative Reasoner, a framework to evaluate and improve the collaborative reasoning abilities of language models, and proposes a self-play method to generate synthetic multi-turn preference data and further train the language models to be better collaborators.

Ansong Ni, Ruta Desai, Yang Li et al. · 7 citations
#small language model Book Open access Sep 2026

Evaluating LLM Social Cognition Through Multi-Agentic Strategic Games

A strategic game arena is introduced that situates LLM agents in social deduction scenarios inspired by Among Us, requiring theory of mind, deception detection, and cooperative deliberation under uncertainty, and motivates architectural changes for virtual agents that must cooperate with humans.

Kevin Kurian, Kevin Scroggins, Emmanuel Dorley et al. · 0 citations

Clueing up LLMs with Tool-Augmented Deductive Reasoning

This paper implements a text-based, multi-agent version of the classic board game Clue as an environment to evaluate multi-step, agentic deductive reasoning, and introduces a tool-augmented approach in which a structured possibility matrix converts implicit game state from generated reasoning logs into an explicit repr...

Rebecca Ansell, Autumn Toney-Wails · 0 citations
#machine learning Preprint Sep 2026

Do Reasoning Representations Help Humans Evaluate LLM Outputs?

This work conducts a controlled human study of six reasoning formats across tasks of varying complexity, supported by a web-based framework that randomizes task domains, problem instances, and representation order.

Jaewoo Lim, Sungbok Shin, San Hong · 0 citations
Sep 2026

AdaptArena: Evaluating Test-Time Personalization of Web Agents

It is found that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when agents align with the target preference.

Dong-Chan Shin, Xing Han Lù, Jiaqi Deng et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.