Jan 2026· arXiv.org· Vol abs/2602.00685· 4 citations· 61 references
Computer Science
TL;DR
This work introduces an open platform and execution engine designed for practitioners to develop and evaluate agents tailored to their target experimental settings, and introduces two complementary metrics that quantify agreement with humans on both the significance conclusion and the effect size.
Abstract
Large language models (LLMs) are increasingly used as simulated participants in social science experiments, but their behavior is often unstable and highly sensitive to design choices. Prior evaluations frequently conflate base model capabilities with experimental instantiation, obscuring whether outcomes reflect the model itself or the agent setup. We instead frame $\textbf{participant simulation as an agent-design problem}$ over full experimental protocols, where an agent is defined by a base model and a specification (e.g., participant attributes) that encodes behavioral assumptions. We introduce $\textit{HUMANSTUDY-BENCH}$, an open platform and execution engine designed for practitioners to develop and evaluate agents tailored to their target experimental settings. The platform reconstructs published human-subject experiments via a human-in-the-loop Filter--Extract--Execute--Evaluate pipeline that preserves the original stimuli, conditions, and statistical procedures end to end, while allowing practitioners to freely explore the agent design space. We introduce two complementary metrics that quantify agreement with humans on both the significance conclusion and the effect size, while accounting for finite-sample uncertainty in the human reference data. In collaboration with social scientists, we validate the platform on a suite of 12 foundational studies covering 6,000+ trials across individual cognition, strategic interaction, and social psychology
This work introduces PersonaForge, a user simulation framework for synthesizing realistic multi-turn user--agent interactions that combines a four-dimensional persona space, SOUL-driven behavioral control calibrated to real-user statistics, and Reverse Deep Construction grounded in authentic seed queries.
Hanglong Lv, Dawei Zhu, Lei Li et al.· 0 citations
The evaluation of 20 frontier model arms demonstrates that high-fidelity user simulation is already attainable: leading models achieve up to 84.7% full-pass adherence under unprompted conditions.
Jin-Tao Huang, Yi-Fan Wang, Hong-Yuan Shen et al.· 1 citation
Computer-use agents (CUAs) are agents powered by vision-language models (VLMs) that perceive a screen and operate an operating system through mouse, keyboard, and terminal interactions to automate everyday digital tasks. Their exposure to untrusted content creates a risk of indirect prompt injection (IPI), where an adv...
Chen Xiong, Zhi-Yuan He, Pin-Yu Chen et al.· 0 citations
An analysis that derives task-specific algorithmic design spaces from human-designed methods, maps both human- and agent-designed methods into these spaces, and quantifies their algorithmic differences at the module level suggests that although current agents can occasionally match or surpass human SOTA performance, th...
Yikang Yang, Zhengxin Yang, Luzhou Peng et al.· 0 citations
UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of agent success are introduced.
Agentic large language model (LLM) systems are commonly implemented as an LLM in a loop with Planning, Memory, Tools, and Control Flow. This application-focused view connects agentic LLM research with deployable systems and leaves open how such systems should be evaluated beyond end-to-end task success. Building on thi...
Nijesh Upreti, Chris Sypherd, Vaishak Belle· 0 citations