Skip to content

HumanStudy-Bench: Towards AI Agent Design for Participant Simulation

Jan 2026 · arXiv.org · Vol abs/2602.00685 · 4 citations · 61 references
Computer Science

TL;DR

This work introduces an open platform and execution engine designed for practitioners to develop and evaluate agents tailored to their target experimental settings, and introduces two complementary metrics that quantify agreement with humans on both the significance conclusion and the effect size.

Abstract

Large language models (LLMs) are increasingly used as simulated participants in social science experiments, but their behavior is often unstable and highly sensitive to design choices. Prior evaluations frequently conflate base model capabilities with experimental instantiation, obscuring whether outcomes reflect the model itself or the agent setup. We instead frame $\textbf{participant simulation as an agent-design problem}$ over full experimental protocols, where an agent is defined by a base model and a specification (e.g., participant attributes) that encodes behavioral assumptions. We introduce $\textit{HUMANSTUDY-BENCH}$, an open platform and execution engine designed for practitioners to develop and evaluate agents tailored to their target experimental settings. The platform reconstructs published human-subject experiments via a human-in-the-loop Filter--Extract--Execute--Evaluate pipeline that preserves the original stimuli, conditions, and statistical procedures end to end, while allowing practitioners to freely explore the agent design space. We introduce two complementary metrics that quantify agreement with humans on both the significance conclusion and the effect size, while accounting for finite-sample uncertainty in the human reference data. In collaboration with social scientists, we validate the platform on a suite of 12 foundational studies covering 6,000+ trials across individual cognition, strategic interaction, and social psychology

View source

Similar papers

#natural language process... Preprint Aug 2026

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems

This work introduces PersonaForge, a user simulation framework for synthesizing realistic multi-turn user--agent interactions that combines a four-dimensional persona space, SOUL-driven behavioral control calibrated to real-user statistics, and Reverse Deep Construction grounded in authentic seed queries.

Hanglong Lv, Dawei Zhu, Lei Li et al. · 0 citations
Preprint Aug 2026

SIR: Self-improving Red-teaming for Compute Use Agents

Computer-use agents (CUAs) are agents powered by vision-language models (VLMs) that perceive a screen and operate an operating system through mouse, keyboard, and terminal interactions to automate everyday digital tasks. Their exposure to untrusted content creates a risk of indirect prompt injection (IPI), where an adv...

Chen Xiong, Zhi-Yuan He, Pin-Yu Chen et al. · 0 citations
#artificial intelligence Preprint Aug 2026

When AI Designs AI: Innovation or Imitation?

An analysis that derives task-specific algorithmic design spaces from human-designed methods, maps both human- and agent-designed methods into these spaces, and quantifies their algorithmic differences at the module level suggests that although current agents can occasionally match or surpass human SOTA performance, th...

Yikang Yang, Zhengxin Yang, Luzhou Peng et al. · 0 citations

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of agent success are introduced.

Ashish Jain, Armaan Sandhu · 2 citations · ⚡1
#artificial intelligence Preprint Oct 2026

Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents

Agentic large language model (LLM) systems are commonly implemented as an LLM in a loop with Planning, Memory, Tools, and Control Flow. This application-focused view connects agentic LLM research with deployable systems and leaves open how such systems should be evaluated beyond end-to-end task success. Building on thi...

Nijesh Upreti, Chris Sypherd, Vaishak Belle · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.