Skip to content
Preprint

Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

Oct 2026 · 1 citation
Computer Science

TL;DR

Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments, and this work evaluates how backbone models, self-evolving methods, and agent harnesses affect agents' learning ability, revealing three findings.

Abstract

Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments. Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge. Existing benchmarks have sought to evaluate this ability, but they primarily evaluate tasks whose rules are provided in the instructions or already familiar to pretrained models, making it difficult to distinguish learning from interactions from reasoning with existing knowledge. To address this, we introduce Learn2Play Bench, a benchmark of newly designed text-based games, whose rules are novel or counterintuitive, requiring agents to acquire knowledge through interaction rather than rely solely on pretrained knowledge. These games provide reproducible feedback and automatic scoring, enabling controlled evaluation of learning across repeated attempts. We also vary game instances to test whether agents can apply what they have learned to new situations. Therefore, we evaluate how backbone models, self-evolving methods, and agent harnesses affect agents'learning ability, revealing three findings: (1) Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies. (2) Human agent gap: Top-performing human players achieve higher peak scores than the evaluated agents. Human explore more varied strategies, and repeat actions less. (3) Harness matters: With the backbone fixed, changing the harness can improve performance while reducing estimated inference cost. Together, these findings provide insights into how LLM agents learn from experience and suggest directions for future work to improve their learning ability. Project website: https://liushiliushi.github.io/learn2play-bench-website/

View source

Similar papers

#artificial intelligence Preprint Oct 2026

ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience?

Large language model agents are increasingly deployed to perform complex tasks in real-world environments. However, the knowledge required for correct behavior in these environments is often implicit, undisclosed, and subject to change over time. Recent continual-learning harnesses seek to address this challenge by ena...

Hai-Zhong Zheng, Yi-Zhu Di, Ranajoy Sadhukhan et al. · 2 citations
Preprint Oct 2026

How Humans Learn Unfamiliar Worlds: A Think-Aloud Dataset on ARC-AGI-3

Learning through exploration and using newly acquired knowledge to achieve goals are central capabilities of general intelligence. Understanding these capabilities requires tracing how learners build, use, and revise world models during interaction. We construct the first dataset that records how humans discover and us...

Shu-Zhi Liu, Archana Warrier, Dat Nguyen et al. · 0 citations

PhantomEnvironments: Training LLM Agents in Fictional Worlds

Training LLM agents with reinforcement learning (RL) is bottlenecked by environments, which must provide verifiable rewards, support long-horizon interaction, and scale cheaply. Existing approaches rely on costly human-curated data or on LLM-generated environments that risk hallucinations and benchmark contamination. W...

Anmol Kabra, S. Selvam, Albert Gong et al. · 0 citations

Schema: Discovering Unknown Environments via Agentic Program Induction

Learning to complete tasks in unfamiliar environments with unknown rules remains a key challenge for LLM agents. Current LLM agents often record their discoveries in prose, which may not provide a compact, explicit account of how the environment works. Inspired by how scientists organize observations into testable, pre...

Guan-Ning Zeng, Jia-Ni Wang, Wenjie Ma et al. · 0 citations

Witness: Discovery, Deciphering, and Epiphany in Interactive Puzzle Environments

Results point to rule acquisition as a major difficulty for frontier models like Opus-5 while smaller models further struggle on rule-based execution, and indicate that RL on hidden-rule puzzles transfers to broader rules and real-world tasks beyond training.

Guang-Han Ning, Ping Liu, Lin-Yi Li et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents

The introduced Agent Unlearning Benchmark (AUBench), a controlled benchmark that extends the fictitious-author setting of TOFU into an agent environment with persistent memory, an offline web snapshot, tool-accessible documents, and provenance-tracked interaction trajectories, and provides a foundation for developing u...

Jun-Kai Lin, Ge Xu, Zheng Xu et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.