To improve self-modeling skill, a scalable synthetic-data pipeline is developed that produces self-modeling training data, and reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks.
Abstract
We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a prompt edit would change the model's final answer. To measure this capability, we introduce a benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.
These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies, and provide a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable se...
Jia-Jun Shi, Siyang Tao, Yu-Hao Wu et al.· 2 citations
This work evaluates ten instruction-tuned models from the LLaMA-3 and Qwen-2.5 families, measuring faithfulness, minimality, and alignment with human-annotated rationales and shows that model scale is the strongest determinant of explanation quality.
Giannis Kalyvas, Giorgos Filandrianos, Orfeas Menis-Mastromichalakis et al.· 0 citations
The Cognitively Bounded User Simulator (CBUS) is introduced, an architectural framework that explicitly models the restricted working memory of young readers through an episodic bottleneck and shows that enforcing architectural constraints is more effective for high-fidelity simulation than simply scaling raw model cap...
This study studies how LLMs learn from iterative experience at test time, a setting the authors refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference.
Hao-Qin Tu, Yun-Hao Fang, Yizhong Wang et al.· 0 citations
This work investigates strategic diversity, or substantive variation among approaches to a problem, as an alternative principle for constructing self-training data with two sampling methods: GROOT, a new method which constructs a hierarchical tree of approaches and samples distinct paths, and Verbalized Sampling, adapt...
Alexander Gurung, Esmeralda S. Whitammer, Mirella Lapata· 0 citations
Self-Routing is proposed, a behavior-conditioned post-training framework that uses rollout correctness and confidence to decide how each sample should be optimized, allowing training to adapt without external teachers, extra annotations, or additional sampling.
Yi-Fei Li, Lingling Zhang, Mu-Ye Huang et al.· 0 citations
