CogGym is introduced, a scalable, unified framework grounded in cognitive science for systematically comparing model and human behavior on matched experimental trials, and intends for CogGym to provide a living evaluation framework that continually incorporates new cognitive science experiments to characterize where model behavior resembles human behavior, where it systematically diverges, and how those patterns change as models and experiments evolve.
Abstract
Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways do model responses resemble human responses, and where do they systematically diverge? The sheer breadth and diversity of the tasks humans can perform and think about pose a challenge for scalable and rigorous comparison between humans and models. We introduce CogGym, a scalable, unified framework grounded in cognitive science for systematically comparing model and human behavior on matched experimental trials. CogGym uses a semi-automated, human-in-the-loop pipeline to standardize diverse experimental paradigms into a task-agnostic Experiment Markup Language (EML), enabling reproducible and faithful comparison at scale. For initial release, we curate and standardize 258 cognitive experiments from 100 papers that focuses on human commonsense reasoning, and evaluate 50 large language models against human responses. We find a clear scaling trend where larger and more recent AI models better reproduce human judgments. Yet AI models'improvement on such common reasoning tasks is considerably slower than the gains observed on formal-reasoning benchmarks like math and coding, and model--human fit remains well below human splithalf reliability ($R^2 = 0.93$ on text, $0.95$ on image, and $0.92$ on video) with the best models achieving $R^2 = 0.59$ on text, $0.58$ on image, and $0.43$ on video experiments. We intend for CogGym to provide a living evaluation framework that continually incorporates new cognitive science experiments to characterize where model behavior resembles human behavior, where it systematically diverges, and how those patterns change as models and experiments evolve.
This work isolates abductive reasoning from deductive reasoning: unlike deductive tasks, its difficulty cannot be inferred from formal structure and offers no shortcuts a model could exploit to mimic effort without genuine search, providing firmer ground for empirical claims of shared effort.
Biological evolution required millions of years for primates to evolve into Homo sapiens, acquiring advanced cognitive abilities along the way. Among all species, humans stand out for their superior intelligence—specifically, the capacity to perceive information, store it as knowledge, and apply it adaptively within co...
L. Jayatissa· Journal of the National Scie...· 0 citations
This research advances human-centred XAI by developing and evaluating explanation methods that help people understand model behaviour, judge when to rely on AI systems, and improve those systems.
It is argued that emergent reasoning in LLMs is a product of three factors: model size, prompting approach, and evaluation metric, and proposed implications for designing benchmarks and assessing capabilities are proposed.
N. Kuotsu· International Journal of Cre...· 0 citations
A cross-layer framework that brings constraints together and examines whether resource-allocation strategies observed in human cognition can inform more sustainable AI design, and extends Green AI by linking these layers and translating insights from human cognition into testable design hypotheses and measurable evalua...
The Automated Cognitive Scientist, a closed-loop discovery system in which LLM agents design theory-discriminating experiments, collect responses, arbitrate between competing theories, and synthesize successors, entirely on behavior simulated by Centaur, a foundation model of human behavior, succeeds despite the simula...
A. K. Jagadish, Younes Strittmatter, Nori Jacoby et al.· 0 citations