Skip to content

CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition

Sep 2026 · arXiv.org · Vol abs/2609.21259 · 0 citations · 76 references
Computer Science

TL;DR

CogGym is introduced, a scalable, unified framework grounded in cognitive science for systematically comparing model and human behavior on matched experimental trials, and intends for CogGym to provide a living evaluation framework that continually incorporates new cognitive science experiments to characterize where model behavior resembles human behavior, where it systematically diverges, and how those patterns change as models and experiments evolve.

Abstract

Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways do model responses resemble human responses, and where do they systematically diverge? The sheer breadth and diversity of the tasks humans can perform and think about pose a challenge for scalable and rigorous comparison between humans and models. We introduce CogGym, a scalable, unified framework grounded in cognitive science for systematically comparing model and human behavior on matched experimental trials. CogGym uses a semi-automated, human-in-the-loop pipeline to standardize diverse experimental paradigms into a task-agnostic Experiment Markup Language (EML), enabling reproducible and faithful comparison at scale. For initial release, we curate and standardize 258 cognitive experiments from 100 papers that focuses on human commonsense reasoning, and evaluate 50 large language models against human responses. We find a clear scaling trend where larger and more recent AI models better reproduce human judgments. Yet AI models'improvement on such common reasoning tasks is considerably slower than the gains observed on formal-reasoning benchmarks like math and coding, and model--human fit remains well below human splithalf reliability ($R^2 = 0.93$ on text, $0.95$ on image, and $0.92$ on video) with the best models achieving $R^2 = 0.59$ on text, $0.58$ on image, and $0.43$ on video experiments. We intend for CogGym to provide a living evaluation framework that continually incorporates new cognitive science experiments to characterize where model behavior resembles human behavior, where it systematically diverges, and how those patterns change as models and experiments evolve.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Thinking effort aligns between humans and reasoning models in abductive reasoning

This work isolates abductive reasoning from deductive reasoning: unlike deductive tasks, its difficulty cannot be inferred from formal structure and offers no shortcuts a model could exploit to mimic effort without genuine search, providing firmer ground for empirical claims of shared effort.

Henry Arthur · 0 citations
Open access Oct 2026

Why does human oversight remain essential in the age of AI?

Biological evolution required millions of years for primates to evolve into Homo sapiens, acquiring advanced cognitive abilities along the way. Among all species, humans stand out for their superior intelligence—specifically, the capacity to perceive information, store it as knowledge, and apply it adaptively within co...

L. Jayatissa · 0 citations
#large language models Review Open access Oct 2026

Energy constraints in large scale artificial intelligence and insights from human cognition for efficient large language models

A cross-layer framework that brings constraints together and examines whether resource-allocation strategies observed in human cognition can inform more sustainable AI design, and extends Green AI by linking these layers and translating insights from human cognition into testable design hypotheses and measurable evalua...

Mahmoud Amiri, Azadeh Mokari, Thomas Bocklitz · 0 citations
#artificial intelligence Preprint Sep 2026

Sparks of In Silico Cognitive Science: Theories from Simulated Data Can Generalize to Humans

The Automated Cognitive Scientist, a closed-loop discovery system in which LLM agents design theory-discriminating experiments, collect responses, arbitrate between competing theories, and synthesize successors, entirely on behavior simulated by Centaur, a foundation model of human behavior, succeeds despite the simula...

A. K. Jagadish, Younes Strittmatter, Nori Jacoby et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.