Skip to content
Preprint

HarnessEval-W: Agentifying the Evaluation of Visual Worlds

Aug 2026 · 3 citations · 89 references
Computer Science

TL;DR

This work introduces HarnessEval-W, an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking, and applies HarnessEval-W to 18 representative world models over 330 evaluation cases.

Abstract

A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified. We introduce HarnessEval-W, an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval-W interprets the context of each evaluation case, decomposes the evaluation question into measurable subproblems, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own subproblem. The parent agent then validates the gathered evidence and summarizes it into the final verdict. This hierarchical workflow turns every evaluation into a transparent evidence tree whose complete reasoning chain justifies the result. We apply HarnessEval-W to 18 representative world models over 330 evaluation cases. Its judgments closely align with human preferences while providing verifiable, fine-grained diagnoses of every generated rollout. We open-source the full pipeline as a live benchmark and invite the broad community to contribute to grow new skills and evaluation cases as world models evolve.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

EvalBuilder: Can LLMs Build Efficient Benchmarks?

Large language model (LLM) agents are increasingly used for long-horizon research and engineering, yet evaluating these capabilities remains expensive, time-consuming, and heavily dependent on human expertise. This motivates a natural question: can LLM agents themselves construct efficient benchmarks that preserve the...

Huai-Sheng Zhu, Hang-Fan Zhang, Teng Xiao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era

AgentIdeaBench is introduced, a multidisciplinary benchmark that evaluates scientific ideation under two matched settings, static observation and active exploration, and Scientific World Modeling is explored, a generation-time loop that refines a draft hypothesis through structured thought experiments.

Yunxiang Mo, Tianshi ZHENG, Yi-Sen Gao et al. · 1 citation
Preprint Aug 2026

From Atomic to Agentic: Towards Interpretable Evaluation of LLMs'Agentic Mathematical Capabilities

Experiments reveal that models with similar end-to-end accuracy can exhibit markedly different agentic capability profiles, demonstrating that process-level evaluation is crucial for interpreting the true potential of LLMs and guiding the development of next-generation mathematical agents.

Jiayi Kuang, Ying-Hui Li, Yun-Ze Song et al. · 0 citations
#artificial intelligence Preprint Oct 2026

AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents

As agents rapidly evolve, existing benchmarks can become saturated, limiting their ability to distinguish capabilities and reveal remaining failure modes. Particularly in scientific domains, constructing and updating benchmarks requires substantial time, labor, and domain expertise, making it difficult to keep evaluati...

Dongki Kim, Namkyeong Lee, Surag Nair et al. · 0 citations
Sep 2026

HappyWorld-Bench

HappyWorld-Bench is introduced, a comprehensive benchmark that evaluates whether generated worlds remain reliable as agents interact with them, and highlights the need to evaluate world models not only by visual quality, but also by state consistency and the correctness of their responses to actions and interventions.

Zhi-Qi Bai, Ju-Nai Cai, Yi-Xin Chen et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.