Skip to content

Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessment

Sep 2026 · arXiv.org · Vol abs/2609.37576 · 0 citations · 90 references
Computer Science

TL;DR

This work introduces SQUARE-Bench, a comprehensive benchmark that systematically evaluates LMM capabilities as evaluators of AI-generated images across four aspects: Semantics, Quality, Authenticity, and Responsibility, and conducts a proof-of-concept study of LMM-guided iterative editing.

Abstract

With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture fine-grained alignment and generative artifacts. While large multimodal models (LMMs) are increasingly adopted as evaluators, existing benchmarks typically study semantic understanding, quality perception, and authenticity identification in isolation, while largely neglecting responsibility detection. This leaves a gap in unified and comprehensive validation. To bridge this gap, we introduce SQUARE-Bench, a comprehensive benchmark that systematically evaluates LMM capabilities as evaluators of AI-generated images across four aspects: Semantics, Quality, Authenticity, and Responsibility. SQUARE-Bench introduces a granular taxonomy of 38 sub-dimensions to evaluate nearly 10K AI-generated images sampled from 22 diverse models, ranging from legacy to state-of-the-art generators, complemented by over 3K real-world images. The images are annotated with curated question-answering pairs. Extensive experiments on 23 LMMs reveal that top proprietary models, such as Gemini-3-Pro, already outperform the individual human expert baseline. However, the performance gap between models remains significant, exhibiting notable disparities in fine-grained inference and domain-specific robustness. Beyond benchmarking, we conduct a proof-of-concept study of LMM-guided iterative editing, in which dimension-specific LMMs provide diagnostic feedback to fixed image editors. The resulting guided system yields selective improvements in semantics, authenticity, and responsibility, while exhibiting a consistent visual-quality trade-off. SQUARE-Bench can serve as both a diagnostic tool for characterizing LMM evaluator capabilities and studying their use in T2I generation refinement. The benchmark and dataset will be released upon publication.

View source

Similar papers

Open access Aug 2026

Structured Creative Evaluation for Text-to-Image Generative AI Models

A multi-dimensional image quality assessment framework for T2I tasks that can not only distinguish overall performance differences across models, but also provide interpretable results at the level of individual dimensions is proposed.

W.-C. Ma, Q. Zhang · 0 citations
Preprint Aug 2026

OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses

D3-Omni is introduced, a balanced and decoupled benchmark for diagnosing fine-grained multimodal understanding, covering 53 orthogonal binary dimensions and 10,671 samples across the three tasks, suggesting that aggregate accuracy may hide systematic blind spots that a balanced and decoupled lens can help expose and, i...

Guang Hu, Ziyue Jiang, Wei Qiao et al. · 0 citations
Open access Sep 2026

Beyond Clean Benchmarks: A Reproducible Severity-Sweep Evaluation of AI-Generated Image Detector Robustness

A concise, replicable evaluation of the two public detectors that function using differing architectures, which indicated that UnivFD was more accurate than CNNSpot in detection of original images and that no transformation crossed the threshold for practical significance set at ten.

Aaranya Singh, Prakash Chand Jain, Ayushka Singh · 0 citations
#diffusion models Conference Oct 2026

Can multimodal large language models evaluate AI-generated industrial design images? An empirical comparison with human expert assessment

Text-to-image (T2I) generation models are increasingly used in industrial design, but assessing the quality of their outputs still depends on slow, small-scale human expert review, which constrains the automation advantage of generative AI and hinders its adoption in this domain. This paper empirically examines whether...

Zai-Xing Liu, Jie-Hong Xie, Peng-Tao Li et al. · 0 citations

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

REALIS provides a unified framework for measuring and improving the reliability of AI-image detectors under conditions that better reflect real-world use, and proposes a robustness protocol covering 35 transformations at five severity levels to analyze detector behavior under image processing.

A. Gushchin, Khaled Abud, G. Bychkov et al. · 0 citations
Review Oct 2026

GIFTBench: Diagnosing Generalization in Image Forgery Localization and Informing Model Design

This work introduces GIFTBench, a multi-axis benchmark of 115,013 manipulated images with pixel-level annotations spanning manipulation source, semantic target, editing operation, and composition complexity, and develops ForenScope, a detection and localization framework combining classification-adapted representations...

Bao-Ke Dou, Zi-Ye Wang, Hao Wang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.