This work introduces SQUARE-Bench, a comprehensive benchmark that systematically evaluates LMM capabilities as evaluators of AI-generated images across four aspects: Semantics, Quality, Authenticity, and Responsibility, and conducts a proof-of-concept study of LMM-guided iterative editing.
Abstract
With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture fine-grained alignment and generative artifacts. While large multimodal models (LMMs) are increasingly adopted as evaluators, existing benchmarks typically study semantic understanding, quality perception, and authenticity identification in isolation, while largely neglecting responsibility detection. This leaves a gap in unified and comprehensive validation. To bridge this gap, we introduce SQUARE-Bench, a comprehensive benchmark that systematically evaluates LMM capabilities as evaluators of AI-generated images across four aspects: Semantics, Quality, Authenticity, and Responsibility. SQUARE-Bench introduces a granular taxonomy of 38 sub-dimensions to evaluate nearly 10K AI-generated images sampled from 22 diverse models, ranging from legacy to state-of-the-art generators, complemented by over 3K real-world images. The images are annotated with curated question-answering pairs. Extensive experiments on 23 LMMs reveal that top proprietary models, such as Gemini-3-Pro, already outperform the individual human expert baseline. However, the performance gap between models remains significant, exhibiting notable disparities in fine-grained inference and domain-specific robustness. Beyond benchmarking, we conduct a proof-of-concept study of LMM-guided iterative editing, in which dimension-specific LMMs provide diagnostic feedback to fixed image editors. The resulting guided system yields selective improvements in semantics, authenticity, and responsibility, while exhibiting a consistent visual-quality trade-off. SQUARE-Bench can serve as both a diagnostic tool for characterizing LMM evaluator capabilities and studying their use in T2I generation refinement. The benchmark and dataset will be released upon publication.
A multi-dimensional image quality assessment framework for T2I tasks that can not only distinguish overall performance differences across models, but also provide interpretable results at the level of individual dimensions is proposed.
D3-Omni is introduced, a balanced and decoupled benchmark for diagnosing fine-grained multimodal understanding, covering 53 orthogonal binary dimensions and 10,671 samples across the three tasks, suggesting that aggregate accuracy may hide systematic blind spots that a balanced and decoupled lens can help expose and, i...
Guang Hu, Ziyue Jiang, Wei Qiao et al.· 0 citations
A concise, replicable evaluation of the two public detectors that function using differing architectures, which indicated that UnivFD was more accurate than CNNSpot in detection of original images and that no transformation crossed the threshold for practical significance set at ten.
Aaranya Singh, Prakash Chand Jain, Ayushka Singh· Iconic research and engineer...· 0 citations
Text-to-image (T2I) generation models are increasingly used in industrial design, but assessing the quality of their outputs still depends on slow, small-scale human expert review, which constrains the automation advantage of generative AI and hinders its adoption in this domain. This paper empirically examines whether...
Zai-Xing Liu, Jie-Hong Xie, Peng-Tao Li et al.· Conference on Optics and Com...· 0 citations
REALIS provides a unified framework for measuring and improving the reliability of AI-image detectors under conditions that better reflect real-world use, and proposes a robustness protocol covering 35 transformations at five severity levels to analyze detector behavior under image processing.
A. Gushchin, Khaled Abud, G. Bychkov et al.· arXiv.org· 0 citations
This work introduces GIFTBench, a multi-axis benchmark of 115,013 manipulated images with pixel-level annotations spanning manipulation source, semantic target, editing operation, and composition complexity, and develops ForenScope, a detection and localization framework combining classification-adapted representations...
Bao-Ke Dou, Zi-Ye Wang, Hao Wang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.