Skip to content
Preprint

Human-AI Perceptual Alignment by Playing Hues and Cues

Aug 2026 · 0 citations · 16 references
Computer Science

TL;DR

It is revealed that highly curated pre-training datasets are significantly more effective than massive, uncurated corpora in mitigating these severe misalignments, and that highly curated pre-training datasets are significantly more effective than massive, uncurated corpora in mitigating these severe misalignments.

Abstract

Evaluating the perceptual alignment between Contrastive Vision-Language Models (CVLMs) and humans is typically constrained by traditional benchmarks that overlook fine-grained semantic and cultural nuances. In this work, we propose a novel evaluation framework that leverages the gamified, discrete color space of the board game Hues and Cues. By mapping the board's 480 color cells to the CIE xy chromaticity diagram, we calculate empirical perceptual distances across a carefully curated 100-word vocabulary spanning seven semantic categories. To properly contextualize model performance, we establish an empirical lower bound of expected error-the Human Consistency baseline-calculated via Leave-One-Out (LOO) cross-validation on a dense dataset of color associations collected from 325 human observers through a custom digital interface. We evaluate 162 models across multiple architectural families and pre-training datasets to assess their semantic color grounding. Our results demonstrate that while CVLMs successfully replicate human cognitive biases, such as idealized memory colors for concrete physical referents (e.g., food and plants), they systematically diverge from the human baseline in abstract, subjective, and pop-culture domains. We identify two distinct failure modes in severely misaligned concepts: semantic misclassification and a systematic uncertainty collapse into a default blue coordinate. Furthermore, we reveal that highly curated pre-training datasets are significantly more effective than massive, uncurated corpora in mitigating these severe misalignments. Ultimately, this work highlights that despite their broad categorization capabilities, current CVLMs still fail to capture the nuanced, localized consensus of human color memory, emphasizing the value of gamified tasks in exposing underlying model biases. The data and code are publicly available to test other metrics.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

NormViz: A Benchmark and Framework for Grounding Multimodal Reasoning in Global Cultures

Though absolute performance remains low, finetuning on NormViz-Train improves pair accuracy relatively by up to 125% and 36% Qwen3-VL 4B and 8B respectively, showing a path forward to teach models to connect visual perception to cultural significance.

Akhila Yerukola, Fabrice Y. Harel-Canada, Simran Khanuja et al. · 0 citations
Preprint Sep 2026

EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception

EviViT, a lightweight attachment that learns where a pretrained vision transformer should acquire detail, is introduced, a lightweight attachment that learns where a pretrained vision transformer should acquire detail without refitting.

Yao-Xin Niu, Zhangquan Chen, Yang Zhang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models

Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek determines if a vehicle can fit between two parked cars; a few seconds of video reveals who holds authority in a room; and a fleeting clip highlights subtle abstract patterns...

Xing-Ming Guo, Jing Gu, Brian Jang et al. · 0 citations
Preprint Aug 2026

Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding

Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model, is proposed, endowing models with native fine-grained region description and flexible reasoning capabilities.

Chang-Jiang Jiang, Qian-Nian Zhao, Lei Xin et al. · 2 citations
Sep 2026

Multidimensional Observer Model and Perceptual Dimensions of Human Image Quality Assessment

A multi-dimensional observer model is proposed that represents images as distributions in a latent perceptual space and that models human judgment as comparing noisy samples, which finds that the perceptual spaces needed to account for image quality judgment in humans are extremely low-dimensional compared to the image...

Sheng Zhao, Wei-Kai Lin, Yu-Hao Zhu · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.