Visual Reasoning Failure Modes Protocol: construction and measurement of figure-based probes designed to separate careful human reading from vision-language model reading

Abstract

When a vision-language model gets a question about a figure wrong, the score alone does not say why. The model may have missed the mark, taken a shortcut the figure offered, or answered from the question text without looking. Existing benchmarks control for one or two of these causes. The author has not found one that controls for all three in the same item. This protocol describes how to build items that do. Each item is one image, one closed question and one answer that a careful person can read from the pixels. Every tempting wrong answer is computed from the scene and labeled with the shortcut that produces it. Each item also has a twin: the same image with the main shortcut removed, scored against the same wrong answers. The difference between twin accuracy and item accuracy is the measure, and it is tested with a paired test of stated power. Three checks run before any model is scored. The image file is checked for hidden text. The question is given to text-only models to show it cannot be answered without the image. At least ten people answer it, and no more than one may miss. The protocol defines nineteen failure modes, from reading a small number to noticing that a field is empty, each with a worked example. It also covers tasks that span several documents and the security risks of each mode. Nothing has been piloted yet and no model results are reported. The pilot study that would test the method is specified, and the rules and evidence draw on 160 sources. Status: first edition, draft for comment. Every failure mode is at evidence status hypothesis. Files: the PDF (267 pages) and the complete bundle (master Markdown, figures, scene files, renderers, measurement scripts, build scripts and checksums), from which every exhibit and the PDF rebuild. Licensing: text CC BY 4.0; code MIT; eight adapted published figures remain CC BY 4.0 under their sources' credits; one photograph is under the Unsplash License. See LICENSE.md in the bundle.