Nicholas DeFlumeri

Visual Reasoning Failure Modes Protocol: construction and measurement of figure-based probes designed to separate careful human reading from vision-language model reading

When a vision-language model gets a question about a figure wrong, the score alone does not say why. The model may have missed the mark, taken a shortcut the figure offered, or answered from the question text without looking. Existing benchmarks control for one or two of these causes. The author has not found one that...