CapGeo-Bench is proposed, a benchmark of 4,641 high-quality figure-caption pairs equipped with a fine-grained keypoint-based evaluation metric that provides high-quality captions consistently and substantially boosts performance of MLLMs, empirically validating the visual perception bottleneck in geometric reasoning.
Abstract
While Multimodal Large Language Models (MLLMs) have achieved remarkable success in difficult purely textual mathematical reasoning tasks, even advanced closed-source models such as GPT-o3 still struggle with geometric problems. This discrepancy motivates us to investigate the root cause: is the bottleneck of multimodal geometric reasoning rooted in reasoning itself, or in the perception of geometric information from diagrams? To answer this question, we conduct an exploratory experiment and find that providing high-quality captions consistently and substantially boosts performance of MLLMs, empirically validating the visual perception bottleneck in geometric reasoning. However, MLLMs'capabilities in visual geometric perception remain insufficiently evaluated. To fill the gap, we propose CapGeo-Bench, a benchmark of 4,641 high-quality figure-caption pairs equipped with a fine-grained keypoint-based evaluation metric. Specifically, CapGeo-Bench encompasses three geometric classes and is divided into four difficulty levels according to the complexity of geometric primitives. Captions are scored at a fine-grained level by extracting keypoints along three dimensions-geometric elements, spatial relationships, and numerical attributes. CapGeo-Bench provides a reliable instrument and guideline for evaluating and advancing MLLMs'geometric perception capabilities. Our code and data are publicly available at: https://github.com/YuYingLi0/CapGeo.
Multimodal large language models can recognize visual content yet struggle to reason reliably over it across multiple steps. In counting, a model may see all objects but lose track of which ones it has counted. We argue that such failures, commonly attributed to perception, reveal a reference gap: the failure to bind l...
Gui-Zhen Chen, Wei-Wen Xu, Wo-Tao Yin et al.· 0 citations
Multimodal Large Language Models (MLLMs) have shown strong performance on mathematical reasoning tasks that combine textual and visual information. However, final-answer accuracy offers limited insight into whether errors arise from incorrect perception, modality-dependent information extraction, or downstream reasonin...
Aniket Mitra, Arpit Chhabra, Aditya Desai et al.· Proceedings of the 35th ACM...· 0 citations
Multimodal Large Language Models (MLLMs) inherit strong textual reasoning capabilities from their language backbones and text pretraining, but often fail to apply the same level of competency to visual inputs. We show that this degradation cannot be attributed to only perception failures, by evaluating models on semant...
Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers. Across twe...
Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo et al.· 1 citation
Multimodal Large Language Models show strong vision-language capabilities, yet their geometry-centric reasoning remains insufficiently understood. Existing mathematical benchmarks typically rely on rasterized figures and cover limited geometric knowledge areas and response formats, making it difficult to assess whether...
Jing Wang, Yi-Tong Yang, Hao Dou et al.· 0 citations