The findings demonstrate that while large pretrained VLMs possess strong zero-shot baselines for structured layouts, their performance strongly decreases on visually complex layouts of infographics and slides, and that visual understanding is the main bottleneck for DocVQA, not a lack of knowledge from the VLMs.
Abstract
Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents. Although Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks, their robustness and transferability to different document domains remains underexplored. In this study, we present a comprehensive evaluation of 8 open-source pretrained VLMs on DocVQA in three different document domains: industrial documents of varying type, infographics, and presentation slides. We systematically assess model performance under zero-shot evaluations, fully supervised finetuning with inter- and intra-dataset evaluations, and few-shot learning evaluations of knowledge transfer between domains. Our findings demonstrate that while large pretrained VLMs possess strong zero-shot baselines for structured layouts, their performance strongly decreases on visually complex layouts of infographics and slides. Although parameter scaling is a dominant factor on performance, supervised finetuning yields higher relative gains in smaller architectures. Furthermore, our cross-domain and few-shot experiments show that visual understanding is the main bottleneck for DocVQA, not a lack of knowledge from the VLMs. Using 50 target domain samples, the models finetuned in DocVQA with datasets of different domains rapidly adapt to the target domain documents, even surpassing their fully supervised counterparts in some cases.
AutoDavis is introduced, a first-of-its-kind automatic and dynamic evaluation protocol that enables on-demand benchmarking of LVLMs across specific capability dimensions and shows effectiveness and reliability, offering a new paradigm for dynamic benchmarking of multimodal intelligence.
Han Bao, Yue Huang, Yanbo Wang et al.· Proceedings of the 32nd ACM...· 0 citations
It is shown that paragraph supervision enables effective use of long token sequences, whereas caption-only training degrades beyond 60 tokens, and paragraph supervision consistently benefits long-description benchmarks and hard negatives prove detrimental in text-only fine-tuning.
Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz et al.· 0 citations
MonkeyOCRv2, a visual-text pretrained model for document AI, is presented, and a pretraining strategy that jointly learns image-to-text generation and pixel-level document reconstruction is proposed: the former aligns visual representations with textual content, while the latter preserves character strokes and layout details.
Yuliang Liu, Zhang Li, Ziyang Zhang et al.· 1 citation
Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.
Abhigya Verma, Khyati Mahajan, Amit Kumar Saha et al.· 0 citations
This work introduces three tasks/objectives for reverse localization of text as an instruction-tuning mechanism, where the model is guided to extract textual content based on spatial localization cues, thereby enhancing its spatial grounding ability.
Visual question answering models often encounter challenges related to data biases and exhibit limited performance in specialized domains such as cultural heritage, where recognizing fine-grained textures is crucial. In this study, we propose a novel spatial-frequency invariant semantic learning model designed to overcome these limitations. By incorporating frequency-domain features as an additional modality and employing invariant feature learning techniques, our model effectively reduces bias without relying on external datasets. The proposed model extracts invariant representations across textual, spatial, and frequency domains, thereby filtering out spurious correlations. Comprehensive experiments on benchmark datasets, including VQA-CP v2 and GQA-OOD, demonstrate that our model achieves state-of-the-art results. Furthermore, our model exhibits enhanced robustness when applied to cultural heritage datasets, proficiently handling complex visual textures and multimodal reasoning tasks. This model enhances the capabilities of visual question answering systems in identifying artistic materials and techniques, providing a robust solution tailored to domain-specific applications.