CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA

Structured multi-level knowledge augmentation via small-to-large evidence-guided collaboration for knowledge-based VQA

Knowledge-based visual question answering requires models to connect question-relevant visual evidence with external knowledge for accurate reasoning. However, existing approaches still face two critical challenges: insufficient alignment between visual content and question intent, which leads to missing or irrelevant evidence, and ambiguity in entity-level semantics, especially for fine-grained or knowledge-intensive concepts. To address these issues, we propose an inference-time evidence augmentation framework for frozen-LLM-based KB-VQA. The framework uses lightweight vision–language models to construct structured textual evidence, which is subsequently provided to a frozen large language model (LLM) for final reasoning. We emphasize that the proposed method does not introduce a new LLM architecture or a new training mechanism; instead, it focuses on how question-relevant multimodal evidence can be systematically constructed, refined, and organized before LLM inference. The framework consists of four complementary evidence-construction modules: (1) a question-oriented image information extraction module that generates query-relevant visual descriptions to enhance visual–semantic alignment; (2) an entity enhancement module that introduces clarifying sub-questions to alleviate entity-level ambiguity; (3) a candidate-guided answer generation module that provides plausible answer cues to constrain the reasoning space; and (4) a contextual exemplar retrieval module that supplies relevant demonstrations to support knowledge-grounded inference. Experiments on OK-VQA and A-OKVQA show that the proposed method achieves 66.72% and 69.51% accuracy, respectively, outperforming strong baselines, while supplementary analyses examine its robustness, output-format reliability, and inference cost.

AutoDavis: Automatic and Dynamic Evaluation Protocol of Large Vision-Language Models on Visual Question-Answering

AutoDavis is introduced, a first-of-its-kind automatic and dynamic evaluation protocol that enables on-demand benchmarking of LVLMs across specific capability dimensions and shows effectiveness and reliability, offering a new paradigm for dynamic benchmarking of multimodal intelligence.

A visual question answering model based on entity knowledge selection

Dynamic Multi-Path Retrieval for Knowledge-based Visual Question Answering

Dynamic Multi-Path Retrieval for KB-VQA (DMRAG) is proposed, which re-trieves candidates through multiple retrieval paths that capture complementary visual and semantic cues and performs Question-Adaptive Gated Fusion to balance contributions from different modalities according to the query’s information need.

Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning

Knowledge-based Visual Question Answering aims to answer questions about an image by integrating external knowledge with visual and textual information. Recent approaches often rely on in-context learning to prompt Large Language Models (LLMs) with multimodal context in a zero-shot or few-shot manner. However, we observe that directly concatenating heterogeneous visual descriptions and retrieved knowledge into long, unstructured prompts often degrades reasoning performance, due to both excessive irrelevant context and the lack of explicit relational structure. In this paper, we propose an LLM-based Structured Context Reasoning (SCoRe) framework that infers both explicit and implicit relationships for prediction. SCoRe consists of three stages: Context Acquisition, which generates diverse visual notes and retrieves explicit knowledge via an efficient two-stage multimodal retrieval strategy; Context Selection, which filters relevant visual, explicit, and implicit knowledge using LLM-guided selection; and Context Compression, which performs Relational Logic Distillation (RLD) to transform raw text into explicit entity-relation triplets. These relational triplets serve as a concise and structured prompt for final answer prediction. Extensive experiments on the OK-VQA and A-OKVQA benchmarks demonstrate that SCoRe consistently outperforms state-of-the-art methods.

Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering

Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.