Structured multi-level knowledge augmentation via small-to-large evidence-guided collaboration for knowledge-based VQA

Abstract

Knowledge-based visual question answering requires models to connect question-relevant visual evidence with external knowledge for accurate reasoning. However, existing approaches still face two critical challenges: insufficient alignment between visual content and question intent, which leads to missing or irrelevant evidence, and ambiguity in entity-level semantics, especially for fine-grained or knowledge-intensive concepts. To address these issues, we propose an inference-time evidence augmentation framework for frozen-LLM-based KB-VQA. The framework uses lightweight vision–language models to construct structured textual evidence, which is subsequently provided to a frozen large language model (LLM) for final reasoning. We emphasize that the proposed method does not introduce a new LLM architecture or a new training mechanism; instead, it focuses on how question-relevant multimodal evidence can be systematically constructed, refined, and organized before LLM inference. The framework consists of four complementary evidence-construction modules: (1) a question-oriented image information extraction module that generates query-relevant visual descriptions to enhance visual–semantic alignment; (2) an entity enhancement module that introduces clarifying sub-questions to alleviate entity-level ambiguity; (3) a candidate-guided answer generation module that provides plausible answer cues to constrain the reasoning space; and (4) a contextual exemplar retrieval module that supplies relevant demonstrations to support knowledge-grounded inference. Experiments on OK-VQA and A-OKVQA show that the proposed method achieves 66.72% and 69.51% accuracy, respectively, outperforming strong baselines, while supplementary analyses examine its robustness, output-format reliability, and inference cost.