TL;DR
EKS is a novel framework that leverages entity relations in commonsense knowledge graphs to dynamically generate knowledge sentences relevant to both visual and textual entities and formulates knowledge selection as a relevance scoring problem, where semantic similarity is used to measure the relevance between knowledge candidates and the input question.
This work proposes a TextVQA framework that integrates external prior knowledge to support multimodal reasoning and demonstrates that this approach effectively leverages external knowledge to improve performance on knowledge-intensive questions.
Dynamic Multi-Path Retrieval for KB-VQA (DMRAG) is proposed, which re-trieves candidates through multiple retrieval paths that capture complementary visual and semantic cues and performs Question-Adaptive Gated Fusion to balance contributions from different modalities according to the query’s information need.
Knowledge-based Visual Question Answering aims to answer questions about an image by integrating external knowledge with visual and textual information. Recent approaches often rely on in-context learning to prompt Large Language Models (LLMs) with multimodal context in a zero-shot or few-shot manner. However, we obser...
KBMR is proposed, the first MLLM-based embedding retriever tailored for KB-VQA, and an MLLM-based semantic discriminator that generates continuous entity-consistency weights is introduced to tackle the challenge of noisy supervision in Wikipedia-scale retrieval.
This work proposes an inference-time evidence augmentation framework for frozen-LLM-based KB-VQA that focuses on how question-relevant multimodal evidence can be systematically constructed, refined, and organized before LLM inference.
This work proposes a visual question answering model based on autonomous multi-modal knowledge learning, in which large-scale multi-modal knowledge is first accumulated by constructing pseudo data, and then dynamically filtered and iteratively updated through an autonomous updating strategy to reduce noise and improve...
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
