Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· 0 citations· 19 references
TL;DR
This work introduces MMHRAG, a novel Multi-Modal Hierarchical Retrieval-Augmented Generation framework that achieves cross-modal interaction on DocQA for the first time, and designs a Summarizing Agent to resolve logical conflicts, information redundancy, and granularity discrepancies among retrieved cross-modal evidence.
Abstract
The Document Question Answering (DocQA) task necessitates the synergistic interpretation of visual and textual information embedded within documents. Although Retrieval-Augmented Generation (RAG) has enhanced the capabilities of Large Vision-Language Models (LVLMs), existing approaches still encounter significant bottlenecks when processing large-scale documents: the inability to capture long-range contextual dependencies within non-textual modalities, the difficulty in facilitating interaction and mutual complementation between different modalities, and the inefficient integration of heterogeneous modal information. To address these challenges, we introduce MMHRAG, a novel Multi-Modal Hierarchical Retrieval-Augmented Generation framework that achieves cross-modal interaction on DocQA for the first time. First, we construct a cross-modal hierarchical retrieval tree via a bottom-up recursive clustering and summarization mechanism. A key innovation of our approach is the structural injection of visual information, where image semantics are integrated as high-level abstract summaries of textual segments, thereby bridging the semantic gap between modalities. Furthermore, we design a Summarizing Agent to resolve logical conflicts, information redundancy, and granularity discrepancies among retrieved cross-modal evidence. Extensive experiments across multiple multi-modal long-document benchmarks demonstrate that MMHRAG significantly outperforms state-of-the-art baselines, achieving superior accuracy and consistency in complex reasoning tasks.
Results demonstrate a 157% improvement in retrieval quality over a Naive-RAG baseline, with only 50 ms additional la tency, while Qwen2-VL-2B-Instruct achieved results comparable to cloud-based models in BERTScore, validate that open-source optimized SLMs, paired with advanced retrieval strategies, can provide competitive performance for document understanding without relying on cloud-based models.
HIEVI-RAG is introduced, a hierarchical, evidence-driven multimodal RAG framework for closed-domain document understanding that significantly outperforms existing open-source baselines and exceeds the strongest reported baseline by an average of 8.05% in accuracy.
Junyu Xiong, Yonghui Wang, Rongjian Gu et al.· 1 citation
Retrieval-Augmented Generation (RAG) has established itself as a compelling strategy for grounding large language model outputs in documentary evidence. However, production deployments continue to rely almost exclusively on homogeneous text corpora, even as enterprise repositories grow increasingly heterogeneous blending technical schematics, radiological images, annotated diagrams, and unstructured prose within the same archival system. This mismatch between system design and data reality motivates the present work. We propose a Multimodal RAG framework that unifies text and image retrieval through four tightly coupled components: a dual-stream embedding engine, a learned four-class AI query router, an adaptive confidence threshold, and a session-aware context store. Text is encoded with the allmpnet-base-v2 Sentence Transformer; images are embedded in the same 768-dimensional space via a domain-adapted CLIP ViT-L/14 model fine-tuned on approximately 120,000 technical and clinical text-image pairs. Concatenating 768-d vectors yields a 1,536-d composite query that drives a single approximate nearest-neighbor (ANN) search simultaneously across both modalities. A fine-tuned DistilBERT router assigns each query to one of four retrieval pathways text-only, image-only, hybrid, or conversational at 91.3% accuracy. Experiments were run on three enterprise corpora (engineering manuals, clinical case summaries, and legal paperwork) with the results being a mean F1@5 of 0.90, an increase of 22% over a dense text-only baseline, and a reduction of 49% in the number of hallucinations. The median first-token latency is 1.34s, which meets the interactive-use target deployment-contexts. These results show that it is possible to implement modality-aware retrieval in a simple and realistic setting with real-world organizational constraints and that the results are reproducible.
E.Vijayakumar, Ganesh A· 2026 4th International Confe...· 0 citations
Dynamic Multi-Path Retrieval for KB-VQA (DMRAG) is proposed, which re-trieves candidates through multiple retrieval paths that capture complementary visual and semantic cues and performs Question-Adaptive Gated Fusion to balance contributions from different modalities according to the query’s information need.
Zeyu Song, Yimin Deng, Yuxin Zhang et al.· 0 citations
Retrieval-augmented generation (RAG) has become a standard paradigm for knowledge-intensive question answering by grounding large language models (LLMs) in external evidence. However, open-domain multi-hop question answering (QA) remains challenging for two reasons. First, evidence dispersion across documents and non-contiguous spans means that critical bridge evidence can be weakly related to query and is easy to miss. Second, semantic-resolution mismatch complicates retrieval: coarser retrieval views offer better global coherence but may obscure the exact bridging detail, while finer-grained views highlight specific mentions but may omit the context needed to reveal the relation. In this paper, we propose MCoRe, a multi-entry complementary retrieval framework with reflection-guided iteration for multi-hop QA. To mitigate the semantic-resolution mismatch, MCoRe enables multi-entry complementary retrieval by indexing entry units at multiple semantic resolutions (entities, sentences, and summaries) with explicit links to chunk evidence, mapping all hits back to chunks, and fusing cross-resolution hits via chunk-level voting to form a compact evidence set for answer generation. To cope with evidence dispersion, MCoRe performs reflection-guided iteration: when evidence is insufficient, it identifies the missing bridge cue and issues a gap-focused follow-up query to recover it. Empirical results demonstrate the effectiveness of MCoRe, which consistently outperforms state-of-the-art baselines by 6.77 EM points and 8.79 F1 points averaged over three multi-hop QA benchmarks, with gains of up to 12.70 EM and 14.06 F1 points on 2Wiki.
Juxiang Zeng, Zhuohui Gao, Zhe Hou et al.· Proceedings of the 32nd ACM...· 0 citations
This paper formalizes the document structure as a Multimodal Hypergraph, utilizing hyperedges as unified semantic containers to encapsulate multi-way associations across text, images, and tables, thereby transcending point-to-point modeling and introducing an Anchor-driven Incremental Refinement mechanism.
Shenao Chen, Yidan Xu, Xiangmin Han et al.· 0 citations
