2026· Journal of Management in Engineering· Vol 42· 0 citations· 32 references
TL;DR
This paper presents a self-reflective multimodal RAG-assisted VLM pipeline that augments existing VLMs with an external multimodal RAG mechanism, moving beyond text-only retrieval by jointly leveraging visual representations and semantic summaries to identify relevant evidence pages.
Abstract
A deep comprehension of construction regulatory documents is essential for successful construction engineering and management. Although vision–language models (VLMs) hold tremendous potential for achieving human-level understanding, their practical deployment remains constrained by input size limitations. Even when integrated with retrieval-augmented generation (RAG), most efforts process visual elements commonly found in regulatory documents as pure texts and struggle to verify the trustworthiness of the retrieved supporting evidences, increasing vulnerability to hallucination. To address these drawbacks, we present a self-reflective multimodal RAG-assisted VLM pipeline that extends standard VLMs from single-pass document interpretation to multiround retrieval-grounded reasoning. Specifically, it augments existing VLMs with an external multimodal RAG mechanism, moving beyond text-only retrieval by jointly leveraging visual representations and semantic summaries to identify relevant evidence pages. Furthermore, an iterative self-reflection is incorporated to evaluate evidence sufficiency and trigger adaptive question refinement, thus enhancing response reliability. This pipeline was evaluated using a 794 question-answering pairs compiled from 12 divisions of a construction specification, covering three representative question types—information extraction, single-modal reasoning, and multimodal reasoning. Results showed substantial performance gains over traditional, single-model, and multimodal RAG-assisted baselines. More importantly, fine-grained analyses revealed that the proposed pipeline is particularly effective for complex engineering management scenarios—such as interpreting multimodal regulatory requirements, verifying compliance conditions embedded across late-document pages, and synthesizing information from text, tables, and figures. By enabling accurate, context-aware comprehension of lengthy construction regulatory documents without additional model training, the proposed pipeline offers a practical and generalizable tool for engineering management workflows. It has the potential to support compliance verification, reduce interpretation errors, and enhance the timeliness and quality of decision-making in construction management.
Retrieval-Augmented Generation (RAG) has established itself as a compelling strategy for grounding large language model outputs in documentary evidence. However, production deployments continue to rely almost exclusively on homogeneous text corpora, even as enterprise repositories grow increasingly heterogeneous blending technical schematics, radiological images, annotated diagrams, and unstructured prose within the same archival system. This mismatch between system design and data reality motivates the present work. We propose a Multimodal RAG framework that unifies text and image retrieval through four tightly coupled components: a dual-stream embedding engine, a learned four-class AI query router, an adaptive confidence threshold, and a session-aware context store. Text is encoded with the allmpnet-base-v2 Sentence Transformer; images are embedded in the same 768-dimensional space via a domain-adapted CLIP ViT-L/14 model fine-tuned on approximately 120,000 technical and clinical text-image pairs. Concatenating 768-d vectors yields a 1,536-d composite query that drives a single approximate nearest-neighbor (ANN) search simultaneously across both modalities. A fine-tuned DistilBERT router assigns each query to one of four retrieval pathways text-only, image-only, hybrid, or conversational at 91.3% accuracy. Experiments were run on three enterprise corpora (engineering manuals, clinical case summaries, and legal paperwork) with the results being a mean F1@5 of 0.90, an increase of 22% over a dense text-only baseline, and a reduction of 49% in the number of hallucinations. The median first-token latency is 1.34s, which meets the interactive-use target deployment-contexts. These results show that it is possible to implement modality-aware retrieval in a simple and realistic setting with real-world organizational constraints and that the results are reproducible.
E.Vijayakumar, Ganesh A· 2026 4th International Confe...· 0 citations
Retrieval-Augmented Generation (RAG) has become a robust architecture for grounding large language models (LLMs) in trusted knowledge. However, standard RAG systems exhibit a structural limitation: retrieved documents carry their own communication styles-professional jargon, formal tone, or academic writings-that shape the behavior of a RAG system before any tone instructions are processed, often causing the system to ignore user requests for a specific tone. We term this phenomenon contextual decoupling, in which a system optimises for factual accuracy while remaining decoupled from the social or operational context of the recipient. Building on prior research in public health peer-support communities, we identify three communicative misalignment-linguistic, cognitive, and relational-that can persist even when retrieval is relevant and the generated response is factually accurate. We conceptualise these as failures of communicative transformation, which remain largely invisible to accuracy-centred RAG evaluation metrics. To address this gap, we propose Tone-Aware RAG (TA-RAG), a conceptual architectural framework that positions communicative alignment alongside factual accuracy as a core design objective. TA-RAG operationalises four constraints-stigma-free language, readability alignment, recipient-sensitive adaptation, and empathetic framing-across the retrieval, context construction, generation, and constraint validation phases in the proposed RAG pipeline. We further highlight an evaluation agenda for jointly assessing factual fidelity and communicative alignment, and identify open challenges. We argue that tone awareness should be treated not as an optional refinement, but as a present design imperative for RAG systems operating in socially sensitive and high-stakes contexts.
Results demonstrate a 157% improvement in retrieval quality over a Naive-RAG baseline, with only 50 ms additional la tency, while Qwen2-VL-2B-Instruct achieved results comparable to cloud-based models in BERTScore, validate that open-source optimized SLMs, paired with advanced retrieval strategies, can provide competitive performance for document understanding without relying on cloud-based models.
HIEVI-RAG is introduced, a hierarchical, evidence-driven multimodal RAG framework for closed-domain document understanding that significantly outperforms existing open-source baselines and exceeds the strongest reported baseline by an average of 8.05% in accuracy.
Junyu Xiong, Yonghui Wang, Rongjian Gu et al.· 1 citation
This work proposes ViSAGE, a multimodal agentic memory framework that constructs self-correcting, entity-centric memories and applies bidirectional memory refinement to propagate delayed identity evidence, retroactively unifying historical records and improving future reasoning.
Xinkui Zhao, Enbo Chen, Yifan Zhang et al.· 0 citations
Document question answering over scanned pages requires two coupled abilities: (i) canonicalizing complex layouts into a faithful textual structure, and (ii) selecting and reasoning over query-relevant evidence from that structure. Most existing pipelines decouple OCR from retrieval-augmented reasoning and optimize OCR for global reconstruction, which often misaligns with evidence needs and causes brittle grounding in multi-page settings. We propose UniDocVLM, an end-to-end framework that unifies OCR and visual RAG within a single vision-language model: the model first generates a structured parse of retrieved pages, then activates question-relevant evidence from the parse to support grounded reasoning and answering. To train UniDocVLM under heterogeneous supervision, we introduce a unified JR-GRPO reinforcement learning recipe with lightweight, verifiable rewards, including format, layout-aware OCR, evidence-consistency, and answer-correctness signals, and route them to the corresponding parts of the output to improve credit assignment and reduce interference. Experiments on multi-page document QA benchmarks show that UniDocVLM yields more reliable evidence grounding and improves downstream accuracy under complex layouts.
Zongsheng Cao, Anran Liu, Jun Xie et al.· Proceedings of the 32nd ACM...· 0 citations