Multimodal Hybrid Retrieval-Augmented Generation for Scientific Document Understanding using Open-Source SLMs

Optimized Hybrid Retrieval-Augmented Generation Framework using Semantic Search and Re-ranking for Reliable LLM Responses

Building question-answering systems that can read a document and answer naturally phrased questions about it is difficult when retrieval is left to either keyword matching or dense vector search alone, since each method has blind spots that surface as missed context, near-miss answers, or content invented by the underlying language model. This paper describes an optimized hybrid Retrieval-Augmented Generation (RAG) pipeline built to reduce these failure modes by combining two complementary retrieval signals: dense semantic similarity computed over a FAISS vector index, and sparse lexical scoring computed with BM25. Candidates returned by both retrievers are merged and passed through a Cross-Encoder re-ranking stage that scores each query-passage pair jointly, pushing the most contextually relevant chunks to the top before they reach the language model. Final answers are produced by Google’s Gemini model under a prompt that restricts it to the supplied context, which keeps the output tied to the source document rather than to whatever the model already “knows.” The pipeline is exposed through a Streamlit application that lets a user upload a PDF and ask questions about it in plain language, returning each answer alongside a confidence estimate and the page it came from. Evaluation on a multi-page technical PDF document shows that the hybrid retrieval and re-ranking stages together raise retrieval precision and reduce irrelevant or unsupported answers compared with retrieval limited to a single method, supporting the use of this approach for reliable, document-grounded question answering.

Multi-Modal Hierarchical Retrieval-Augmented Generation for Document Question Answering

This work introduces MMHRAG, a novel Multi-Modal Hierarchical Retrieval-Augmented Generation framework that achieves cross-modal interaction on DocQA for the first time, and designs a Summarizing Agent to resolve logical conflicts, information redundancy, and granularity discrepancies among retrieved cross-modal evidence.

A Multimodal Retrieval-Augmented Generation Framework for Context-Aware Semantic Querying

Retrieval-Augmented Generation (RAG) has established itself as a compelling strategy for grounding large language model outputs in documentary evidence. However, production deployments continue to rely almost exclusively on homogeneous text corpora, even as enterprise repositories grow increasingly heterogeneous blending technical schematics, radiological images, annotated diagrams, and unstructured prose within the same archival system. This mismatch between system design and data reality motivates the present work. We propose a Multimodal RAG framework that unifies text and image retrieval through four tightly coupled components: a dual-stream embedding engine, a learned four-class AI query router, an adaptive confidence threshold, and a session-aware context store. Text is encoded with the allmpnet-base-v2 Sentence Transformer; images are embedded in the same 768-dimensional space via a domain-adapted CLIP ViT-L/14 model fine-tuned on approximately 120,000 technical and clinical text-image pairs. Concatenating 768-d vectors yields a 1,536-d composite query that drives a single approximate nearest-neighbor (ANN) search simultaneously across both modalities. A fine-tuned DistilBERT router assigns each query to one of four retrieval pathways text-only, image-only, hybrid, or conversational at 91.3% accuracy. Experiments were run on three enterprise corpora (engineering manuals, clinical case summaries, and legal paperwork) with the results being a mean F1@5 of 0.90, an increase of 22% over a dense text-only baseline, and a reduction of 49% in the number of hallucinations. The median first-token latency is 1.34s, which meets the interactive-use target deployment-contexts. These results show that it is possible to implement modality-aware retrieval in a simple and realistic setting with real-world organizational constraints and that the results are reproducible.

An Intelligent PDF Question-Answering System; A Retrieval-Augmented Generation Approach

Due to the surge in digital document creation, intelligent systems that can effectively retrieve accurate information from unstructured PDF content have become a necessity. Current keyword-based search techniques lack semantic meaning, while large language models (LLMs), when utilized individually, produce hallucinated results when they are not contextually aware of document content. In this paper, we introduce a novel real-time question answering system for PDF content using a lightweight approach to Retrieval-Augmented Generation (RAG). Our system combines a HuggingFace sentence transformer model for semantic embeddings with a FAISS vector similarity search for efficient retrieval, leveraging a Groq-hosted LLaMA 3.3 70B model for context-grounded answer generation. The system is built using LangChain for orchestration and Streamlit for web-based deployment, and requires no GPU, no fine-tuning, and no pre-indexed corpus. Evaluation across five complementary metrics—Exact Match, Token-level F1, BLEU, ROUGE, and semantic similarity—demonstrates strong alignment between generated and reference answers, with a peak semantic similarity of 0.8709 and a mean Token F1 of 0.4701, confirming the viability of this approach for real-world document intelligence.

FUSE-RAG: A Unified Architecture for Retrieval-Augmented Generation

Large language models may suffer from insufficient context use and unsupported generation in question answering tasks that require external knowledge. This study compares the main strategies affecting retrieval and generation performance in retrieval-augmented generation systems within a common experimental setting. In addition to a standard dense retrieval baseline, we evaluate multi-query, hypothetical document, hypothetical question, sparse-dense hybrid retrieval, and chunk compression, and we also propose an integrated method (FUSERAG) that combines these components. Experiments are conducted on a dataset containing 223 documents, 7933 chunks, and 1488 question-answer instances. Results show that the proposed method achieves the highest Mean Reciprocal Rank, nDCG at 5, and recall at 5 at the retrieval level, while the sparse-dense hybrid approach yields the best generation results.

STaR: a soft-labeling and triplet-aware retriever for efficient retrieval-augmented QA