Intent-Aware Adaptive Trust Retrieval-Augmented Generation (IAT-RAG): A Framework for Trustworthy Document Question Answering Using Open-Source Large Language Models
TL;DR
The paper summarizes the evolution of the conversational AI, Transformer-based LLMs, and RAG architectures, provides an illustrative evaluation protocol and literature-based comparison, and summarizes comparative results which suggest that the IAT-RAG system should outperform all seven baselines on the tasks of citation accuracy and retrieval F1-score, particularly on multi-hop and comparative queries.
Abstract
Large Language Models (LLMs) show impressive natural-language understanding and generation skills but are limited to knowledge that is pre-stored and trained, and can hallucinate when asked about private, domain-specific or newly generated documents. While conventional RAG and its recent variants (Hybrid RAG, Corrective RAG (CRAG), Adaptive-RAG, FLARE, RAPTOR) each make improvements to only a single stage of the retrieval pipeline, they do not effectively condition retrieval on the intent of the query and quantify the trustworthiness of evidence before generation. We introduce the Intent-Aware Adaptive Trust Retrieval-Augmented Generation (IAT-RAG) framework that integrates (i) query-intent classification to guide each query to a suitable retrieval strategy, (ii) an Adaptive Trust Score (ATS) to adjust the retrieval confidence based on the model's intent classification and (iii) an Evidence Quality Score (EQS) to filter the credibility and internal consistency of each retrieved passage before it is used for generation. Conventional RAG passes all top-k retrieved passages to the generator, whereas the passages supplied to the generator are only those that meet Combined Trust threshold from an intent perspective. The paper summarizes the evolution of the conversational AI, Transformer-based LLMs, and RAG architectures, provides an illustrative evaluation protocol and literature-based comparison, and summarizes comparative results which suggest that the IAT-RAG system should outperform all seven baselines on the tasks of citation accuracy and retrieval F1-score, particularly on multi-hop and comparative queries. The current work is a design and protocol stage contribution, whereas the actual implementation is fully open-source (Sentence-Transformer embeddings, a FAISS vector index, and an open-source instruction-tuned LLM), while complete empirical validation on real data, including statistical-significance testing, is identified as the next immediate step. Keywords: Retrieval-Augmented Generation, Large Language Models, Intent Classification, Adaptive Trust Score, Evidence Quality Score, Hallucination Mitigation, Open-Source LLMs.