Decoupling retrieval quality from generative reasoning: A multi-dimensional benchmark of RAG architectures

Abstract

Traditional search engine returns ranked lists for humans to interpret. Retrieval Augmented Generation pipelines go further, feeding retrieved context directly into large language models to synthesize knowledge rather than simply surface it. This study addresses a focused question: When the generative layer is held constant, how do distinct retrieval architectures perform across diverse query types? We fix the LLM and chunking strategy while independently varying the retrieval backend and embedding model across three configurations: Memory Resident Specialist (MRS), Industrial Semantic Index (ISI), and Agile Hybrid Scaler (AHS) evaluated over a six-tier query taxonomy. Our results demonstrate that retrieval quality acts as a rigid performance ceiling for generative correctness. Notably, the "Efficiency Paradox" emerged: the AHS configuration, leveraging lightweight MiniLM embeddings with Pinecone cloud indexing, achieved parity with proprietary high-dimensional configurations on complex logical reasoning tasks. However, all three architectures share a critical weakness for conversational queries. Multi-turn contexts revealed a systemic failure stemming from the absence of anaphora resolution in naive RAG pipelines. These findings suggest that for RAG applications, strategic database indexing is a more critical performance lever than embedding parameter scale: architectural retrieval decisions matter more than model size.