Beyond Document Retrieval: Architectural Challenges When LLM Agents Query Structured Enterprise Data

Abstract

Retrieval-augmented generation (RAG) has become a common architecture for connecting large language models to enterprise knowledge. Most RAG systems retrieve unstructured documents (PDFs, wiki pages, support tickets) and feed them to an LLM for summarization or question answering. A growing class of enterprise agents, however, must query structured data: relational databases, data warehouses, and analytics APIs where the answer is a computed result, not a retrieved passage. Structured-data querying forces decisions that a document-RAG pipeline never has to make. We group them into seven dimensions: retrieval semantics, authorization, intent recognition, entity resolution, evaluation, failure modes, and latency. For each dimension, we characterize the baseline assumption, explain its limitation for structured data, and describe a generic architectural pattern. As supporting evidence, a controlled synthetic study shows that a staged agent built on this framework eliminates the authorization violations of a direct translate-and-execute baseline under controlled conditions. The primary result is a design-oriented framework, an evaluation protocol, and a set of open problems for governed structured-data agents.