AutoPrunedRetriever: Persistent Reasoning Graphs for Efficient and Extensible Retrieval-Augmented Generation

Abstract

Retrieval-augmented generation (RAG) has become a standard approach for knowledge-intensive LLM tasks. Complex reasoning remains challenging because queries often require composing dispersed evidence through multiple dependent reasoning steps. Yet existing RAG systems process queries independently, repeatedly retrieving overlapping evidence and reconstructing established reasoning. Preserving such reasoning makes previously retrieved evidence and intermediate results reusable for subsequent queries. However, inference-time reuse becomes increasingly costly as the history grows, motivating a compact representation that preserves useful information without substantial loss in quality. To address these challenges, we present AutoPrunedRetriever (APR), a graph-based RAG system that encodes corpus facts and query-derived reasoning in a shared symbolic codebook. The codebook serves both as the retrieval index and as persistent memory, representing facts, questions, and answers as compact edge sequences, so that the previously established structure can be directly retrieved and reused. At inference time, APR retrieves query-specific reasoning paths and constructs compact prompts, while incremental consolidation keeps the growing memory compact. For deployments where the retriever cannot be replaced, we introduce AutoPruned Layer (APL), which leaves host retrieval unchanged but replaces the retrieved context with the corresponding entries in the shared symbolic codebook, writing each question and answer back as a compact structure. Across five complex-reasoning benchmarks, APR improves accuracy over the strongest baseline by 1.8-10.8 points while using up to 96% fewer input tokens. APL reduces generator input by 60-96% across host systems and preserves answer quality on document-style contexts.