Improving Retrieval-Augmented Generation Grounding Using Document Hierarchy Chunk Graph

Abstract

Retrieval-Augmented Generation (RAG) is widely used to enhance question-answering systems across various domains. However, while real-world source documents are inherently structured, conventional RAG approaches primarily rely on semantic similarity between isolated text chunks, which can overlook document hierarchy and limit retrieval effectiveness. To address this issue, this study introduces a document hierarchy-based Chunk Graph approach to improve retrieval grounding in RAG systems. The proposed framework preserves document hierarchy during chunking and models inter-chunk relationships using a weighted graph that combines structural proximity and semantic similarity. The approach was evaluated using the StructuredQA and CUAD benchmark datasets, with performance measured via Precision, Recall, and F1-Score. Experimental results demonstrate that the effectiveness of the Chunk Graph depends heavily on the source document format. On the highly structured StructuredQA dataset, the proposed method successfully connects fragmented information, improving the F1-Score from 47.62% to 50.23%. Conversely, on the CUAD dataset which consists of raw text with implicit hierarchy and no nested structure, the model becomes redundant and does not yield performance gains. These findings conclude that integrating structural-semantic relationships significantly improves context selection, specifically for documents with explicit hierarchical structures.