Improving Tree Traversal in Hierarchical RAG via Reasoning-Aware Approach with Interleaving Chain-of-Thought

Abstract

Retrieval-Augmented Generation (RAG) has improved the factual grounding of large language models; however, conventional retrieval strategies remain limited for long-document question answering, as relevant information is often distributed across multiple document sections and may be inferential rather than lexically similar to the query. This paper proposes a reasoning-aware hierarchical traversal mechanism that interleaves chain-of-thought generation at each depth-first search node evaluation step, incorporating the resulting reasoning embedding into a combined node scoring function alongside query similarity. The hierarchical tree is constructed through iterative chunking, embedding, clustering, and summarization; retrieval is then guided by this combined score under a dual-threshold pruning mechanism that adaptively controls traversal depth and breadth. The proposed approach is evaluated on the NarrativeQA and QuALITY benchmarks against a semantic similarity-based traversal baseline. On QuALITY, the method achieves marginal gains in overall accuracy (+0.7%) and F1 (+0.1%), while substantially reducing the abstain rate from 4.1% to 1.8%, with improvement concentrated on normal-difficulty questions; on hard questions, the baseline outperforms the proposed method. On NarrativeQA, ROUGE-L F1, BLEU-1, and BLEU-4 improve slightly, though METEOR decreases, reflecting inconsistent metric-level effects. These results suggest that step-wise CoT reasoning can improve node selection quality in hierarchical traversal, with gains most evident on standard-difficulty questions. Performance on hard questions and metric-level consistency across NarrativeQA remain open challenges, indicating that the quality of the generated reasoning signal is a key bottleneck for further improvement.