Structuring The Top-K Results: The Case for Narrative Extraction, Structuring, and Visualization in Scholarly Research Tools
Abstract
Scholarly search is dominated by top-k retrieval, which returns a ranked list of documents judged relevant in isolation, leaving the relationships among them for the reader to reconstruct. Retrieval-augmented generation leaves this structure implicit as well, since it synthesizes summaries over a top-k result set without exposing the structure that connects the retrieved work. We argue that this missing structure is central to scholarly sensemaking, where understanding literature means grasping how its ideas connect and develop. Building on the Narrative Trails algorithm, we present a narrative extraction pipeline that imposes this structure on the retrieved set. The proposed method uses a large language model to decompose a natural-language query into two narrative endpoints, retrieves a candidate set of documents, constructs a document graph, and extracts the maximum-capacity path between the endpoints as a coherent storyline, which a language model then synthesizes into a structured summary. Code for a demo of our system can be found at https://github.com/faustogerman/Scholar-Narrative-Extraction