Ranking Passages in Relevant Documents Using LLMs
The ad hoc document retrieval task is to rank documents by their presumed relevance to a query. Most TREC benchmarks provide relevance judgments only at the document level, without indicating which parts of a document are actually relevant. Focused relevance judgments, which highlight relevant text at the character level, are valuable but scarce. In this work, we present a study of ranking approaches, including lexical, dense and zero-shot prompted large language models (LLMs), to rank passages in relevant documents based on the presumed fraction of relevant text they contain. Our analysis shows that LLM-based rankings are highly effective and outperform strong sparse and dense retrieval baselines. We demonstrate the merits of our approaches in utilizing relevance feedback: constructing relevance models from top-ranked passages in relevant documents yields performance that transcends that of relevance models constructed from the entire documents.