Improving legal reasoning reliability of large language models through citation resolution enhancements
TL;DR
This project seeks to create the infrastructure needed to make legal citation in OpenJustice rigorous and transparent through various enhancements to the extraction and connection of case metadata, and fully integrating the resulting citation graph into the reranker, maximizing LLM reasoning capabilities.
Abstract
As AI enters the high-stakes domain of law, firmly grounding large language models in accurate legal information is essential. OpenJustice, a legal knowledge and research platform, gives users case-specific reasoning flows that are built on trustworthy precedent case citations. However, those citations are only truly as trustworthy as the systems that retrieve and resolve them. A citation that is outdated, lacks authority, or silently conflicts with another ruling not only weakens answers, but also risks misleading the recipients. This project seeks to create the infrastructure needed to make legal citation in OpenJustice rigorous and transparent through various enhancements to the extraction and connection of case metadata. This will be leveraged by the reranker, a module that filters and structures retrieved case information before passing the most relevant information to the model's reasoning stage. The approach used LLMs themselves to process legal documents, identifying citation relationships that other methods such as regex cannot, including the treatment of each precedent case (followed, distinguished, not followed, etc.) and updates to their appeal histories (reversed, affirmed, quashed, etc.) as typed edges between case nodes. Together these relationships form a citation graph, a traversable web displaying how cases interact, that the reranker can draw on to weigh authority and surface conflicts automatically. Testing at a scale of 5000 recently filed cases produced over 70000 citation edges spanning 26 distinct relationship types, with citation linking accuracy of 97.2% against independent legal data. Token costs were also validated at this scale, using a smaller, three times more cost-efficient language model with a drop in accuracy of only ~1%. Future goals include further improving linking accuracy through a continued focus on citations that predate the standardized citation format set out in 1999, and fully integrating the resulting citation graph into the reranker, maximizing LLM reasoning capabilities. Faculty Supervisor: Xiaodan Zhu Supporting Instructors: Chu Fei Luo, Kevin Yu
