Tool-supported Expert Validation of CONTEXT-sensitive Data Extraction from Scientific Articles
TL;DR
ContEXT is presented, a domain-agnostic tool that enables in-context expert judgment over LLM-extracted outputs, enabling efficient in-context assessment, robust to typical LLM noise such as paraphrasing and hallucination.
Abstract
Extracting structured information from scientific literature (e.g., in life sciences) is a considerably valuable and knowledge-intensive task. Relevant target information is often highly context-sensitive and scattered across unstructured, heterogeneous documents spanning text, tables, figures, and supplementary materials. For instance, statistical measures may not be apparent in the text, requiring the interpretation of multiple methodological and experimental details across documents. Large Language Models (LLMs) offer promising support for automating this process, yet they remain prone to error, making expert curation essential. Validation is, however, expensive and time-consuming, since extracted values must be traced back to their precise location in long, multi-document sources. Here, we present CONTEXT, a domain-agnostic tool that enables in-context expert judgment over LLM-extracted outputs. Given a pipeline that returns, for each target field and scientific article, the identified value with its source location --file, section, subsection, and exact passage-- our tool maps this output back to the original document, enabling efficient in-context assessment, robust to typical LLM noise such as paraphrasing and hallucination. We demonstrate CONTEXT's utility with a use case in genetics, involving the curation of metadata for polygenic risk score models. Validated extractions are exported as structured ground-truth records, providing a rich annotation format for downstream use in knowledge-intensive tasks. Demonstration video and source code are available[9, 10].
