Measured extraction performance varied substantially by matching definition, whereas exact-link decisions varied with the availability of matched terminology evidence, support separate evaluation of extraction, retrieval, evidence-grounded linking, and extension candidacy.
Abstract
Background: Clinical terminology pipelines must first extract candidate spans from narrative notes and then determine whether those spans map to existing concepts or warrant further review. Evaluation is difficult because span boundaries vary between annotators and because downstream decisions depend on the terminology evidence retrieved for each span. Objective: We evaluated clinical concept extraction, terminology linking across controlled evidence conditions, and ontology-extension triage for terms that remained unmatched after initial terminology screening. Methods: We conducted 3 complementary pilot evaluations that used distinct units of analysis and were analyzed separately. Study 1 compared 5 automated extraction pipelines and a union-merge analysis with 2 human annotation sets in 66 deidentified clinical notes from 3 health systems. Agreement was evaluated by exact string matching and BGE-large-en-v1.5 embedding matching. Study 2 evaluated 56 clinical spans, including 28 with reference Unified Medical Language System concepts and 28 adjudicated as unsuitable for ontology extension, under complete retrieval, matched-concept masking, and large language model-only inference, yielding 168 span-condition outputs. The graph retrieval pipeline used BGE-large-en-v1.5 embeddings, and the decision model was Gemma 3 27B. Study 3 applied full vector retrieval to 84 terms previously not matched in either UMLS or BioPortal. Results: In Study 1, interannotator exact-match F1 was 0.29 and embedding-match F1 was 0.75. Automated exact-match F1 scores ranged from 0.07 to 0.17; embedding-match F1 was highest for MedGemma (0.55), followed by Gemma (0.53), sci_md and SciBERT (each 0.43), and Llama 3.3 (0.32). In Study 2, complete retrieval returned a reference-matched link for 28/28 known-concept spans (100%; 95% CI, 87.9%-100%). Masking assigned POSSIBLE_CANDIDATES to all 28; large language model-only inference assigned POSSIBLE_CANDIDATES to 25/28 (89.3%) and LINKED to 3/28 (10.7%). Across the 3 evidence conditions, the same 12/28 unsuitable-extension spans were classified as NOT_MEANINGFUL (42.9%) and the same 16/28 as POSSIBLE_CANDIDATES (57.1%). In Study 3, the pipeline assigned PLAUSIBLE_EXISTING_CONCEPT to all 84 terms, none was flagged for extension, and top-candidate similarity averaged 0.914 (SD 0.027); extension status was not independently adjudicated. Conclusions: Measured extraction performance varied substantially by matching definition, whereas exact-link decisions varied with the availability of matched terminology evidence. In the follow-up sample, initial nonmatching did not establish ontology novelty: after semantic retrieval, the pipeline classified all 84 terms as plausible existing concepts and proposed none for extension. These findings support separate evaluation of extraction, retrieval, evidence-grounded linking, and extension candidacy.
Abstract Background Large language models (LLMs) can accurately classify biomedical documents, but strong benchmark performance does not establish that predictions are grounded in the supplied text. In biomedical literature tasks, titles, abstracts, digital object identifiers (DOIs), journal metadata, and trial identifiers may have been seen during pretraining and can trigger parametric knowledge or learned associations. Objective This study aimed to test whether oncology randomized trial success classification is driven by abstract evidence or by identifier-conditioned prior knowledge, and assess whether models follow counterfactual outcome evidence when it conflicts with original trial identifiers. Methods We evaluated 250 two-arm oncology randomized controlled trials from 7 major journals published between 2005 and 2023, each with a single primary endpoint and previously adjudicated positive or negative ground-truth label. The corpus included 58.4% (146/250) positive and 41.6% (104/250) negative trials. GPT-5.2, Gemini 3 Flash, and Claude Opus 4.5 were queried via vendor APIs under default settings using a single-token output instruction. For each trial, we created 5 deterministic input conditions: title+abstract, title only, DOI only, counterfactual title+abstract in which the primary endpoint outcome statement was minimally flipped, and the same counterfactual input paired with the original DOI to create an identifier-text conflict. Performance was assessed using valid format rate, accuracy, sensitivity, specificity, and F1-score. Results The models showed high format adherence, with valid prediction rates of 97.2% to 100%. In the title+abstract condition, all models achieved high and balanced performance (accuracy and F1-score=0.96-0.97; sensitivity=0.96-0.97; specificity=0.96-0.98). Removing evidence reduced performance stepwise: title-only accuracy and F1-score fell to 0.79 to 0.88, and DOI-only performance fell to 0.63-0.67, exceeding the 58.4% majority class baseline but indicating limited identifier-driven signal. Counterfactual edits were concentrated in outcome-bearing text, with the Results and Conclusions sections modified for all trials, whereas the titles and Methods sections required edits in only 5.2% (13/250) and 1.6% (4/250) of trials. Against inverted labels, models followed counterfactual evidence with near-ceiling performance (accuracy and F1-score=0.96-0.99). Reintroducing the original DOI caused little change for GPT-5.2 (accuracy and F1-score=0.99) but modestly reduced F1-scores for Gemini (0.97) and Claude (0.95), mainly through lower sensitivity. Conclusions The evaluated LLMs robustly followed explicit end point statements in abstracts, including when those statements contradicted original trial outcomes. However, above-chance title-only and DOI-only performance, together with small decrements under counterfactual DOI conflicts, showed that identifiers can carry predictive signal and occasionally compete with textual evidence. Progressive content removal combined with counterfactual identifier-text conflicts offers a practical, reproducible audit for grounding in biomedical LLM evaluations.
P. Windisch, C. Koechli, Fabio Dennstädt et al.· JMIR AI· 0 citations
Pathology reports anchor cancer diagnosis and staging, yet their narrative structure limits reliable translation into structured, machine-actionable knowledge, creating a bottleneck between expert interpretation and scalable clinical intelligence. Despite decades of clinical natural language processing (NLP) research, pathology text remains among the most complex and consequential sources of medical data to operationalize at scale. Large language models (LLMs) offer new approaches for reading, extracting, and interpreting these reports. We synthesize current LLM work in cancer pathology using a four-level capability framework across the pathology report data lifecycle: (level 1) text preparation and quality checks, (level 2) information extraction, (level 3) guideline-based clinical reasoning, such as TNM staging and registry coding, and (level 4) interpretive synthesis, such as explanations, summarization, or decision support. Rather than grouping studies by NLP task labels, this framework tracks how LLM applications progress from preprocessing and extraction toward higher-level interpretation and synthesis. We followed PRISMA-ScR guidelines and searched four databases through September 2, 2025, identifying 41 eligible studies. Most studies focus on level 2 tasks, with fewer addressing level 3 and level 4 tasks. Encoder-based models, including domain-specific variants such as BioBERT, were commonly used for structured extraction tasks, whereas generative models, including GPT, LLaMA, and Mistral-family models, were increasingly evaluated for prompting-based extraction, staging, and summarization. Reported performance was often high for well-defined extraction tasks, but external validation was uncommon, and metrics varied across studies, limiting direct comparison. Overall, the evidence suggests that success in lower capability levels does not consistently translate to higher-level reasoning, especially when reports are inconsistent, required staging inputs are missing, or clinical assumptions must be inferred, which helps explain gaps between benchmark results and practical adoption. Future work should prioritize robust multi-site validation, clinically meaningful error analysis, transparent evaluation, and privacy-preserving implementation strategies to support safe integration in oncology.
Maryam Seifaddini, Mohammad Beheshti, Steven Richberg et al.· Modern Pathology· 0 citations
Summary Biomedical information extraction (BIE) is fundamental for transforming unstructured biomedical text into structured, computable knowledge, yet the effectiveness of large language models (LLMs) remains limited by dataset heterogeneity and lack of unified benchmarks. We present InfoFlowEX, a unified framework for generalizable biomedical knowledge extraction with LLMs. InfoFlowEX incorporates an automated data integration pipeline using ontology-guided alignment to construct BIE-Corpus, a large-scale multi-domain benchmark unifying 40 public datasets for named entity recognition and relation extraction. We further introduce a task-conditioned schema instruction tuning strategy encoding 28 biomedical entity and relation types into a schema codebase, enabling LLMs to align heterogeneous annotations and generalize across settings. Finally, we evaluated InfoFlowEX in diverse applications, including evidence retrieval for question-answering, clinical diagnosis from electronic health records, and knowledge graph expansion. Results demonstrate that InfoFlowEX equips LLMs with robust adaptability, achieving consistent gains over baselines with minimal task-specific customization, highlighting InfoFlowEX for real-world biomedical applications.
Wuyang Lan, Siqi Zhang, Wenzheng Wang et al.· Cell Reports Medicine· 0 citations
Medical guidelines encode rich, evidence-based decision logic, yet the specific decision artifact a clinician needs is hard to locate within a guideline, let alone across guidelines covering plausible diseases and treatments. While guideline passages have supported end-to-end question answering, flowcharts remain largely underused in decision support despite their ability to encode actionable clinical pathways. We therefore introduce Case2Flow, a task designed to retrieve the most relevant guideline flowchart for a given patient case from a collection of guideline documents. To support it, we construct FlowAtlas, a curated corpus of 202 flowcharts extracted from 2,080 medical guidelines, together with a pipeline that synthesises 1,911 aligned case-flowchart pairs. Our evaluation of multimodal retrieval methods reveals systematic failure modes, including overreliance on keywords and spurious token-patch matches induced by uninformative background regions in flowcharts. Motivated by this, we propose CRISP, a training-free scoring method that sharpens late-interaction retrieval by suppressing uninformative patches, discounting ambiguous token matches, and incorporating bidirectional query-image alignment. CRISP improves Recall@1 by up to 18.71 percentage points, while a blinded physician assessment on published case narratives provides preliminary feasibility evidence beyond synthetic queries.
Jiale Wei, Yufan Chen, Alexander Jaus et al.· 0 citations
Precision medicine terminology is increasingly used across clinical, laboratory, payer, regulatory, and policy settings; however, distinct terminology may be applied interchangeably or inconsistently across stakeholders, and foundational concepts may not be uniformly understood across multidisciplinary audiences with varying levels of expertise. This misalignment may create confusion for stakeholders as they attempt to navigate guideline recommendations and payer coverage policies, potentially impacting patient access to testing and treatment. In 2025, AMCP held a 2-day partnership forum on the topic of advancing precision medicine in oncology. Forum participants recommended development of a consolidated lexicon of commonly used precision medicine terms to support clearer communication and foundational understanding across stakeholders and to clarify distinctions between related concepts. Nineteen definitions were ultimately generated using a natural language processing process, an artificial intelligence approach used to extract information and derive meaning from large text samples. A comprehensive literature review was first conducted to develop an index of publicly available precision medicine terminology sources. Natural language processing-based textual analysis was then used to systematically evaluate terminology usage patterns across the index, including both consistent and inconsistent usage of terms across sources. Draft definitions generated through this process were subsequently reviewed and refined by forum participants. The resulting lexicon included relevant terms related to (1) foundational concepts in precision medicine; (2) biological foundations and molecular variation; (3) biomarker and molecular testing; (4) testing technologies, specimen collection, and regulatory classification; and (5) testing approach. The objective of this work was to organize commonly used precision medicine terms into a single reference framework, a foundational precision medicine reference for managed care stakeholders, and to clarify distinctions between related concepts where appropriate.
Tianyi Wang, Kimberly Tsai, Steven S Kheloussi et al.· Journal of Managed Care & Sp...· 0 citations
Current LLMs do not achieve inter-rater reliability levels comparable to medical professionals in clinical information extraction from ENT documentation, suggesting they are best suited for initial extraction with human verification rather than autonomous operation.
L. Barrett, N. Joshi, A. S. North et al.· medRxiv· 0 citations