Aug 2026· Network Modeling Analysis in Health Informatics and Bioinformatics· Vol 15· 0 citations· 97 references
TL;DR
This review provides researchers and practitioners with a structured framework for method selection based on their specific constraints and identifies six prioritized research directions for future investigation, identifying critical research gaps including the preservation of multi-word clinical concepts, scarce evaluation in real-world clinical workflows, and persistent hallucination risks in model-based extraction.
The growth of biomedical literature poses significant challenges for researchers conducting systematic and scoping reviews. In fields such as the use of digital biomarkers for the treatment of heart failure and cardiovascular disease, manually screening thousands of papers is time-consuming and not scalable. To address this problem, we developed AI-assisted tools for large scale analysis and structured knowledge extraction from biomedical research articles. Our approach emphasizes cross-graph analytics, facilitating the exploration of relationships among key biomedical concepts, including digital biomarkers and digital health technologies. By automating the extraction and structuring of knowledge from unstructured text, our system aims to accelerate evidence of synthesis and support more comprehensive and up-to-date reviews in rapidly evolving biomedical domains.We developed a knowledge graph generation pipeline that extracts subject-predicate-object triplets representing scientific claims from research articles. To address redundancy caused by linguistic variation across documents, we implemented a multi-stage consolidation process focused on normalizing entities and relations. This process begins by validating and filtering extracted triplets, then applies lexical normalization to unify entity representations by removing stop words, resolving variants, and merging acronyms with their full forms. Entity types and relations are similarly standardized to ensure uniformity and clarity. The pipeline is designed to be modular and extensible, allowing for the integration of additional normalization strategies or domain-specific ontologies as needed.To further consolidate equivalent triplets, we leverage embedding-based semantic similarity, enabling the merging of semantically similar entities and relationships even when expressed differently across sources. Additionally, our pipeline utilizes biomedical ontologies such as RxNorm and MeSH to map entities to standardized concept identifiers. This ontology-based normalization ensures that references to the same biomedical concept are unified, regardless of linguistic or spelling differences. We evaluated our approach on a pilot corpus of 150 biomedical research articles, processing over 2,000 extracted triplets. The normalization pipeline reduced the number of unique entity variants by more than 50%, consolidating these into approximately 900 unique, semantically unified relationships. Manual review of a representative sample indicated entity normalization accuracy in the range of 90-95%. In conclusion, the integration of lexical, semantic, and ontology-based normalization offers a robust framework for reducing ambiguity and improving the interoperability of the resulting knowledge graphs. Moreover, this structured and unified representation of knowledge facilitates systematic reviews, meta-analyses, and data-driven decision-making in biomedical science, while enabling advanced querying, trend analysis, and the identification of novel associations between biomedical concepts
S. Krovvidi, Finn Vos, Laurent D. Hasson et al.· AHFE International· 0 citations
Background/Objectives: Pharmacovigilance workflows rely heavily on unstructured text across diverse sources. Here, we systematically reviewed how large language models (LLMs) are being explored as support tools for adverse drug reaction (ADR) detection, extraction, triage, and documentation, highlighting their potential for precision medicine and big data-enabled safety monitoring. Methods: Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 guidelines, we systematically searched PubMed, Scopus, and Web of Science for studies published between January 2022 and March 2026. Ultimately, 83 empirical studies satisfied the inclusion criteria. A narrative synthesis was conducted to address methodological heterogeneity across these studies. Results: LLM applications were concentrated in constrained information-extraction and classification tasks, including signal evaluation, clinical-note extraction, social media surveillance, and literature screening. Quantitative performance varied substantially by system design: error-correction prompting yielded an F1-score of 0.921 for ADR named entity recognition, whereas retrieval-augmented generation improved data-retrieval accuracy from 8.3% to 78.3%. Most studies were retrospective, benchmark-based, or proof-of-concept evaluations. Across 581 paired pre-consensus domain judgements, observed inter-rater agreement was 90.4% and Cohen’s κ was 0.837 (95% CI 0.772–0.895). Hallucination, low specificity, prompt sensitivity, narrow datasets, and weak external validation remained common limitations. Conclusions: Current evidence supports supervised, task-specific applications of LLMs for extraction, triage, retrieval, and documentation rather than autonomous pharmacovigilance decision-making. Prospective evaluation, external validation, transparent reporting, and accountable human oversight are required before high-stakes clinical or regulatory deployment.
Tae You Kim, Won-Sik Oh, Dong-Hwa Jeong· Diagnostics· 0 citations
BACKGROUND
Real-world data (RWD) in unstructured electronic health records (EHRs) is crucial for understanding complex diseases like cancer, but extracting structured information is challenging due to linguistic variability, semantic complexity, and privacy concerns. This study evaluates the performance of four small, locally deployable language models for information extraction from Italian EHRs.
METHODS
We examine three prompting strategies (zero-shot, few-shot, and annotated few-shot) across English and Italian, involving clinicians with varying expertise to assess the impact of prompt design on accuracy. We evaluate the performance of four open-source small language models (SLMs) for clinical information extraction from Italian electronic health records (EHRs) in the APOLLO 11 trial on non-small cell lung cancer (NSCLC). The extraction protocol involves four steps: problem definition, data preprocessing, Large Language Model (LLM)-based information extraction, and output evaluation.
RESULTS
We show that general-purpose models (e.g., LLaMA 3.1 8B) outperform biomedical models in most tasks, particularly in extracting binary features. Multiclass variables such as TNM (Tumor, Node, Metastasis) staging, PD-L1 (Programmed death-ligand 1), and ECOG-PS (Eastern Cooperative Oncology Group-Performance Status) are more difficult due to implicit language and lack of standardization. Few-shot prompting and native-language inputs significantly improve performance and reduced hallucinations. Clinical expertise enhances consistency in the extraction, particularly among students using annotated examples.
CONCLUSIONS
The study confirms that privacy-preserving SLMs can be deployed locally for efficient and secure cancer data extraction. Findings highlight the need for hybrid systems combining SLMs with expert input and underline the importance of aligning clinical documentation practices with SLM capabilities. This is the first study to benchmark SLMs on Italian EHRs and investigate the role of clinical expertise in prompt engineering, offering valuable insights for the future integration of SLMs into real-world clinical workflows.
Federica Corso, V. Peppoloni, L. Mazzeo et al.· Communications Medicine· 0 citations
Clinical text is an important part of healthcare systems because it is used to store and manage patient information in documents such as discharge summaries, doctor notes, and diagnostic reports. Among these documents, discharge summaries are especially important because they provide a brief overview of a patient’s diagnosis, treatment procedures, medications, and follow-up instructions after hospitalization. These summaries are also useful for healthcare research and medical data analysis. However, strict privacy regulations and hospital policies restrict access to real clinical records, making it difficult for researchers to collect large datasets for developing and testing machine learning models in healthcare. To address this issue, this study proposes a framework for generating and validating synthetic clinical discharge summaries using transformer-based biomedical language models. Initially, the clinical text is preprocessed using cleaning, formatting, and tokenization techniques to improve consistency and readability. Biomedical language models such as BioBERT, RoBERTa, and DistilBERT are then used to generate contextual embeddings and capture semantic relationships within medical text. In addition, semantic similarity analysis, entailment-based validation, and faithfulness evaluation are applied to verify the consistency and reliability of the generated summaries while preserving patient privacy and maintaining clinical relevance.
Mohammad Imran, M. Irfan, Sajida Sultana.Sk et al.· 2026 7th International Confe...· 0 citations
Background OpenNotes allows patients to access their electronic health record (EHR) notes through online patient portals. However, EHR notes contain abundant medical jargon, which can be difficult for patients to comprehend. One way to improve comprehension is by reducing information overload and helping patients focus on the medical terms that matter most to them. Objective This study aimed to evaluate both closed-source and open-source large language models (LLMs) for extracting and prioritizing medical jargon from EHR notes relevant to individual patients, leveraging prompting techniques, fine-tuning, and data augmentation. Methods We evaluated the performance of closed-source and open-source LLMs on a dataset of 90 expert-annotated EHR notes. We tested various combinations of settings, including (1) general and structured prompts, (2) zero-shot and few-shot prompting, (3) fine-tuning, and (4) data augmentation. To enhance the extraction and prioritization capabilities of open-source models in low-resource settings, we applied data augmentation using GPT-4o and integrated a ranking technique to refine the training process. Additionally, to measure the impact of dataset size, we fine-tuned the models by incrementally increasing the size of the augmented dataset from 10 to 9995 and tested their performance. The effectiveness of the models was assessed using 10-fold cross-validation, providing a comprehensive evaluation across various settings. We report the F1-score and mean reciprocal rank for performance evaluation using two different string matching algorithms (relaxed string matching and Jaccard Index). We also conducted an error analysis classifying the erroneous outputs from the models. Results Our results show that open-source models achieved the highest performance, particularly when using fine-tuning with a gold-standard dataset. Under Jaccard Index–based string matching, DeepSeek 8B set the benchmarks with an F1-score of 0.431 (SD 0.046); similarly, BioMistral 7B showed a mean reciprocal rank of 0.577 (SD 0.109). However, under relaxed string matching, open-source models were unable to match the performance of closed-source models, even with data augmentation or fine-tuning. We analyzed our experiment from several perspectives. First, few-shot prompting did not show an advantage over zero-shot prompting in vanilla models. Second, when comparing general and structured prompts, we found that model performance could deviate largely based on prompting styles. Third, fine-tuning with a small gold-standard dataset improved performance. Finally, data augmentation yielded performance comparable to or even surpassing the fine-tuning strategy. However, it also underscored the importance of the quality of the augmented dataset. Conclusions The evaluation of both closed-source and open-source LLMs highlighted the effectiveness of prompting strategies, fine-tuning, and data augmentation in enhancing model performance in low-resource scenarios.
W. Jang, Sharmin Sultana, Zonghai Yao et al.· JMIR AI· 1 citation
Pathology reports anchor cancer diagnosis and staging, yet their narrative structure limits reliable translation into structured, machine-actionable knowledge, creating a bottleneck between expert interpretation and scalable clinical intelligence. Despite decades of clinical natural language processing (NLP) research, pathology text remains among the most complex and consequential sources of medical data to operationalize at scale. Large language models (LLMs) offer new approaches for reading, extracting, and interpreting these reports. We synthesize current LLM work in cancer pathology using a four-level capability framework across the pathology report data lifecycle: (level 1) text preparation and quality checks, (level 2) information extraction, (level 3) guideline-based clinical reasoning, such as TNM staging and registry coding, and (level 4) interpretive synthesis, such as explanations, summarization, or decision support. Rather than grouping studies by NLP task labels, this framework tracks how LLM applications progress from preprocessing and extraction toward higher-level interpretation and synthesis. We followed PRISMA-ScR guidelines and searched four databases through September 2, 2025, identifying 41 eligible studies. Most studies focus on level 2 tasks, with fewer addressing level 3 and level 4 tasks. Encoder-based models, including domain-specific variants such as BioBERT, were commonly used for structured extraction tasks, whereas generative models, including GPT, LLaMA, and Mistral-family models, were increasingly evaluated for prompting-based extraction, staging, and summarization. Reported performance was often high for well-defined extraction tasks, but external validation was uncommon, and metrics varied across studies, limiting direct comparison. Overall, the evidence suggests that success in lower capability levels does not consistently translate to higher-level reasoning, especially when reports are inconsistent, required staging inputs are missing, or clinical assumptions must be inferred, which helps explain gaps between benchmark results and practical adoption. Future work should prioritize robust multi-site validation, clinically meaningful error analysis, transparent evaluation, and privacy-preserving implementation strategies to support safe integration in oncology.
Maryam Seifaddini, Mohammad Beheshti, Steven Richberg et al.· Modern Pathology· 0 citations
