Jul 2026· BMJ Health & Care Informatics· Vol 33, pp. e101894· 0 citations· 46 references
Medicine
TL;DR
It is indicated that while fine-tuned LLMs offer superior accuracy, the observed performance variability across heterogeneous electronic health record sources poses significant technical challenges.
Abstract
Objectives The rapid evolution of large language models (LLMs) and their growing application in clinical text processing have created an urgent need for reliable de-identification mechanisms. While LLMs show promise in identifying sensitive health information (SHI), their capabilities require rigorous evaluation. This study aims to conduct a comprehensive benchmarking analysis of various LLM-based, traditional rule-based and hybrid de-identification methods. Methods Our benchmark analysis used five datasets (i2b2-2006, MIMIC-2008, i2b2-2014, i2b2-2016 and OpenDeID v1) from different countries. We developed three baseline and eight LLM-based models. The experimental setup encompassed nine different settings using various combinations of training and testing sets to assess model robustness and cross-dataset performance. Results In the baseline models, the approach trained on the combined corpus of all five datasets (setting 3) significantly outperformed the other settings, achieving a strict F1 micro-average score of 0.8172. Regarding LLM-based models, the supervised fine-tuning approach using the same combined configuration (setting 9) achieved the highest performance with a strict F1 score of 0.9447. Discussion The harmonisation of corpora ensured standardised data formatting and SHI management across five diverse datasets, highlighting the necessity for uniform categorisation to enhance the reliability of de-identification results. Conclusions Our findings indicate that while fine-tuned LLMs offer superior accuracy, the observed performance variability across heterogeneous electronic health record sources poses significant technical challenges. Real-world implementation must address these inconsistencies to overcome the ethical and technical hurdles associated with deploying LLMs for handling sensitive health data.
The findings support the feasibility of applying LLM-based natural language processing tools in resource-limited, non-English healthcare settings and should assess emerging high-parameter models and explore additional clinical domains.
Breno Gabriel Araújo Sampaio de Jesus, Tomaz Castrillon Figueiredo, Clariele de Almeida Pereira et al.· Cadernos de Saúde Pública· 1 citation
BACKGROUND
Unstructured clinical narratives in electronic health records contain essential information for healthcare delivery and research. However, the presence of personally identifiable information poses significant privacy risks, which limit the secondary use of data. Therefore, reliable automated de-identification is a prerequisite for the reuse of clinical texts. This study aims to evaluate different strategies for Spanish clinical text de-identification by assessing the performance of language models on synthetic and real-world datasets.
METHODS
This study presents a comparative evaluation of de-identification approaches using two datasets in Spanish: MEDDOCAN, a publicly available synthetic corpus, and ObstEHR, real-world clinical narratives from an obstetrics department. Three de-identification strategies were assessed: (i) inference with a pre-trained task-specific model used as-is, (ii) prompt-based inference using locally deployed large language models (LLMs), and (iii) fine-tuned task-specific models. Model performance was evaluated using precision, recall, and F1 score on test sets. In addition, a text preservation metric was introduced to assess prompt-based LLMs, and the impact of training set size was analyzed using progressively larger subsets of annotated ObstEHR data.
RESULTS
Models without fine-tuning and prompt-based LLMs showed limited performance, with macro-averaged F1 scores ranging from 0.14 to 0.5 on ObstEHR and from 0.19 to 0.5 on MEDDOCAN. Fine-tuned models achieved higher performance, reaching macro-averaged F1 scores of up to 0.956. Learning curve analyses showed consistently high precision and gradual improvements in recall as the amount of training data increased, with high performance achieved with moderate amounts of annotated data.
CONCLUSION
Fine-tuning-based approaches outperform models without task-specific adaptation and prompt-based strategies. Despite their flexibility in generative settings, LLM-based prompt strategies show limited reliability and information preservation in clinical de-identification. Prompt-based LLMs make slight textual modifications that cause token misalignment, leading to a subsequent decrease in evaluation metrics. Therefore, domain-specific fine-tuning remains the most effective strategy for real-world clinical text de-identification.
D. Vallejo-Sanchez, A. F. Giraldo-Forero, A. Orozco-Duque· Journal of Biomedical Semant...· 0 citations
Open LLMs can extract clinical findings from Finnish pediatric records with accuracy comparable to published English benchmarks, and uncertainty-based triage substantially reduces required expert workload.
J. Leinonen, J. Knuutila, S. Kurki et al.· medRxiv· 0 citations
VITA's advantages in accuracy and completeness persisted under the neutral judge; its communication scores were lower, and this results indicate that a purpose-built clinical RAG system remains competitive with frontier LLMs on an open benchmark, consistent with corpus specificity as a design variable that improves grounding at some cost to communication polish.
Praveen Reddy, C. Mandke, Suvrankar Datta et al.· 0 citations
Abstract Objectives To develop a pipeline for evaluating large language models (LLMs) on the task of capturing symptoms from clinical encounters. Materials and Methods We created a gold standard dataset of symptom annotations from simulated doctor-patient encounter excerpts (264 encounters; 16 symptoms; double-coded and adjudicated). Nine different LLMs from 4 vendors (OpenAI, Meta, DeepSeek, Moonshot AI) were used as examples to test our evaluation pipeline; outputs were assessed for correct structure and symptom information. Results Of 3085 excerpts, 2087 (68%) contained symptoms. Pain, cough, and shortness of breath were most common; LLMs achieved F1 scores ranging 0.66-0.88 for these symptoms with minimal prompt engineering. Of tested models, GPT-4.1 demonstrated the best overall performance. Discussion Our evaluation pipeline and benchmarking dataset are publicly available and applicable to various LLMs, including open-source models. Conclusion This work supports the development and optimization of models that seek to improve patient symptom understanding.
Joshua Davis, B. Durieux, Chloe Van Dongen et al.· JAMIA Open· 0 citations
For automated assessment metadata extraction, reliability rather than accuracy determines whether an LLM can be deployed, and national origin is not a meaningful predictor.
Hui Zhang, Lihui Qu, Hongbo Bai et al.· Frontiers in Artificial Inte...· 0 citations