Performance of large language models in prescription review: a comparative study

Abstract

Objective Large language models (LLMs) have emerged as a novel technical solution for intelligent prescription review. Nevertheless, comparative analyses of their performance remain limited. This study comprehensively evaluates five mainstream LLMs (ChatGPT, DeepSeek, Kimi, Qwen and Doubao) in prescription review, aiming to furnish data-based references for model selection, optimization and clinical deployment of intelligent prescription review systems. Methods This study employed a cross-sectional comparative design. The evaluation dataset was derived from the prescription review question bank of the Chinese Pharmacist Skills Competition. In total, 100 prescriptions from 10 clinical specialties were randomly sampled, encompassing 270 standardized medication assessment items. Official standard answers served as the gold standard. Quantitative comparisons were performed on overall detection rate, question type identification and adaptability across clinical specialties. Prescription-level repeated-measures analysis of variance (ANOVA) and Holm-adjusted pairwise paired t-tests were used to examine inter-model statistical differences. Results The five models yielded item-level prescription detection rate ranging from 45.19% to 70.37% across 270 standardized medication assessment items. Their performance ranked from highest to lowest as follows: Kimi, DeepSeek, Qwen, Doubao and ChatGPT. Overall, the four Chinese domestic models included in this study outperformed the single tested international model (ChatGPT) in this China-focused prescription review task. Repeated-measures ANOVA confirmed statistically significant within-prescription differences in overall performance (F = 22.46, P < 0.001). While all models performed well at identifying inappropriate dosage forms and administration routes, they were less effective in detecting issues involving absent allergy test documentation and flawed result interpretation. Prescription reviews for the respiratory department yielded the best results, whereas detection rate was substantially lower for specialized clinical specialties such as oncology, pediatrics and pain management. Even the best-performing model presented a high false-negative rate and cannot yet satisfy the strict reliability criteria for clinical application in prescription review. Conclusion Performance varied markedly across the five LLMs in prescription review. Although these models can effectively identify routine and straightforward medication issues, they demonstrate prominent deficiencies in assessing complex prescriptions from specialist clinical specialties and detecting hidden medication risks. Currently, LLMs are only suitable for preliminary auxiliary screening and cannot substitute dedicated prescription review software or manual pharmacist evaluation.