Can large language models safely prioritize ENT referrals? A comparison with junior ENT doctors and consultant otolaryngologists

Abstract

Background: Prompt escalation of ear, nose, and throat (ENT) referrals is vital for identifying potentially life-threatening diseases and is important in diagnosing head and neck carcinomas to avoid any delays in diagnosis. Large language models (LLMs) have shown the potential to prioritize referrals in clinical set up. But their safety is still not well established.Methods: We classified one hundred standardized ENT referral scenarios into four categories independently. Urgency was indicated as emergency, urgent, semi-urgent or routine by three consultant otolaryngologists, four junior ENT doctors, and three general-purpose LLMs: ChatGPT, Claude, and Gemini. Out of the hundred, sixteen contained suspected head and neck cancer referrals. Reference standard was taken as majority decision of three consultants. Degree of agreement with reference standard, over-triage, under-triage, quadratic Cohen’s κ were analysed.Results: Junior ENT doctors had 50–62% agreement with consultant opinion. Over-triage was calculated as 28–44%. Under-triage occurred in 5–12%. Quadratic Cohen’s κ values ranged from 0.45 to 0.74. It indicated moderate to substantial agreement. All three LLMs recognized all sixteen head and neck cancer scenarios. Their prioritization was comparable with reference standard. Disagreements were observed in chronic or benign conditions. They were usually limited to adjacent urgency categories. Junior clinicians revealed a pattern of early escalation.Conclusions: Current LLMs demonstrated promising performance in prioritizing simulated ENT referrals specially head and neck carcinomas. Their consistency suggests potential value as an adjunct to clinical decision making. Prospective studies using hospital referrals and safety monitoring must be carried before application.