Generative AI versus Neurosurgeon in the Management of Head Injury Patients

Abstract

Abstract Introduction Generative AI models are large language models capable of interacting with humans in a man–machine interface. Endowed with extensive training on an expansive corpus of textual data, these GPTs excel across diverse domains, notably in the realms of medicine and healthcare. Generative AIs like ChatGPT, if incorporated into routine practice, can prove beneficial in patient diagnosis, management, and timely referral. Objectives This study aims to evaluate and compare the currently available GPT models with the expertise of neurosurgeons. Material and methods We explored the most widely used LLMs that are free to use to test and compare them with neurosurgeon's expertise viz, ChatGPT with GPT 4o, Claude Sonnet 4, Gemini 2.5 Flash, DeepSeek v3, Perplexity (AI search engine based on multiple LLMs), Microsoft Copilot, and Grok 3. Data were collected from 32 patients with head injury coming to the OPD or emergency department of a tertiary care center in India. Statistical analysis Neurosurgeon's opinion on whether to operate or provide conservative management versus the opinions of the different GPT models were compared using McNemar's exact test, and raw agreement was calculated. Results Out of 32 patients, the neurosurgeon advised conservative management in 29 patients (90.6%) versus surgery in three patients (9.4%). Raw agreement between the neurosurgeon and each AI model ranged from 50.0 to 81.2%. DeepSeek V3 (conservative 71.9% vs. surgery 28.1%; 81.2%) and Perplexity (conservative 71.9% vs. surgery 28.1%; 81.2%) achieved the highest agreement with the neurosurgeon's preferred method of treatment, followed by ChatGPT-4o (conservative 59.4% vs. surgery 40.6%; 68.8%) and Gemini 2.5 (conservative 59.4% vs. surgery 40.6%; 68.8%), Claude Sonnet 4 (conservative 16 patients vs. surgery 16 patients; 59.4%) Copilot (conservative 16 patients vs. surgery 16 patients; p -value 59.4%), and Grok 3, which had raw agreement of only 50%. On McNemar's exact test, five of seven models – ChatGPT-4o, Gemini 2.5 Flash, Claude Sonnet 4, Copilot, and Grok 3–showed statistically significant asymmetric disagreement with the neurosurgeon after Bonferroni correction; DeepSeek V3 and Perplexity did not reach significance ( p  = 0.031 for each, above the adjusted threshold of 0.0071). Conclusion In this small, single-center retrospective comparative study – freely available generative AI tools showed a tendency to recommend surgery more often than neurosurgeons. The disagreement occurred in a single direction only across all models. We are still at the initial stages of AI development. However, as the technology advances, these AI tools can prove to be beneficial adjuncts for the care of patients, and neurosurgeons should learn and incorporate these tools in their practice.