Skip to content
Open access

Cross-sectional comparative evaluation of five large language model–driven chatbots on an expert-curated 44-question set of parent-facing questions about pediatric vitamin D deficiency

Sep 2026 · Frontiers in Pediatrics · Vol 14 · 0 citations · 45 references
Medicine

TL;DR

To compare the safety, medical accuracy, empathy, empathy, information reliability, educational quality, transparency, global quality, and readability of five large language model-driven chatbots when answering questions about pediatric vitamin D deficiency, a single-run evaluation is conducted.

Abstract

Background Parents and caregivers increasingly use generative artificial intelligence chatbots for child health information. Pediatric vitamin D deficiency is a clinically relevant topic because advice about supplementation, testing, rickets, high-risk children, toxicity, and emergency symptoms can influence caregiver decisions. Objective To compare the safety, medical accuracy, empathy, information reliability, educational quality, transparency, global quality, and readability of five large language model-driven chatbots when answering questions about pediatric vitamin D deficiency. Methods This cross-sectional comparative study was reported with reference to CHART. An expert-curated 44-question set was selected from a 92-question candidate pool informed by search trends, caregiver-facing sources, clinical guidelines, and expert discussion. Each of the 44 questions was submitted once to each of five chatbot services—ChatGPT-5.5, Gemini 3.1 Pro, Qianwen 3.6-Plus, DeepSeek V4, and Doubao-Seed-2.0 Pro—using the same parent-oriented instruction. Responses were assessed for safety, accuracy, empathy, DISCERN, EQIP, JAMA criteria, GQS, and readability. Paired question-level differences were analyzed using Friedman tests, Kendall's W, and Cochran's Q; results were interpreted as a single-run, time-specific snapshot. Results Inter-rater agreement was good to excellent (Fleiss' kappa = 0.842 for safety; ICCs 0.846–0.914 for other subjective metrics). Twenty of 220 responses (9.1%) were unsafe; safety did not differ significantly across models (Cochran's Q = 3.704, df = 4, P = 0.448). Significant inter-model differences were observed in accuracy, empathy, DISCERN, EQIP, JAMA, GQS, and all readability indices. ChatGPT had the highest median accuracy [5.00 (4.00, 5.00)] and DISCERN score [69.60 (65.25, 72.40)]; DeepSeek and Doubao had the highest empathy scores [5.00 (4.80, 5.00)]. ChatGPT and Doubao shared the highest EQIP median (86.00), Doubao had the highest GQS [5.00 (4.00, 5.00)], and Gemini had the highest FRES. JAMA scores were low across models. Conclusions In this single-run evaluation, the five chatbot services showed domain-specific performance differences without a significant safety difference. Findings are exploratory rather than evidence of stable model superiority and support multidimensional evaluation with clinician involvement for high-risk or individualized pediatric advice.

Read PDF

Similar papers

#large language models Review Open access Sep 2026

A cross-sectional evaluation of large language model chatbot interfaces for patient-facing herpes zoster information: safety, information quality, and readability

In this standardised benchmark, LLM chatbot interfaces provided generally favourable accuracy and information-quality scores but showed clinically relevant safety limitations, sparse source attribution and disclosure, and readability challenges.

Da-Lian Liang, Cong Mai, Zheng-Kun Zhang et al. · 0 citations
Open access Oct 2026

Generative AI chatbots as sources of phimosis-related consumer health information: a cross-sectional comparison of quality, source transparency, and readability

Background Publicly accessible generative artificial intelligence chatbot products are increasingly used for sensitive health questions, but the quality, source transparency, and readability of phimosis-related information remain uncertain. Methods We conducted an internet-based cross-sectional comparative study on 1 J...

Qing-Hua Luo, Lei-Hua Cao, Pan Ding · 0 citations
#large language models Open access Oct 2026

Development and Nationwide Multicentre Evaluation of Guideline-Grounded Large Language Model Chatbots to Support Patient Self-Management and Education in Rheumatology

Patients with rheumatic diseases have persistent information needs that are not fully addressed in routine care. We developed and evaluated guideline-grounded, large language model (LLM) chatbots to support patient self-management and education in rheumatology. Ten disease-specific chatbots based on German guidelines w...

Tim Wilhelmi, Vanessa Bartsch, Marius Platt et al. · 0 citations
Review Open access Oct 2026

Efficacy of artificial intelligence-driven conversational agents (chatbots) for mental health promotion among university students: a systematic review and meta-analysis

To evaluate the effects of AI-mediated conversational interventions on depressive and anxiety symptoms in university students, while distinguishing outcome domains, comparator intensity, intervention architecture, and the evidential requirements for standardized post-intervention effect estimation, the search date was...

Chen Shao, Xiao-Chen Chen, Sheng-Nan Qiao et al. · 0 citations
Oct 2026

Evaluation of the reliability, quality, and readability of multiple large language model responses (ChatGPT, Gemini, Copilot) on silicosis.

BackgroundInadequate or misleading health information about silicosis can delay diagnosis, reporting, and exposure removal, potentially affecting risk communication and timely occupational health interventions.ObjectiveThis study evaluates the reliability, quality, and readability of Large Language Models (LLMs) includ...

Nejdiye Güngördü, Seher Kurtul, Büşra Albayrak · 0 citations
Open access Oct 2026

Benchmarking Generative AI Chatbots in Preventive Medicine: A Comparative Analysis of Accuracy, Completeness, and Inter-Rater Reliability

Background/Objectives: Artificial intelligence (AI) chatbots are increasingly used for healthcare information retrieval, education, and clinical support; however, concerns remain regarding the accuracy and completeness of AI-generated information in preventive medicine, where evidence-based recommendations, epidemiolog...

N. Z. Alshahrani, A. Alsabaani, Ali Hassan Almaqsudi et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.