Comparative Evaluation of ChatGPT-5.2, Claude Sonnet 4.5, and DeepSeek-V3.2 for Rosacea-Related Information: Accuracy, Reliability, Readability, and Reference Hallucinations
TL;DR
Generative AI chatbots demonstrate considerable potential as sources of health-related information for rosacea-related queries, however, variability in performance and reference hallucination rates highlights the need for careful validation before their widespread use as complementary sources of patient health information.
Abstract
Background/Objectives: Rosacea is a chronic inflammatory skin disease that requires long-term management and continuous patient education regarding triggers, skincare practices, and treatment adherence. In recent years, patients have increasingly turned to online platforms and artificial intelligence (AI)-based chatbots for health-related information. Although ChatGPT has been evaluated in the context of rosacea, evidence regarding the performance of other AI chatbots remains limited. This study aimed to evaluate the accuracy, reliability, quality, readability, and diagnostic relevance of AI-generated responses to common rosacea-related patient questions and to assess their potential role as sources of health-related information. Methods: Between 21 December 2025 and 22 February 2026, rosacea-related questions were collected from the publicly accessible Quora platform using a systematic screening process. Twenty clinically relevant and representative questions covering diagnosis, triggers, treatment options, skincare practices, and disease manifestations were selected. Each question was independently submitted to three AI chatbots (Claude Sonnet 4.5, ChatGPT-5.2, and DeepSeek-V3.2). Responses were evaluated by domain experts using the modified DISCERN (mDISCERN) for reliability, the Global Quality Scale (GQS) for overall quality, the Flesch Reading Ease Score (FRES) for readability, and a 5-point Likert scale for accuracy. Reference hallucinations were assessed through manual verification of cited sources. Statistical comparisons were performed using the Friedman test with Bonferroni-adjusted post hoc analyses, and effect sizes were calculated using Kendall’s coefficient of concordance (Kendall’s W). Results: Significant differences were observed among the AI chatbots across all evaluation domains (p < 0.05), with moderate to large effect sizes (Kendall’s W = 0.272–0.683). ChatGPT-5.2 and DeepSeek-V3.2 demonstrated significantly higher reliability and accuracy scores than Claude Sonnet 4.5. DeepSeek-V3.2 achieved the highest overall quality scores, whereas ChatGPT-5.2 produced the most readable responses. Reference analysis revealed variable hallucination rates among the evaluated models. Conclusions: Generative AI chatbots demonstrate considerable potential as sources of health-related information for rosacea-related queries. However, variability in performance and reference hallucination rates highlights the need for careful validation before their widespread use as complementary sources of patient health information.