Evaluating the readability and quality of ai-generated stress urinary incontinence education materials: a comparative analysis of five large language models

Abstract

Stress urinary incontinence (SUI) substantially impairs quality of life, yet accessible and comprehensible patient education remains limited. Large language models (LLMs) may support health-information delivery, but their outputs require systematic evaluation. To compare the readability, reliability, and patient-oriented usability of SUI education materials generated by five LLMs. ChatGPT Free, Claude Free, Microsoft Copilot Free, DeepSeek Free, and Gemini Free were tested through their official web interfaces on March 15, 2026. Each model answered 12 core SUI questions, yielding 60 responses. Six readability indices were assessed: Flesch-Kincaid Reading Ease (FKRE), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Score (GFS), Simple Measure of Gobbledygook (SMOG), Coleman-Liau Index (CLI), and Automated Readability Index (ARI). Information quality and patient usability were evaluated using the Global Quality Scale (GQS), DISCERN, Ensuring Quality Information for Patients (EQIP), and Patient Education Materials Assessment Tool (PEMAT). Overall differences were examined using Friedman tests with Kendall’s W, followed by paired Wilcoxon signed-rank tests with Holm correction. Significant inter-model differences were found for FKRE ( P  = 0.005), FKGL ( P  = 0.016), GFS ( P  = 0.002), SMOG ( P  = 0.007), and CLI ( P  < 0.001), but not ARI ( P  = 0.107). Gemini had higher FKRE than ChatGPT (30.06 ± 12.63 vs. 23.60 ± 11.72; adjusted P  = 0.036), Claude had higher GFS than Copilot (18.41 ± 4.56 vs. 15.43 ± 1.59; adjusted P  = 0.018), and DeepSeek had higher SMOG than ChatGPT and Copilot (adjusted P  = 0.013 and 0.049). GQS, DISCERN, EQIP, and PEMAT all differed significantly among models (all P  < 0.001; Kendall’s W = 0.212–0.373). Gemini scored higher than Copilot on GQS; Claude scored higher than all other models on DISCERN; Gemini scored lower than all other models on EQIP; and ChatGPT scored lower than all other models on PEMAT (all adjusted P  < 0.05). Inter-rater reliability was excellent (ICC [2, 1] = 0.954). EQIP correlated positively with PEMAT ( r  = 0.69), whereas DISCERN showed limited associations with all readability measures. The five LLMs differed significantly on all quality instruments, but no model performed best on every dimension. AI-generated SUI education materials therefore require independent factual verification, targeted simplification and final review by qualified healthcare professionals before any patient-facing use.