Large language models versus expert clinicians in optimizing Chinese patient education materials for temporomandibular disorders: a mixed-methods study integrating readability, accuracy, actionability, and cultural adaptability
TL;DR
The human-AI collaboration model represents a promising and balanced approach for producing Chinese TMD patient education materials, combining LLM strengths in structural optimization with expert clinician oversight for accuracy and cultural appropriateness.
Abstract
Objective To evaluate the performance of large language models (LLMs) and expert clinicians in optimizing Chinese patient education materials (PEMs) for temporomandibular disorders (TMD) across readability, accuracy, actionability, and cultural adaptability, and to determine whether a human-AI collaboration model can achieve an optimal balance among these dimensions. Methods Fifteen TMD education topics were selected through a Delphi consensus process. Four groups of PEMs were generated: original texts (Group A), LLM-rewritten texts (Group B, Claude Opus 4.6), clinician-rewritten texts (Group C), and human-AI collaboration texts (Group D). Blinded assessments were conducted by a professional panel (n = 3) and a health literacy-stratified patient panel (n = 6). Readability was measured by sentence length and common character proportion, accuracy by 5-point expert ratings against standardized checklists, actionability by the PEMAT instrument, and cultural adaptability by qualitative thematic analysis. Prompt sensitivity was tested across three distinct styles. Results LLM-rewritten texts demonstrated significantly shorter sentences (21.1 vs. 33.9 characters, p < 0.001) and substantially higher actionability (95.6% vs. 16.7%, p < 0.001) compared with original texts. Clinician-rewritten texts achieved the highest accuracy (4.48 vs. 3.23, p < 0.001) but low actionability (46.7%). The human-AI collaboration model matched clinician accuracy (4.49, p = 1.000) while preserving LLM actionability (95.6%), with 72% less clinician time. Sensitivity analysis confirmed actionability robustness under empathetic and authoritative prompts but attenuation under a concise prompt. Qualitative analysis identified four cultural adaptability themes, with LLMs excelling in terminology localization and behavioral structuring but showing limitations in TCM conceptual depth. Conclusion The human-AI collaboration model represents a promising and balanced approach for producing Chinese TMD patient education materials, combining LLM strengths in structural optimization with expert clinician oversight for accuracy and cultural appropriateness.