Comparative Analysis of Large Language Model Outputs for Pre-hypnotherapy Support in Irritable Bowel Syndrome - An Experimental Feasibility Study.
TL;DR
The findings primarily highlight the need for expanded evaluation with multiple expert raters, patient participants, and more robust quality assessment methods before considering any practical or clinical use of LLMs.
Abstract
Background/Aims : Gut directed hypnotherapy can assist with irritable bowel syndrome, but access remains limited. This feasibility study conducted an exploratory comparison of 4 large language models (LLMs) to evaluate the readability and information quality of model generated responses to common patient questions about pre- hypnotherapy support. Methods : Four LLMs (ChatGPT-4o, Claude 3.7 Sonnet, Gemini, and DeepSeek-R1) were queried using a pre-specified set of 14 standardized patient-oriented prompts. Responses were assessed for readability using validated indices (FKGL, FRES, and CLI) and for information quality using the DISCERN instrument and 5 item Likert scale, rated by a single evaluator. Paired statistical tests were used, and results are reported with effect sizes and confidence intervals. Results : In this exploratory evaluation, DeepSeek-R1 and ChatGPT-4o generated responses with relatively higher scores across readability metrices (Flesch-Kincaid Grade Level, Flesch Reading Ease Score, and Coleman-Liau Index) and information quality measures (DISCERN and Likert), which are distinct constructs assessed by separate instruments. Claude 3.7 Sonnet produced denser with lower readability and reliability scores, and Gemini showed intermediate performance. Interpretation is limited using automated readability metrics and single-rate design. As the DISCERN and Likert scores were generated by a single rater, statistical comparisons involving these measures cannot be considered definitive. The results below are presented solely to illustrate observable trends within this exploratory dataset. Conclusion LLMs show potential for generating patient oriented pre-hypnotherapy information, but this feasibility study is not sufficient to determine clinical applicability. The findings primarily highlight the need for expanded evaluation with multiple expert raters, patient participants, and more robust quality assessment methods before considering any practical or clinical use.