Preliminary evaluation of a large language model for pediatric uroflowmetry interpretation: a single-center comparative analysis with local expert consensus

Abstract

This study aimed to evaluate the performance of a large language model (LLM)-based artificial intelligence system in interpreting pediatric uroflowmetry (UFM) results using both conceptual knowledge and real-world clinical data. This retrospectively designed study involved a two-stage evaluation. In the first stage, 15 conceptual questions related to UFM were presented to the model. In the second stage, anonymized UFM recordings from 225 children aged 5–18 years were evaluated in five sequential analysis steps. Responses were reviewed by a pediatric urologist and a pediatric nephrologist on a 0–2 point scale for accuracy, clinical reliability, and comprehensibility. All responses to conceptual questions were clinically accurate; however, only 64.4% of cited sources were considered reliable, with a substantial proportion identified as hallucinated citations. A total of 1125 responses from real clinical cases were analyzed. The cohort comprised a heterogeneous case mix including normal ( n = 96, 42.7%), abnormal ( n = 93, 41.3%), and suboptimal quality ( n = 36, 16.0%) recordings. Model responses demonstrated complete agreement with local, non-blinded expert consensus across all cases and all subgroups, with clinical accuracy, reliability, and comprehensibility rates of 100%. Visual reading differences due to handwritten PVR values were observed in six cases but did not affect the diagnostic approach or recommended management. The model’s mean self-reported confidence level was 88.4%. In this single-center, preliminary evaluation, the large language model’s responses showed complete raw agreement with local expert opinion in pediatric UFM interpretation; these findings should be interpreted as hypothesis-generating rather than confirmatory. The findings support the potential use of artificial intelligence as a clinical decision support tool under physician supervision, rather than as an independent decision-maker.