A Methodology to Evaluate the Efficacy and Patient-Centeredness of a Patient-Facing AI Chatbot: Development and Validation of a Standardized-Patient Method

Abstract

Background: Patient-facing AI intake chatbots can enter clinical use without regulatory review, so their errors may first surface with actual patients. Objective: To develop and pilot a remote, voice-based standardized patient (SP) method for evaluating an intake chatbot's history-taking, documentation, and patient-centeredness. Methods: Four primary care cases had Easy and Difficult variants. In Phase 1, five SPs completed 40 encounters with internal medicine residents rating portrayal fidelity. In Phase 2, SPs completed 80 encounters with a ChatGPT 4o chatbot, recorded whether critical elements were asked about and addressed satisfactorily, and rated summaries; four debriefing interviews were analyzed. In Phase 3, two blinded clinicians rated transcripts and summaries from 101 encounters (32 resident-led, 69 chatbot-led). Results: Portrayal fidelity averaged 4.52 of 5. The chatbot asked about 59.3% of critical elements in Easy versus 50.2% in Difficult variants (adjusted OR 1.49; 95% CI, 1.15 to 1.93) and addressed 73.3% versus 64.6% (OR 1.47; 1.12 to 1.93); near-ceiling summary quality did not differ. SPs reported missed information and rushed pacing. Reviewer agreement was moderate to substantial for element judgments (AC1 0.49 to 0.70) but poor for totals (ICC 0.10 and 0.21). In exploratory comparisons, residents asked about more elements (adjusted difference 12.5 percentage points; 95% CI, 7.0 to 18.1) and satisfactorily addressed more (9.2 points; 3.8 to 14.6); totals did not differ. Conclusions: SP portrayal was consistent enough for pre-clinical chatbot evaluation, and the method detected lower coverage in harder cases. Documentation ratings alone may miss conversational gaps, which require direct measurement.