Medical Knowledge Is Not All You Need: When Medical Q&A Becomes Situated Patient Assistance
TL;DR
It is shown that when Q&A is embedded within ongoing care, reliability depends on more than what the system knows medically, and proposes a design space for situated medical assistance that connects what the assistant and patient can each reliably establish to the form of assistance provided.
Abstract
Reliability in medical Q&A is often pursued by grounding responses in authoritative medical information. We show that when Q&A is embedded within ongoing care, reliability depends on more than what the system knows medically. In a study with 73 skin cancer patients practicing postoperative wound care, 41.9% of response-requiring questions depended on information beyond the procedure, including visual or physical state, environmental context, or prior actions. These demands varied across patients, consistent with patients recruiting the assistant into different informational roles. We then replayed the questions to seven LLMs while adding procedural and postoperative guidance. Errors remained substantial, including treating unknown states as known, even under explicit guardrails; with full procedural context, six of seven models more often introduced later steps prematurely. Based on these findings, we propose a design space for situated medical assistance that connects what the assistant and patient can each reliably establish to the form of assistance provided.
