A Policy-guided LLM Pipeline for Safer Clinical Decision Support: Error Analysis on High-risk Medication Queries

Abstract

: Large language models can generate fluent and often convincing answers in medical contexts, but in high-risk settings, fluency alone is not enough. This paper presents a policy-guided LLM pipeline for medication-related clinical decision support, designed to classify prompts into safety-oriented decision categories: ACCEPT, WARN, DEFER, ESCALATE, or REFUSE. The system combines rule-based risk detection, guardrail routing, and a decision policy layer to handle prompts involving dosage, pregnancy, drug interactions, self-adjustment, and other safety-critical contexts. The pipeline was evaluated on a benchmark of 55 medication-related questions using two baseline models. Both models produced the same overall system accuracy of 60.0%, with a false accept rate of 16.4%, suggesting that the main limitations are not model-specific but structural. Three recurring failure modes emerged: false acceptance of implicitly risky prompts, over-escalation of educational or professional-context questions, and under-escalation of self-adjustment or dangerous-intent cases. These findings make the system useful not only as a prototype, but also as a transparent framework for studying where safety-oriented LLM pipelines succeed and where they still fail.