Risk-Calibrated Prompting: Selective Answering for Robustness Under Prompt Perturbations

Abstract

Large language models (LLMs) can produce fluent responses that remain unreliable when a question is rephrased, surrounded by irrelevant context, or intrinsically ambiguous. Most prompt-based hallucination mitigation methods optimize answer quality or average factual error, whereas deployment also requires a decision about whether an answer should be released at all. This paper presents Risk-Calibrated Prompting (RCP), an inference-time selective answering framework that combines semantic consistency across controlled prompt perturbations with model-verbalized confidence. RCP constructs a canonical prompt, a meaning-preserving paraphrase, and a controlled stress variant; obtains independent responses from the same model; estimates semantic agreement; combines agreement with verbalized confidence through a calibrated risk score; and releases the canonical answer only when the score crosses a validation-selected threshold. The framework requires neither parameter fine-tuning nor an external retrieval index. We define a paired evaluation protocol over TruthfulQA and the question-answering subset of HaluEval using open-weight instruction models, fixed perturbation templates, risk–coverage analysis, calibration metrics, robustness measures, cost accounting, and human audit. The protocol explicitly separates uncertainty signaling from factual truth: high agreement is treated as evidence of behavioral stability rather than proof of correctness. This formulation provides a reproducible basis for studying when an LLM should answer, when it should abstain, and how selective reliability changes under prompt perturbations.