Nov 2026· Proceedings of the 35th ACM International Conference on Information and Knowledge Management· 0 citations· 9 references
TL;DR
Evaluating 140 manually curated underdetermined prompts across seven categories on six LLMs suggests that asking for the missing criterion is a practical and transparent strategy for LLM systems facing under-specified user requests.
Abstract
Large language models are increasingly used in question answering, recommendation, and conversational decision support. In many such settings, the user asks for a single answer even though the prompt does not specify the criterion needed to justify a unique choice. We study this setting as semantic underdetermination: multiple answers remain compatible with the prompt unless an additional ranking criterion is supplied. Rather than claiming a formal impossibility result, we study an empirical tension among three properties: premise-entailment correctness, criterion neutrality, and single-turn decisiveness. We evaluate 140 manually curated underdetermined prompts across seven categories on six LLMs, from 4B to frontier scale, under four response policies that handle under-specification differently. Forcing a direct answer makes models far more decisive (decisive rate 32%?93%) but lowers judged utility (4.06?3.53 out of 5, a large effect), because a confident answer must commit to a criterion the user never gave. Asking for the missing criterion instead gives the highest judged utility (4.53/5) and LLM-predicted satisfaction (4.48/5) but is rarely decisive in one turn (5.4%). A criterion-sensitivity score supports the mechanism: differences are small in size, but every one of the six models is most criterion-sensitive when forced to answer and least when it asks for the criterion. These findings suggest that asking for the missing criterion is a practical and transparent strategy for LLM systems facing under-specified user requests.
LLM-as-a-Judge has become a standard for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable evaluation instrument. Prior work has primarily explored deliberate prompt modifications such as scoring rubrics, role descriptions, or chain-of-thought reaso...
Ilija Subasic, Brett Levert, Andrew Rabinovich et al.· 0 citations
It is suggested that likelihood preferences over controlled declarative alternatives and task-conditioned answer selection probe distinct aspects of model behavior, and should not be treated as interchangeable.
Alessandro Bondielli, Lucia C. Passaro, D. Bacciu et al.· arXiv.org· 0 citations
When user queries are incomplete, large language models should ask targeted questions before committing to an answer, a capability we call proactive elicitation, yet evaluation on real user-assistant queries across seven LLMs (8B to frontier) shows that fewer than 11% of underspecified queries trigger a follow-up quest...
Rong-Yu Zhang, Mu-Tian Bao, Jin-Jian Zhang et al.· Proceedings of the 2026 Conf...· 0 citations
Across eight text, image, and video benchmarks and eighteen models, pooling is harder for every model, the elimination credit is largest for the weakest models, and seven of eight open-weight models answer 87 to 100% of unanswerable questions.
In extractive document question answering whose questions were generated from the passages that contain their answers -- so that retrieval recovers 92-99.8% of what any mode combination could reach, whatever its absolute accuracy -- confidence-driven mechanisms have little to gain. Fine-tuning an open language model on...
Risk-Calibrated Prompting is presented, an inference-time selective answering framework that combines semantic consistency across controlled prompt perturbations with model-verbalized confidence and explicitly separates uncertainty signaling from factual truth.
Neelu Verma· International Journal of Inn...· 0 citations