2026· Annual Meeting of the Association for Computational Linguistics· pp. 33883-33891· 2 citations· 22 references
Computer Science
TL;DR
A semantic-entropy-based method, using task uncertainty to guide prompt optimization, which requires no training, works with black-box models, and integrates easily into existing prompt optimizers.
Abstract
Prompt optimizers are widely used to create high-quality prompts for Large Language Models (LLMs), but their effectiveness remains unstable in practice. This instability is caused by the misalignment between conservative needs (e.g., safety compliance) and open-ended goals (e.g., creative writing). To address this, we propose a semantic-entropy-based method, using task uncertainty to guide prompt optimization. Specifically, we measure the task’s uncertainty level with pre-defined templates, then use this measure to direct prompt optimization: selecting high-entropy prompt candidates for creative tasks and low-entropy candidates for conservative ones. Extensive experiments across various model families demonstrate that our method consistently outperforms baselines by effectively adjusting entropy levels. Our approach requires no training, works with black-box models, and integrates easily into existing prompt optimizers. Code is available at https://github.com/ syzonline/Prompt-Evolving .
This work introduces a Task-Aware Prompt Rewriter (TAPR), a model that reformulates user prompts into task-optimized prompts with the explicit goal of improving downstream LLM performance.
Oliver Savolainen, E. Bastianelli, H. Azarbonyad· 1 citation
Large Language Models (LLMs) demonstrate strong capabilities in automated essay scoring (AES), but contemporary approaches typically employ fixed prompt selection, failing to address operational cost concerns and evolving optimal configurations. We propose a novel cost-aware approach that treats each prompt type as an arm in a multi-armed bandit (MAB) controller, enabling adaptive selection of optimal prompting strategies during inference. Our experiments on IELTS Writing Task 2 essays show that the MAB framework achieves comparable scoring accuracy to exhaustive grid search while reducing LLM calls by 78.4\% to find the best grading approach. We implemented four distinct grading recipes (multi-step vs. single-step assessment, with vs. without calibration examples) and found that the multi-step approach with examples achieves the highest accuracy. By tracking token usage and latency alongside agreement metrics, we produce the first cost-reliability learning curves for essay scoring, providing actionable insights for educational technology platforms that must balance operational costs against assessment validity. This work represents the first application of online control mechanisms to adaptively select prompting strategies in AES, transforming prompt selection from an offline hyperparameter optimization problem into an efficient online learning task.
Large language models are increasingly used for information seeking, yet semantically equivalent questions phrased in different ways can receive answers of considerably different quality. System prompts are widely employed to steer response behavior, but they are typically optimized for average-case quality, so some question phrasings may still receive incomplete or low-quality answers. To address this, we formulate a constrained mixed-strategy GroupDRO framework for system-prompt selection. Instead of optimizing the system-prompt text, the framework assigns weights to system prompts in an existing pool to minimize the worst-case information-quality loss across evaluation metrics and groups, while constraining the mean loss to stay close to that of average-based selection. Because pool generation and selection are decoupled, the method applies to any system-prompt pool and can leverage an ensemble of complementary system prompts rather than a single one. Across five LLMs on two bilingual medical and consumer-finance benchmarks, the constrained method reduces the Overall Mean, Worst 25% Mean, and Worst by 13.1%, 13.2%, and 13.7% on average relative to no mitigation while keeping overall quality close to Average selection. Its multi-prompt weights reveal complementarity across metric-group pairs. Code and data are available at https://github.com/Rainxu09/equitable-system-prompt-selection.
Mengyu Xu, Qiaoxin Yang, Zhihan Liu et al.· 0 citations
SyRuP is introduced, a decoding-time framework for improving system-prompt adherence while keeping the base LM frozen, and results suggest that explicit token-level guidance is an effective and practical mechanism for reliable system-prompt following.
Seoyeon Kim, Minjae Kang, Jaehyung Kim· 0 citations
This work proposes answer instability, defined as the variability of a model’s final answer across repeated stochastic generations of the same prompt, as a simple, label-free, and black-box uncertainty signal, and demonstrates its utility for selective prediction and targeted repair, improving reliability without access to internal probabilities or additional training.
J. M. Araña, Kristine Ann M. Carandang, Ethan Robert A. Casin et al.· Annual Meeting of the Associ...· 0 citations
It is suggested that using standard CoT prompting increasingly acts as a source of distraction as models grow stronger, because standard CoT prompting also demands style adaptation, formatting compliance, and potentially undesired contextualization, which can distract models from the core reasoning task.
Denys Pushkin, Albert Q. Jiang, Aryo Lotfi et al.· 0 citations
