A novel alignment framework, Reinforcement Learning for Selection Reward (RLSR), is proposed, which targets the area under the risk-coverage curve (AURC) -- a popular SP performance metric -- as its alignment objective and achieves substantially better risk-coverage trade-off compared to multiple alignment baselines on both in-domain and out-of-domain tasks.
Abstract
Large language models (LLMs) are increasingly deployed as critical decision-making components in high-stakes real-world AI systems, rendering LLM reliability a foremost practical concern. In this paper, we focus on enhancing LLM reliability through selective prediction (SP), a strategy that allows an LLM to only predict for inputs where it is likely to be correct (i.e., coverage) and hence reduce the error rate (i.e., risk) on that portion of inputs -- flagging the remaining inputs for future human discretion. In other words, SP improves LLM reliability by balancing the risk-coverage trade-off and enabling seamless human-AI collaboration. To integrate SP into LLMs, we focus on the LLM post-training alignment stage and propose to align LLMs with SP performance metrics, in contrast with existing LLM alignment methods that focus primarily on correctness or calibration metrics. Specifically, we propose a novel alignment framework, Reinforcement Learning for Selection Reward (RLSR), which targets the area under the risk-coverage curve (AURC) -- a popular SP performance metric -- as its alignment objective. RLSR achieves substantially better risk-coverage trade-off compared to multiple alignment baselines on both in-domain and out-of-domain tasks.
This work finds that likelihood-trained TPMs can result in failed generations due to overly large corrections to the LM’s logits, and trains TPMs with LM-aligned objectives that better align with the LM token-probability space.
Hanzhang Liu, William Zhao, Zilei Shao et al.· 0 citations
It is found that, in this benchmark, supervised fine-tuning (SFT) provides a strong baseline, substantially improving argument language consistency and end-to-end function call accuracy and, under consistent model selection, SFT achieves performance comparable to, and sometimes exceeding more complex reinforcement learning (RL) approaches.
Siddharth Chauhan, Thomas Butler, Abhishek Singhania et al.· 0 citations
ConfidenceBench, a calibration benchmark that evaluates verbalized confidence estimates in 15 frontier LLMs using the Brier score, a proper scoring rule that incentivises truthful probability reporting shows that verbalized confidence calibration is a distinct and practically important axis of LLM reliability, complementary to standard accuracy-based evaluation.
M. ffrench-Constant, Daniel Yang, Xinmeng Huang et al.· 1 citation· ⚡1
P-Bench is built, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine and introduces Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning.
Jia-Cheng Miao, Jin Mu, Guanhua Chen et al.· 0 citations
Evaluation-Conditioned Training (ECT), a post-training framework that uses natural language to condition each training sample on the fidelity of the feedback the authors provide and then elicits the desired behavior by conditioning the LLM on a high-fidelity monitor in deployment, is introduced.
Alec Harris, Kasey Corra, Archie Chaudhury et al.· 0 citations
Evolutionary rule-based machine learning (ERBML) algorithms can capture complex relationships while still yielding highly interpretable models comprised of IF:THEN rules. During prediction, 'matching' rules contribute to, and form the explanation for, the model's prediction. However, rules and their associated parameters (in their raw form) are likely too technical for their intended users. This study examines the feasibility of using a large language model (LLM) to translate the prediction evidence from matching rules into natural language text for different audiences, e.g. layman, clinician, expert. Using models trained by the 'HEROS' ERBML on MUX benchmarks, we evaluate LLM text quality metrics under different scenarios (i.e. 1,800 prediction explanations). We observe that (1), intuitively, LLM quality performance improves on HEROS models that have been more ideally trained, (2) making a glossary available to the LLM to define feature names generally raises explanation audience-fit scores and sometimes lowers overstatement (beyond rule-evidence), but it also lengthens explanations and often increases hallucination rate, and (3) audience customization creates some LLM performance trade-offs. These results suggest that constrained LLM translation of rules for natural language prediction explanations is feasible, while highlighting the importance of carefully designing the LLM prompts and evidence input from the ERBML.
H. Bandhey, Gabriel Lipschutz-Villa, Khoi Dinh et al.· Annual Conference on Genetic...· 0 citations