Preprint

MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment

Tony Tu Sayan Chakraborty Ruomeng Xu Tony Qin Austin Tian

Aug 2026 · 0 citations · 15 references

Computer Science

TL;DR

Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.37 to 0.64, surpassing human experts and persisting across full multi-turn rollouts.

Abstract

Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help. The root issue is that an additive reward has no notion of balance. We introduce Mint (MIN-selection preference disTillation), a one-line change to preference distillation: rather than ranking sampled candidates by a weighted sum of rewards, we rank them by their weakest objective, distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective. This is the p ->negative infinity limit of a generalized-mean family spanning additive to worst-case selection. Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.37 to 0.64 (p<10^-40), surpassing human experts and persisting across full multi-turn rollouts. A turn-by-turn analysis yields our central finding: min-selection corrects imbalance in proportion to how imbalanced the reference policy is, and its benefit endures over an interaction precisely as long as that imbalance does.

arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

This work proposes \methodname, a stability-guided active-set controller for controlled objective admission, a stability-guided active-set controller for controlled objective admission in reward-vector RLHF, which positions objective-entry timing as a concrete control variable in reward-vector RLHF.

Yong-Qi Tong, Z. Zhang, Ruirui Wang et al. · 1 citation

#machine learning Preprint Oct 2026

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this...

Minjae Lee, Kyunghyun Cho, Sangdon Park · 0 citations

#artificial intelligence Preprint Oct 2026

Lexicographic Multi-Objective On-Policy Distillation

Reinforcement learning from verifiable rewards (RLVR) usually optimizes answer correctness, yet useful language-model behavior also requires high-quality reasoning and concise responses. Existing multi-reward post-training methods typically scalarize rewards or combine specialists without explicitly protecting a reward...

Doseok Jang, Jon Ander Campos, You-Ran Qi · 0 citations

#artificial intelligence Preprint Sep 2026

On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning

A natural extension to preference-conditioned multi-objective RL (MORL) relabels transitions with the preference direction the agent achieved rather than the one asked for, and a single-parameter convex combination pulling the achieved direction back towards the requested preference is introduced.

Baptiste Bonin, Caro Strickland, Audrey Durand · 0 citations

#machine learning Preprint Aug 2026

Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment

Evolutionary Soups, a mixture-of-experts framework for fine-grained generation control, with gating networks trained via an evolutionary algorithm, achieves the best hypervolume, linear utility, and Tchebyshev utility among controllable methods on all tasks.

Lingxiao Kong, Steffen Staab, Cong Yang et al. · 0 citations

#natural language process... Preprint Oct 2026

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that m...

Andreea Dutulescu, Stefan Ruseti, Mihai Masala et al. · 0 citations