This evaluation combines Preference Reward Responsiveness (PRR) with coverage and utility metrics to compare reward-level, advantage-level, loss-level, and PCGrad-based aggregation across two humanoid-control frameworks.
Abstract
Modern robotic agents are increasingly expected to support steerable control, where a single policy can adjust its behavioral style at runtime under changing user preferences. Preference-conditioned reinforcement learning provides this interface, but conditioning alone does not ensure that a requested behavior change is strong, does not leak into unrelated behaviors, and does not compromise task performance. Our evaluation combines Preference Reward Responsiveness (PRR) with coverage and utility metrics. We use these measures to compare reward-level, advantage-level, loss-level, and PCGrad-based aggregation across two humanoid-control frameworks. No aggregation level performs best across all environments and criteria. Experiments show that aggregating objectives early produces a wider range of behavioral responses but greater unintended changes in other objectives, while aggregating them later improves control precision at some cost to responsiveness.
PreferenceFlow, a framework for improving a pretrained flow policy at test time without environment rewards or updates to the base policy, andlations support the roles of gradient regularization and correctly ordered preference labels in the evaluated settings.
Yiqi Tang, Di-Yuan Shi, Run-Ze Li et al.· arXiv.org· 0 citations
Generalist robot policies increasingly learn from data collected across diverse tasks and embodiments, yet typically assume a single controller for each robot. This assumption limits transfer because different controllers can assign different physical meanings to the same action and may require different strategies to...
Jonathan Yang, Chelsea Finn, D. Sadigh· 0 citations
Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize s...
Kevin Frans, Alexander M. Rush, Sergey Levine et al.· 0 citations
Subjective user experience is important to human-robot interaction, but the outcomes users value, and how those preferences vary across individuals and contexts, are often unknown. While inverse learning approaches using human data can help identify user rewards, in many assistive settings the experimental costs of exe...
Kyeongwon Park, Steve H. Collins· arXiv.org· 0 citations
Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hardcodes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learnin...
Amr Mousa, Rifny Rachman, Neil Karavis et al.· 0 citations
Generalist robot policies and language-model agents provide complementary capabilities, but completing a task also requires adapting their use to the consequences of physical interaction. We study execution as a locus of adaptation: feedback guides the current action and supplies evidence for revising subsequent decisi...
He-Yang Yu, Hao-Min Li, Si-Hang Li et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.