Skip to content
Preprint

One Policy, Many Styles: Where to Aggregate Preferences for Steerable Humanoid Control?

Oct 2026 · 0 citations · 16 references
Computer Science

TL;DR

This evaluation combines Preference Reward Responsiveness (PRR) with coverage and utility metrics to compare reward-level, advantage-level, loss-level, and PCGrad-based aggregation across two humanoid-control frameworks.

Abstract

Modern robotic agents are increasingly expected to support steerable control, where a single policy can adjust its behavioral style at runtime under changing user preferences. Preference-conditioned reinforcement learning provides this interface, but conditioning alone does not ensure that a requested behavior change is strong, does not leak into unrelated behaviors, and does not compromise task performance. Our evaluation combines Preference Reward Responsiveness (PRR) with coverage and utility metrics. We use these measures to compare reward-level, advantage-level, loss-level, and PCGrad-based aggregation across two humanoid-control frameworks. No aggregation level performs best across all environments and criteria. Experiments show that aggregating objectives early produces a wider range of behavioral responses but greater unintended changes in other objectives, while aggregating them later improves control precision at some cost to responsiveness.

View source

Similar papers

Sep 2026

PreferenceFlow: Test-Time Guidance of Flow-Matching Robot Policies from Human Interventions

PreferenceFlow, a framework for improving a pretrained flow policy at test time without environment rewards or updates to the base policy, andlations support the roles of gradient regularization and correctly ordered preference labels in the evaluated settings.

Yiqi Tang, Di-Yuan Shi, Run-Ze Li et al. · 0 citations
Preprint Oct 2026

Cross-Controller Transfer via In-Context Behavior Adaptation

Generalist robot policies increasingly learn from data collected across diverse tasks and embodiments, yet typically assume a single controller for each robot. This assumption limits transfer because different controllers can assign different physical meanings to the same action and may require different strategies to...

Jonathan Yang, Chelsea Finn, D. Sadigh · 0 citations
#machine learning Preprint Nov 2026

Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning

Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize s...

Kevin Frans, Alexander M. Rush, Sergey Levine et al. · 0 citations
Sep 2026

Simultaneous Forward and Inverse Human-in-the-Loop Optimization

Subjective user experience is important to human-robot interaction, but the outcomes users value, and how those preferences vary across individuals and contexts, are often unknown. While inverse learning approaches using human data can help identify user rewards, in many assistive settings the experimental costs of exe...

Kyeongwon Park, Steve H. Collins · 0 citations
#artificial intelligence Preprint Oct 2026

PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots

Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hardcodes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learnin...

Amr Mousa, Rifny Rachman, Neil Karavis et al. · 0 citations
Preprint Oct 2026

RoboDoctor: Training-Free Recursive Self-Improvement for Robot Policies

Generalist robot policies and language-model agents provide complementary capabilities, but completing a task also requires adapting their use to the consequences of physical interaction. We study execution as a locus of adaptation: feedback guides the current action and supplies evidence for revising subsequent decisi...

He-Yang Yu, Hao-Min Li, Si-Hang Li et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.