Skip to content
Preprint

Steered Not Stirred: Effective Reward Tilting using Surrogate Reward Mixing

Oct 2026 · 0 citations
Computer Science

TL;DR

This paper introduces Mixing And Reward Tilting for INference-time Improvement (MARTINI), which learns a surrogate composition for plug-and-play use with standard inference-time steering methods, and characterizes surrogate information limits and relates population recovery error to ideal-target KL at small tilt strength.

Abstract

Continuous-time generative models, from diffusion models, flow matching, and now flow maps, owe their practical value to high-fidelity generation and steerability. In many applications, a ground-truth latent ``oracle''reward $R^*$ is unavailable or expensive to evaluate, e.g., human judgments or experimental measurements. Although the oracle reward is not directly observable, it can often be approximated through multiple cheaper surrogates that, individually, provide incomplete characterizations but collectively reveal substantial information about the downstream task. In this paper, we present a method for composing such surrogates to steer a pretrained model toward the distribution tilted by an unavailable $R^*$. Given a small feedback set of oracle rankings or numerical rewards, we introduce Mixing And Reward Tilting for INference-time Improvement (MARTINI), which learns a surrogate composition for plug-and-play use with standard inference-time steering methods. Our theory characterizes surrogate information limits and relates population recovery error to ideal-target KL at small tilt strength. For preferences, the bound additionally accounts for an irreducible mismatch between the Plackett-Luce population target and the conditional mean. Experiments demonstrate improved oracle-tilt fidelity in language generation and ImageNet alignment, and improved joint pose accuracy and validity in protein-ligand cofolding, without inference-time oracle queries.

View source

Similar papers

Diffusion Reward Models

Downstream RLHF experiments additionally show that using DRM as the training-time reward leads to improved policy performance, directly validating the practical benefit of diffusion-based reward modeling for RLHF training.

Xiang-Yang Wang, Bing-Xiang He, Ze-Yuan Liu et al. · 0 citations

Aligning One-Step Generative Models with Reward-Weighted Transport Distillation

Theoretical analysis shows that the fixed-point distributions of RWTD interpolate between off-policy reward tilting of the reference and on-policy tilting of the current model, providing a principled approach to balancing reward adaptation with retention of prior knowledge.

Austin S. Wang, Zi-Heng Cheng, Le-Xing Ying · 1 citation
#machine learning Preprint Oct 2026

Decoupled Marginal Sharpening for Training-Free Inference-Time Scaling

Decoupled Marginal Sharpening (DMS) is proposed, which separately controls how strongly the model favors particular reasoning paths and how strongly it favors final answers after aggregating support across those paths.

Wei-Xin Wang, Wei Deng, Anderson Schneider et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments

Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions...

Gu-Heon Kang, Minhae Kwon · 0 citations
Sep 2026

HiRE: Hindsight Reward Editing for Policy Finetuning

Hindsight Reward Editing (HiRE) bridges the broad knowledge of foundation representation models with physical control awareness, by contrasting successful and failed trajectories in hindsight, and consistently outperforms other reward recipes by delivering dense, control-aware feedback that prevents value function coll...

Haoyi Niu, Zhen Han, Yu-Feng Ji et al. · 0 citations
Sep 2026

WTF?! Simulation-Free Reinforcement Learning with Wasserstein-Tilted Flow Maps

Reward fine-tuning aims to update a pre-trained flow-based generative model to improve the downstream reward of its generated samples. Existing methods typically formulate this problem as sampling from a reward-tilted distribution, the solution to a KL-regularized reward-maximization problem. Here, we introduce an opti...

Abbas Mammadov, Jerry Huang, Justin Lin et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.