Skip to content

Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization

Sep 2026 · arXiv.org · Vol abs/2609.32394 · 1 citation · 63 references
Computer Science

TL;DR

An Agentic Reward Black-box Optimization (ARBO) framework is proposed, in which an LLM agent builds the search strategy at run time from an evaluation history maintained as its persistent workspace, which comprises two components: observations maintained by the evaluation oracle and an agent-maintained belief that records diagnoses and intended next steps.

Abstract

Designing dense reward functions for low-level reinforcement learning (RL) control remains difficult. Recent work uses large language models (LLMs) to iteratively generate and refine reward functions using policy-training feedback within scripted search algorithms. However, evaluating each candidate requires a full RL training run, making sample efficiency a central challenge for reward search on complex control tasks. To address this limitation, we propose an Agentic Reward Black-box Optimization (ARBO) framework, in which an LLM agent builds the search strategy at run time from an evaluation history maintained as its persistent workspace. The evaluation history comprises two components: observations maintained by the evaluation oracle, including candidate scores, per-term training curves, and error tracebacks; and an agent-maintained belief that records diagnoses and intended next steps. The agent queries both with tools and generates the next batch of reward candidates, rather than generating them in a single pass from a fixed prompt. Across four control domains, ARBO achieves gains of 29.9% in manipulation success rate and 192.8% in power-grid score over baseline means under a shared evaluation budget. Ablations examine each component's contribution and sensitivity to backbone choice.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks

T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier by executing each task's own verifier.

Jun-Yao Yang, Yu-Cheng Shi, Zhong-Zhi Li et al. · 9 citations
Preprint Aug 2026

Harness-RL: Black-Box Reinforcement Learning with Action-Args Decoupling for Central-Agent Multi-Agent Harnesses

Harness-RL is introduced, a structured reinforcement learning framework that combines Conflict-Aware Policy Optimization (CAPO) with interface-level black-box trajectory construction and supports both central-only and joint multi-agent training.

Xin-Ke Jiang, Zhi-Xin Zhang, Zhi-Bang Yang et al. · 1 citation
Preprint Oct 2026

Reward Engine: Reward-Centric Self-Evolution for Open-Ended Tool Agents

Open-ended tool agents can complete the same objective through different tool paths, and their quality depends jointly on the interaction trace, environment state, and final artifacts, so such tasks rarely admit an executable ground-truth reward. The central challenge is therefore to construct a reward from existing ex...

Linda Feng, Ji Qi, Yi-Ran Cheng et al. · 0 citations
#artificial intelligence Preprint Oct 2026

RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation

RewardWeaver is introduced, a self-evolving reward adaptation framework for language agents in long-horizon interaction that maintains a validated capability space in which the semantics of admitted Rubrics remain fixed, and closes the loop between policy optimization, task evaluation, failure attribution, and reward a...

Heng-Bo Xiao, Bo-Yao Zhang, Pu-Rui Liu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents

Multimodal agents commonly generate free-form reasoning before each action. For small models, limited model capacity can result in lengthy reasoning that provides little useful guidance for action generation while incurring substantial inference cost. To address this challenge, we introduce Selection-based Structured R...

Feiyu Zhu, Xiao-Yu Zhu, Ji-Qian Yang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

FARL: A Fully Automated Pipeline of RL Interface and Solver Design

Fully Automated Reinforcement Learning is introduced, a framework that automates RL design by jointly searching observation and reward programs, learning algorithms, and training settings and outperforms the evaluated automated baselines on challenging navigation, locomotion, and manipulation tasks.

Tian Jin, Liu-Yu Xiang, Chao-Wen Yang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.