Skip to content

RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

Sep 2026 · arXiv.org · Vol abs/2609.39551 · 0 citations · 61 references
Computer Science

TL;DR

Results characterize when runtime-controlled composition of coding-agent products improves execution accuracy, as well as when runtime-controlled composition of coding-agent products improves execution accuracy.

Abstract

Auto-research agents, LLM systems that propose, implement, train, and evaluate model changes across iterations, promise to automate applied ML's experimental loop. Over long horizons, execution accuracy is a binding constraint: a change can silently leak held-out data, omit normalization, disconnect a gradient, or leave a train/eval flag unwired, invalidating expensive runs and compounding error across iterations. We present RankEvolve, an auto-research framework for evolving generative ranking models. An Executable Operating Protocol (EOP) declares phases, gates, branches, and loops, and the runtime enforces the compiled state machine. A meta-meta-harness composes complete black-box coding-agent products, including Claude Code and Codex, as execution-graph nodes that review and repair one another's work. In a budget-matched evaluation, heterogeneous composition raises all-oracle execution accuracy from the best single-product baseline of 45.8 percent to 62.5 percent (paired +16.7 points, 95 percent CI [6.6, 26.7]) while achieving a 10.4 percent silent critical-defect rate. An implemented knowledge layer carries findings, including negative results, across iterations. In a twelve-iteration deployment on the open-source HSTU recommender, RankEvolve reported NDCG@10 of 0.2192 on MovieLens-20M LARGE (+4.48 percent over the published anchor) and 0.1948 on BASE (+2.80 percent). ExecML-HSTU, seeded by incidents from that deployment, provides the oracle benchmark for the execution-accuracy evaluation. A pre-specified LitGPT transfer split replicates the heterogeneous-composition effect beyond recommendation (+12.5 points, 95 percent CI [3.0, 22.0]), and a paired ablation isolates per-step from full-protocol instruction injection. These results characterize when runtime-controlled composition of coding-agent products improves execution accuracy.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

AutoSFT: An Agentic System for Automated Data Research in Supervised Fine-Tuning

Language model agents increasingly automate post-training, but when given the full training stack they converge on supervised fine-tuning and spend most effort revising training data. Existing systems give the agent one kind of control, selecting rows from a fixed pool or calling a generator, whereas a human data engin...

Bing-Xiang He, Peng Sun, Shi-Hao Peng et al. · 0 citations
#natural language process... Preprint Sep 2026

EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

This work introduces early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task within each task, and instantiates EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features and hal...

Yu-Ling Shi, Zhensu Sun, Jun-Sen Dong et al. · 4 citations · ⚡1
#artificial intelligence Preprint Sep 2026

Agentic ML Exploration (A-MLE) for Ads Ranking

Agentic ML Exploration (A-MLE), an autonomous LLM-agent system that systematically explores ML techniques across a portfolio of ads ranking models, is presented and suggested that agentic exploration is a practical force multiplier for ML engineers in industrial recommenders.

Erwin Gao, Vinodh Kumar Sunkara, Jing Guan et al. · 0 citations
Preprint Aug 2026

AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

The proposed AutoSaddler, an automatic harness optimization framework that formulates harness improvement as an offline learning problem and iteratively updates the harness using failure signals from mini-batches, suggests that automatic harness optimization is a promising path toward more performant and reliable agent...

Sungho Park, Wonjoong Kim, Rongyuan Tan et al. · 17 citations · ⚡1
#natural language process... Open access Oct 2026

SEDIMA: Cross-Run Hierarchical Insight Memory for Evolutionary Search Agents

SEDIMA is introduced, a persistent hierarchical insight memory for evolutionary search agents that distills raw traces into natural-language insights, clusters them by semantic similarity using attention-weighted centroids, and retrieves relevant guidance to condition future mutations, accumulating transferable knowled...

Amirhossein Abaskohi, Mahdi Mostajabdaveh, Zi-Rui Zhou · 0 citations
#machine learning Preprint Oct 2026

Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents

Retrieval-augmented generation (RAG) is a widely used approach for grounding large language models (LLMs) in external knowledge. However, configuring a pipeline is an expensive hyperparameter optimization problem over many interacting choices, from chunking and embedding model to reranking and generation. Existing opti...

Lasse B. Strand, Robert Jakob, Kevin O'Sullivan et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.