Skip to content

Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers

Sep 2026 · Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track · Vol abs/2609.25237 · 2 citations · 37 references
Computer Science

TL;DR

This work ran four frontier agents end to end end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring, and ran a human FDE arm under the same oracle and compare every agent against it.

Abstract

Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned, evaluated, and deployed model under a budget, a human-approval gate, and reproducibility requirements. Seating an LLM agent in the FDE seat raises a question existing benchmarks cannot answer: not whether an agent can raise a metric, but whether it can be trusted to deliver. We answer it on a governed delivery plane, where an agent drives ten stages and an oracle scores each stage from platform-recorded facts. The central silent failure is the run that trains but does not learn (TBDL): loss falls, every signal stays green, and the delivered model is no better than the base. An operator-run acceptance gate catches every such run before payment, and a detector calibrated on known-corrupted runs flags severe corruption mid-run. We ran four frontier agents (Claude Opus 5, GPT-5.6-luna, Gemini 3.7 Flash, DeepSeek V4-Pro) end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring. We also ran a human FDE arm under the same oracle and compare every agent against it.

Read PDF

Similar papers

#artificial intelligence Review Sep 2026

LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails

ProCTOR is described, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply, and a Teacher grades those mutations under five deterministic guardrails: hermetic sandboxes, capability-disjoint roles, acceptance checks tha...

Vansh Wahi · 3 citations
Preprint Sep 2026

K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments

K-Bench is introduced, a benchmark that scores LLM unlearning under agentic deployment and certifies forgetting by reading the model's final answer, where a model that refuses to answer already counts as having forgotten.

Guang-Sheng Yu, Yan-Na Jiang, Qin Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses

Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input carries an injection. System-1 decision models answer such questions in a single forward pass with class probabilities, promising large cost and latency savings over LL...

Jia-Wei Li · 13 citations · ⚡1
#artificial intelligence Review Aug 2026

What is Missing from AI Post-Training AI: An Empirical Analysis

Analyzing a large corpus of publicly released post-training trajectories, it is found that across different tasks, the agent's training strategy is locked in at the very beginning, and the entire remaining budget is spent on local adjustments within the selected strategy.

J. Lim, Xin Huang, Hao Peng et al. · 2 citations · ⚡1

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.