Sep 2026· Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track· Vol abs/2609.25237· 2 citations· 37 references
Computer Science
TL;DR
This work ran four frontier agents end to end end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring, and ran a human FDE arm under the same oracle and compare every agent against it.
Abstract
Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned, evaluated, and deployed model under a budget, a human-approval gate, and reproducibility requirements. Seating an LLM agent in the FDE seat raises a question existing benchmarks cannot answer: not whether an agent can raise a metric, but whether it can be trusted to deliver. We answer it on a governed delivery plane, where an agent drives ten stages and an oracle scores each stage from platform-recorded facts. The central silent failure is the run that trains but does not learn (TBDL): loss falls, every signal stays green, and the delivered model is no better than the base. An operator-run acceptance gate catches every such run before payment, and a detector calibrated on known-corrupted runs flags severe corruption mid-run. We ran four frontier agents (Claude Opus 5, GPT-5.6-luna, Gemini 3.7 Flash, DeepSeek V4-Pro) end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring. We also ran a human FDE arm under the same oracle and compare every agent against it.
ProCTOR is described, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply, and a Teacher grades those mutations under five deterministic guardrails: hermetic sandboxes, capability-disjoint roles, acceptance checks tha...
K-Bench is introduced, a benchmark that scores LLM unlearning under agentic deployment and certifies forgetting by reading the model's final answer, where a model that refuses to answer already counts as having forgotten.
Guang-Sheng Yu, Yan-Na Jiang, Qin Wang et al.· 0 citations
Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input carries an injection. System-1 decision models answer such questions in a single forward pass with class probabilities, promising large cost and latency savings over LL...
This work introduces StoreBench, a live-commerce environment in which an agent runs a mid-size online apparel store on a production-grade commerce backend, testing long-horizon planning and economic judgment under uncertainty.
Daksh Raghuvanshi, Ved Vedere, Yi-Fan Wang· 1 citation
Analyzing a large corpus of publicly released post-training trajectories, it is found that across different tasks, the agent's training strategy is locked in at the very beginning, and the entire remaining budget is spent on local adjustments within the selected strategy.
J. Lim, Xin Huang, Hao Peng et al.· 2 citations· ⚡1
Results characterize when runtime-controlled composition of coding-agent products improves execution accuracy, as well as when runtime-controlled composition of coding-agent products improves execution accuracy.
Zheng-Yu Chen, Lin-Feng Liu, Hong Li et al.· arXiv.org· 0 citations
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.