Skip to content

ControlScope: Workflow Revision and Reliability in LLM Agents

Sep 2026 · 2 citations · ⚡ 1 influential · 33 references
Computer Science

TL;DR

ControlScope compares continuing generated code, editing the next tool call's data arguments, and replacing the unfinished workflow from the same public execution state to evaluate one-time and repeated reviews across filesystem tasks, ALFWorld, and AppWorld.

Abstract

How much of a running workflow should a language model agent revise? ControlScope compares continuing generated code, editing the next tool call's data arguments, and replacing the unfinished workflow from the same public execution state. The nested permissions separate available repairs from the actions an agent selects. We evaluate one-time and repeated reviews across filesystem tasks, ALFWorld, and AppWorld. Across two source programs per task and three reasoning-reviewer draws on 20 filesystem tasks, FULL completes 15-16 tasks versus 13 for KEEP; across four fast draws it completes 10-13 versus 13. Fresh student-record confirmation reproduces a batch-read repair. ALFWorld fast panels yield KEEP/ARG/FULL scores of 85/86/87 on 87 tasks across 52 scenes and 134/134/127 on 134 tasks across four scenes; reasoning on the 87-task cohort also yields 85/86/87 with substantial review cost. An AppWorld V1 official-test panel of 585 task instances from 195 scenario templates shows small net differences. Frozen replays expose viable agent-written replacements interrupted by later revision in two failed file-organization runs. An offline source-trajectory midpoint comparison shows later reviews completing an insufficient repair. Five-call protection saves 19.4% of logged model output and loses one success across 20 fresh source runs. An argument-only shortcut shows that the broader sampled policy can overlook a cheaper successful edit available in both operation sets. These outcomes tie repair access to actual choices and subsequent execution.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Rules to Tools: Executable Checks for LLM Agents in Scientific Computing

Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements. Matched SciCode repair groups share written checks, starting programs, model, and bu...

Jing-Jie Ning, Guo-Jiang Zhao, Cheng-Xin Xu et al. · 1 citation
Preprint Aug 2026

CAPRI: Contract-Aware Proof Repair for Isabelle

CAPRI is presented, a contract-aware repair workflow in which Isabelle checks the proof and an independent checker enforces a machine-readable edit contract, in which Isabelle checks the proof and an independent checker enforces a machine-readable edit contract.

Jim Woodcock, Gabriel Leite, Augusto Sampaio et al. · 1 citation
Preprint Aug 2026

Metis: Typed Runtime Mediation for Tool-Using Software Agents

Metis, a multi-provider runtime that converts provider streams into typed events before admitted calls reach external effects is presented, a multi-provider runtime that converts provider streams into typed events before admitted calls reach external effects.

Jun'an Yu · 0 citations
Review Sep 2026

WideSWE: Can Coding Agents Coordinate Changes Across Repositories?

Coding-agent evaluation has progressed from resolving individual issues to carrying out long-horizon development, yet task completion is still largely assessed within a single codebase. In software ecosystems, many features and bug fixes require coordinated changes across multiple repositories. We introduce WideSWE to...

Bao-Yi Wang, Xing-Liang Wang, Jin-Yang Wu et al. · 0 citations

MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding

Recently, the rapid development of large language models (LLMs) has reshaped software engineering by enabling autonomous code agents that plan, execute, and utilize external tools iteratively to tackle complex tasks. Beyond achieving functional correctness, these agents must faithfully follow process instructions and c...

Bo-Si Wen, Cun-Xiang Wang, Jia-Yi Gui et al. · 1 citation

Vestrum: Improving Agent Harnesses by Adapting Their Verification, Structure and Memory

V Vestrum, a framework that turns failures in execution traces into scoped harness changes without training the task model, is introduced, a framework that turns failures in execution traces into scoped harness changes without training the task model.

Jayant Parashar, Eugene F. Douglass, William C. Bastian et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.