This primer draws on fieldwork in a computational biology laboratory to examine what human oversight of AI agents requires in practice and shows that effective oversight has four components: adequate knowledge of system capabilities and limitations, sufficient observation of system actions, meaningful control of system behavior, and timely intervention in system failures.
Abstract
As AI agents move from speculative promise into daily deployment, their relative autonomy challenges the traditional means of human oversight. When a user gives an agent goals, they also give it room to decide how those goals should be pursued. If the user’s goal and the agent’s execution drift apart, failures — including altered data, deleted files, and misallocated resources — can become consequential. Because agents move fast and can act across many systems at once, a small mistake can cascade through workflows.
This primer draws on fieldwork in a computational biology laboratory to examine what human oversight of AI agents requires in practice. Our research shows that effective oversight has four components: adequate knowledge of system capabilities and limitations, sufficient observation of system actions, meaningful control of system behavior, and timely intervention in system failures. Crucially, intervention becomes meaningful only when users have enough knowledge, visibility, and control to act before small divergences become consequential failures.
To address the challenges of effective oversight, we must move beyond treating it as an individual user burden to framing it as a distributed responsibility — one shared by builders and deployers. We hope this primer contributes to a broader public conversation about how to create the technical and organizational conditions needed to keep AI agents accountable and governable.
Deploying agentic AI in regulated contexts requires knowing two things about a deployment: what the system can do—its agency—and how much it acts without human involvement— its autonomy. Though often treated independently, the two are coupled: at higher autonomy, human error correction is less available, so reliable operation requires constraining agency accordingly, and compliance rules reinforce this by mandating human involvement as the consequences of actions grow. Yet no established approach addresses them jointly as a design problem, leaving practitioners without a principled basis for deciding where oversight should sit and how errors can be caught before they propagate. We introduce a two-dimensional design space in which both dimensions are organised into five operational levels, making the coupling explicit and navigable, and we propose six architectural tactics—checkpoints, escalation, multi-agent delegation, tool provisioning, tool fencing, and write staging—for adjusting a deployment’s position within it. We ground the tactics in a public-sector document classification system, tracing a path from manual operation to near-full autonomy under realistic compliance constraints. Together they offer a shared vocabulary for compliance-aware agentic AI design in which responsibility, auditability, and reversibility are explicit design choices rather than retrofitted properties.
AI agents pose significant risks as they are granted increasing autonomy. A commonly proposed solution is human oversight and keeping a''human in the loop'', but this is not a simple solution: Not only do current approaches to AI agent design impede effective human oversight, but the cognitive capacities required for it are also themselves degraded by extended use of AI systems. This position paper argues that current approaches to the development and deployment of AI agent systems do not support effective human oversight -- they contribute to its degradation. To address this, a top priority in the advancement of AI agents should be supporting the situated goals and cognitive requirements of effective human oversight, treating the human needs of overseers at the same level of importance as AI agent capability. To put this idea into practice, we connect work on automation and human-computer interaction to AI agent processes, outlining design-level affordances and organizational protocols that (1) support overseers in exercising critical judgement and (2) counteract the skill atrophy that arises from extended use of automation. We urge developers and deployers to adopt these or similar approaches. Without explicit support for the cognitive demands of effective human-agent interaction, AI agent systems will continue to passively incentivize the degradation of the very human skills they rely on.
Margaret Mitchell, Avijit Ghosh, Samir Passi· 0 citations
AI agents are poised to become a primary interface to digital products, acting across email, files, payments, and personal data. People without professional software backgrounds need understandable, reusable ways to control actions across services. We examine a mechanism in which a language model maps actions to plain-language consequence categories with user-authored"allow","ask", or"never"rules. We ask what is gained and lost when decisions are made in advance as reusable rules rather than separately for each action. We analyzed 113 participants without professional software backgrounds across three conditions: per-action human-in-the-loop approval (HITL), automated per-action model review (AUTO), or user-authored consequence policy (POLICY). Participants judged 2 examples in each of 4 consequence categories; POLICY participants then set one rule per category. All supervised an 18-action simulated day, including 7 overreach actions. POLICY blocked less overreach than HITL (-20.1 percentage points, 95% CI [-32.1, -8.1]) and AUTO (-14.5 points, 95% CI [-25.8, -3.2]). POLICY lowered runtime prompts from 18.0 to 10.9, but total intervention time was not reliably lower when rule setup was included. Exploratory analysis showed that participants chose"ask"for 114 of 140 POLICY rules, returning most overreach actions to runtime. Of the 148 overreach actions executed in POLICY, 133 followed human approval and 15 ran automatically under"allow"rules. Across all 7 overreach actions, POLICY had the highest approval rate. Counterintuitively, user-authored rules did not by themselves provide stronger protection: many actions outside users'original requests went through after users approved them. These results reveal a gap between preference and commitment: repeatedly choosing"ask"preserves case-by-case choice but prevents a standing policy from settling decisions in advance.
Enterprise deployments of autonomous AI agents inherit a control model built for human users and long-lived services, and the fit fails in three specific ways: agent principals are ephemeral, appearing and vanishing faster than provisioning; their actions are selected by a model rather than programmed, so the set of things they may attempt is not known in advance; and the population is discovered rather than provisioned, because anyone who can call an API can create one. We argue that governing such agents is a runtime problem -- not a model-alignment problem and not a build-time problem -- and we derive five primitives from the questions that must be answered before an action takes effect and after it has: discovery, identity, governance, attestation, and supply chain. For each we state what fails if it is absent and why the others cannot structurally supply it. We describe an implementation in which an agent's action is mediated against policy before it takes effect, authorised against a per-tenant action vocabulary, and recorded in a hash-linked signed ledger a third party can verify with the vendor out of the loop. We report what the architecture costs: the enforcement point sits on the request's critical path, identity requires a sidecar per workload, and fail-closed mediation converts availability incidents into denial. We are explicit about implementation status: four primitives are built and running in private pilots, and the fifth is built as separate tooling and not yet integrated into the request path. We keep it in the set deliberately: a five-part decomposition that exactly matches what its authors happened to build is not a taxonomy but a description of a codebase.
These findings establish context measurement as a validated preflight signal for agent reliability and position context engineering as an auditable layer of agent evaluation and governance.