Skip to content
Preprint

Hidden in Plain Sight: an Empirical Study about The Instruction Stacks of Coding Agents

Oct 2026 · 0 citations · 86 references
Computer Science

TL;DR

A longitudinal, cross-agent study of the instruction stacks captured in the first model requests of 1,886 releases of Claude Code, Codex, Gemini CLI, Qwen Code, and OpenCode, exposing an evolving, incompletely documented layer of agent design.

Abstract

Coding agents send their models more than the user's prompt: they add instructions, tool definitions, and environmental context that form an instruction stack. This stack guides execution and changes across releases, yet remains largely unexposed to users and insufficiently characterized in research. We present a longitudinal, cross-agent study of the instruction stacks captured in the first model requests of 1,886 releases of Claude Code, Codex, Gemini CLI, Qwen Code, and OpenCode. Under controlled conditions, we analyze their composition and evolution and trace changes in the four open-source agents to documented motivations. Our findings show why examining the system prompt or reporting the model alone provides an incomplete account of agent inputs. Tool definitions dominate three agents'latest requests, while occasional concentrated revisions alter instructions governing tool use, attribution, and coding style. Such revisions can complicate how researchers identify agent-authored contributions and interpret generated code. We also found that only 48% of examined change units directly explain the motivation for stack revisions. Across agents, shared capabilities emerge despite increasingly distinct wording. These findings expose an evolving, incompletely documented layer of agent design. The findings motivate documenting captured instruction stacks alongside agent releases, models, and user prompts, and preserving the rationale behind their revisions.

View source

Similar papers

Sep 2026

Merged, Not Measured: An Empirical Study of Performance Issues Fixed by Coding Agents

Coding agents open pull requests (PRs) that claim to speed up software, but studies of human performance fixes say little about how maintainers respond to such a fix or whether its claim holds. From the 71,677 agent PRs of AIDev v4, a text filter and codebook coding by language models and by the authors select 1,262 pe...

Zhen-Yu Qi, Haotang Li, Jin-Fu Chen et al. · 0 citations
Review Oct 2026

Do Coding Agents Catch Their Own Bugs? (They Don't.)

In software development, engineers need to foresee and fix potential problems with their code that are not explicitly specified in requirements---such as pre-emptively considering vulnerabilities or other bugs from a code change. But current benchmarks often do not measure these future-looking consequences. We introduc...

Shaswat Patel, Michel Liao, Peter Henderson et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Position, Role or Label? A Controlled Study of Instruction Conflicts in LLM Agents

LLM agents take instructions from the operator, repository configuration, user, tool results and saved memory, which often conflict. Instruction-hierarchy benchmarks ask whether the higher-ranked source wins, but a source sets position, role and label at once. First, we build WhoWins, 1,080 code-scored scenarios in fiv...

Anish Laddha, Nitesh Pradhan, Gaurav Srivastava · 2 citations
Preprint Oct 2026

SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?

Coding agents increasingly interact with users by generating interactive software on the fly rather than replying in text, yet many bugs in such software reveal themselves only after multi-step interactions. Today, users must discover these bugs and wait for fixes, making the development loop slow and expensive. Existi...

Yang Yao, Hao-Zhe Chen, Bing-Yi Kang et al. · 1 citation
#artificial intelligence Review Oct 2026

What Open-Source Projects Tell Their Coding Agents: A Census of AGENTS.md and CLAUDE.md

Coding agents can be guided with instruction files in a repository. We examine this emerging practice through a census of 22,854 popular, active GitHub repositories in thirteen languages, treating each repository's root'AGENTS$.$md'and'CLAUDE$.$md'as one instruction set. Agent instructions appear in 26.3% of repositori...

Pawel Zarzycki · 1 citation
Preprint Oct 2026

UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users

As coding agents become more capable, developers increasingly use them to fix bugs through multi-turn interactions. Existing benchmarks do not adequately cover this scenario. Most present a complete issue report in a single turn without interaction. Their evaluation can produce both false positives and false negatives,...

Zhan-Yu Wu, Ji Luo, Alyssa Wu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.