PREreview of "The Role of Agentic Instruction Categories for Code Generation: A Study of Five Python Libraries"

Abstract

This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/23446298. Summary. The authors test five instruction categories for an AGENTS.md file (Table I) on five small Python libraries (Table II), using Gemma 4-26B-A4B and GPT-5.4. They cover all 31 non-empty combinations, so 310 agent files are generated with the OpenCode harness. Each library is then rebuilt from scratch using only that file (Sections II-B, II-C). Token usage for writing the files depends on the LLM and the library, not on the number of instructions (Section III-A). Compile rates are 88.39% for Gemma 4 and 98.06% for GPT-5.4, but no Gemma 4 variant passes all human tests and 12.99% of GPT-5.4 variants do (Section III-B). I liked that the original human-written tests are used unmodified as ground truth (Section II-C c), that the threats section is candid about one run per condition, and that the replication package with all agent files is public (Section VII, Zenodo DOI 10.5281/zenodo.23359831). I read it as a practitioner who runs coding agents every day. In my company the software development is done by AI agents, so maybe my practical side is useful here. Four comments. 1. On which cost is measured - RQ1 counts tokens spent generating the agent files, and the authors say this is only the economic cost of writing them (Sections III-A, IV-B a). In my measurement of 722 agent sessions (150,902 model calls, DOI 10.5281/zenodo.22759216), more than 85% of the modeled cost was work with context (cache reads and writes), not output. In a separate run of 168 sessions, 94.4% of billed tokens were re-reading context already sent (contextburn, DOI 10.5281/zenodo.22712985) - a share of tokens, not of the bill. An agent file sits in that context, so in my experience its cost shows up in the sessions that use it, call after call. From the replication package, can you report the length of each generated agent file and the token usage of each reconstruction session? 2. On length and relevance - the authors conclude that relevance matters more than the number of categories, and that agents could select instructions for the current task (Sections IV-A b-c). In my guides I keep CLAUDE.md or AGENTS.md short, up to about 200 lines, because a bloated file is followed worse, and specific rules are followed more reliably than vague ones. Things needed only now and then go into separate skills loaded on demand. I test every line with one question - if I remove it, will the agent start making mistakes? If not, it goes. Could you run a removal test of that kind per category against the human tests, and use file length as a covariate, since categories leak into each other (Section IV-B b)? 3. On a check the agent can run - during reconstruction the working directory held only the agent file, and the human tests were run afterwards for grading (Sections II-C a, II-C c). The Testing category gives only examples of test cases or verification methods (Table I). In my guides I write that every task needs its own check the agent can run; without it, "looks done" is the only signal the agent has. The distance between the compile rates and the full-pass rates (Table V, Table VI) looks like that gap to me. Related work like Commit0 shows execution feedback helping (Section V-C). Could you add one condition where the agent can run part of the human test suite and the rest is held out for grading? 4. On repeats - each library and combination has one reconstruction run (Section IV-B b), and the lead of the best category is only 1.25 points for Gemma 4 and 3.75 points for GPT-5.4 (Section III-B b). In my own experiment on clearing agent context I ran six policies with six repeats each, 36 runs (DOI 10.5281/zenodo.22759217). One policy came out about a third more expensive (+33.5%, p = 0.002, 0.011 after Holm correction), while the difference for "never clear" was not significant. With one run per condition, such small leads may be noise. Could you run three to five repeats for the best and runner-up combinations only, and report an interval on the pass rates? Thank you for a clear and useful paper. Competing interests The review cites the reviewer's own technical reports (DOI 10.5281/zenodo.22759216, 10.5281/zenodo.22712985, 10.5281/zenodo.22759217). No connection to the authors of the preprint. Use of Artificial Intelligence (AI) The author declares that they used generative AI to come up with new ideas for their review.