Skip to content

Making Agents More Consistent: Skills Should Form Habits for Repeat Tasks

Sep 2026 · arXiv.org · Vol abs/2609.25299 · 0 citations · 41 references
Computer Science

TL;DR

This work proposes skill habit formation, a model that mines its own execution history for candidate skills, deterministic variants that compete against the incumbent rather than replacing it, and measures what this costs in accuracy.

Abstract

On repeated work, agents are inconsistent. We ran 42 tasks three times each and found that, depending on the model, 38% to 74% returned answers that did not agree. Consistency is what a buyer, an auditor, or a regulator requires, and agents do not have it. They are wasteful too: 95.3% to 97.2% of what an agent generates goes to re-deriving a plan the system already knows. We propose skill habit formation. An agent mines its own execution history for candidate skills, deterministic variants that compete against the incumbent rather than replacing it. A candidate declares the region of input space it claims, so the common case runs as a script and the rest falls through to reasoning. Four gates of ascending cost admit candidates; the central one tests a candidate's execution trace against a retained reference, within a tolerance measured from that reference's own run-to-run variability. On text-to-SQL, three of four reasoning arms reproduced their own output on 11 to 13 of 42 repeated questions and the fourth on 26 of 42, while a habit-formed variant reproduced on all 456 dispatches we repeated and was non-inferior to every arm it replaced (p<0.0001). It also used 14% to 56% fewer tokens, turning net positive after 7 to 53 reuses. We measured what this costs in accuracy. The guard admitted work it should have deferred on 2.6% of natural paraphrases and 26% of inputs near its boundary, and 11 of 13 such failures were invisible to the trace-conformance gate at any threshold. Deterministic errors repeat exactly: a bad habit is as reliable as a good one, and that is the price of the property that makes the system auditable. Separating routing from parameter extraction raised end-to-end accuracy from 0.888 to 0.952 at 43% of the cost.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

Agent evaluations should be evaluated as configurable systems themselves, and that some desired behaviors are more effectively implemented in the system than requested through prompting, according to a new benchmark of four scientific tasks.

Luis Wiedmann, Leander Girrbach, Cordelia Schmid et al. · 0 citations

How Strongly Should Task State Influence an LLM Agent?

Across three models, two reasoning regimes, and two domains, four findings hold without per-turn reasoning: displaying accurate state is unreliable, an unverified ledger the agent writes itself beats an accurate checklist it is shown, directives help in proportion to the model's obedience, and enforcement needs no obed...

C. Zhang, Wonbin Kweon, Jia-Wei Han · 0 citations

Who Holds the Pen? Let Specifications, Not Agents, Sign Off

Experiments on guideline-following and artifact-generation tasks show that specifications can serve not merely as behavioral guidance, but as authority over compliant execution and completion.

Hai-Qing Li, Xin-Yu Ma, Yin-Hao Wu et al. · 0 citations
Preprint Aug 2026

What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Files

A deterministic routing stress test over 20,000 skills shows the functional impact: skills with valid routing metadata are retrieved more reliably from startup descriptions than skills with routing defects, while AI-marked skills show more safety and portability problems.

Chi Zhang, Yimin Liu, Xinze Chen et al. · 0 citations
Preprint Aug 2026

TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents

Results show that TRACE converts high model potential into stable, consistent performance gain, and bridge the gap between potential and reliable performance to just 4.0 points.

Wen-Hao Wu, Meng-Hao Zhang, X. Wang et al. · 4 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.