Token Reduction Is Not Cost Reduction

Abstract

Token-reduction tools for coding agents are often evaluated by the number of tokens they remove, but token count alone does not determine end-to-end inference cost. We evaluate three token-reduction approaches against an unmodified Claude Code baseline across controlled coding tasks, measuring provider-billed cost, task success, cache traffic, and agent behavior. The largest compression setup reduced delivered tool-output tokens by 38.4% but increased billed cost by 6.8%, while lighter compression produced only small and statistically uncertain savings. Across tasks, token reduction was weakly correlated with cost reduction (Pearson r = 0.15). Cost decomposition shows that prompt-cache creation and reads dominate the measured input-side cost, leaving only a limited fraction of total spend directly addressable by tool-output compression. We also find that compression can alter agent trajectories through additional retrieval, diagnosis, testing, and turns, offsetting local token savings. On a SWE-bench Go subset, aggressive compression also reduced successful patch application. These results show that token reduction is not a reliable proxy for cost reduction in tool-heavy coding agents. Effective optimization should therefore be evaluated at the level of cost per successful task, including cache behavior, trajectory changes, and correctness rather than token counts alone.