AnonAgentBench: Measuring Coding-Agent Utility Under Reversible Repository Anonymization
Abstract
Organizations may want to use hosted coding agents on private repositories. However, a coding session can expose sensitive text through file reads, search results, test logs, stack traces, and patches. Cleaning only the initial prompt leaves these later channels exposed. We introduce AnonAgentBench, a benchmark for evaluating repository anonymization across the complete agent interaction. The benchmark replaces planted sensitive strings before the agent sees them. It restores the original strings locally in the generated patch and evaluates that patch on the original task. The private mapping remains in the local harness. We test three forms of full-loop anonymization. Level B replaces protected strings throughout the interaction. Level C uses consistent placeholders across all channels. Level D adds limited type or role hints to those placeholders. Across 488 available prompt-only traces, the configured scanner found 3,375 Devstral and 2,852 Qwen exact planted-string occurrences. It found none in 1,462 available full-loop traces; this zero is a deterministic coverage and implementation check over observed model-visible channels. A separate inventory-independent fragment/Gitleaks audit of the same full-loop traces found no task-specific fragments (minimum nine characters) or stock-rule matches. Without anonymization, Devstral solved 235 of 500 tasks, and Qwen solved 259 of 500. We then applied full-loop anonymization to those solved tasks. Devstral continued to solve 127–141 of 235 tasks (54.0–60.0%), while Qwen continued to solve 197–203 of 259 tasks (76.1–78.4%). These raw-resolved percentages are conditional retention, not full-denominator or deployment estimates. In two sensitivity comparisons, positive submitted-instance differences disappeared or reversed after excluding pairs with an empty patch or evaluator error. AnonAgentBench therefore benchmarks observed-channel exact-string coverage, private restoration, and conditional utility; it does not establish semantic or formal privacy.