Skip to content
Book Open access

AnonAgentBench: Measuring Coding-Agent Utility Under Reversible Repository Anonymization

Nov 2026 · Proceedings of the 1st Workshop on Security, Privacy, and Safety of Agentic AI Systems · 0 citations · 9 references

Abstract

Organizations may want to use hosted coding agents on private repositories. However, a coding session can expose sensitive text through file reads, search results, test logs, stack traces, and patches. Cleaning only the initial prompt leaves these later channels exposed. We introduce AnonAgentBench, a benchmark for evaluating repository anonymization across the complete agent interaction. The benchmark replaces planted sensitive strings before the agent sees them. It restores the original strings locally in the generated patch and evaluates that patch on the original task. The private mapping remains in the local harness. We test three forms of full-loop anonymization. Level B replaces protected strings throughout the interaction. Level C uses consistent placeholders across all channels. Level D adds limited type or role hints to those placeholders. Across 488 available prompt-only traces, the configured scanner found 3,375 Devstral and 2,852 Qwen exact planted-string occurrences. It found none in 1,462 available full-loop traces; this zero is a deterministic coverage and implementation check over observed model-visible channels. A separate inventory-independent fragment/Gitleaks audit of the same full-loop traces found no task-specific fragments (minimum nine characters) or stock-rule matches. Without anonymization, Devstral solved 235 of 500 tasks, and Qwen solved 259 of 500. We then applied full-loop anonymization to those solved tasks. Devstral continued to solve 127–141 of 235 tasks (54.0–60.0%), while Qwen continued to solve 197–203 of 259 tasks (76.1–78.4%). These raw-resolved percentages are conditional retention, not full-denominator or deployment estimates. In two sensitivity comparisons, positive submitted-instance differences disappeared or reversed after excluding pairs with an empty patch or evaluator error. AnonAgentBench therefore benchmarks observed-channel exact-string coverage, private restoration, and conditional utility; it does not establish semantic or formal privacy.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.