ScopeBench is introduced, a benchmark of 30 dead-end agentic security tasks in which the stated objective is reachable only by violating the stated scope, and the frozen pilot benchmark, evaluation code, and all 2160 ATIF trajectories are released.
Shane Caldwell, Max Harley, Ads Dawson et al.· 1 citation
Stealth, the discipline of achieving an objective without revealing your presence, capabilities, or collected intelligence, is what separates sophisticated operators from detectable ones. Elite security researchers and advanced persistent threats achieve their objectives unnoticed; autonomous agents increasingly inheri...
The "solve rate"metric (clean passes only) is introduced to distinguish genuine capability from cheated outcomes, and it is argued it should be standard practice in any evaluation where cheating vectors are available.
Michael Kouremetis, Ads Dawson, Raja Sekhar Rao Dheekonda et al.· arXiv.org· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.