Skip to content
Preprint

V$^3$AST-Bench: Verifiable, Valid, and Versatile Safety Evaluation for Coding Agents

Oct 2026 · 0 citations · 25 references
Computer Science

TL;DR

Across model-harness combinations, agents with comparable competence exhibit markedly different safety outcomes, showing that task capability alone does not guarantee safe execution, and highlighting the need to move beyond risk recognition toward enforcing safety throughout execution.

Abstract

Large language models increasingly act through code, modifying repositories, executing programs, and interacting with services in agent form. Their actions introduce risks that cannot be assessed from generated responses alone. Safety evaluation must verify actual consequences, distinguish harm avoidance from task incapability, and cover realistic workflows. We introduce V$^3$AST-Bench, a benchmark for Verifiable results, Valid interpretation, and Versatile risk coverage of Agent Safety. It comprises 212 risk tasks spanning 18 categories. Dedicated runtime environments verify harm and task outcomes through executable checks over states, records, and artifact behavior, while paired capability twin tasks verify workflow competence. Our experiments span 51 combinations of 11 models and five harnesses, spanning 86,496 trials and over 9 billion tokens of model interaction. Results show that V$^3$AST-Bench remains challenging even for frontier coding agents, as Claude Opus 4.8 with Claude Code still causes verified harm on 20.9% of tasks. Across model-harness combinations, agents with comparable competence exhibit markedly different safety outcomes, showing that task capability alone does not guarantee safe execution. More importantly, even explicit recognition of risks does not reliably prevent harmful actions, revealing a critical gap between risk awareness and safe execution. These findings highlight the need to move beyond risk recognition toward enforcing safety throughout execution.

View source

Similar papers

Preprint Oct 2026

Executable but Unfaithful: Mock Object, Stub, and Assertion Generation by LLMs and Coding Agents

Mocking is a foundational practice in unit testing, enabling developers to isolate the code under test by replacing real dependencies with simulated alternatives. Automating the generation of mock objects, stubs, and mock assertions has been a long-standing research goal. However, capture-and-replay approaches rely on...

Hao Ning, Jia-Hui Dong, Zhen Yang et al. · 0 citations
#natural language process... Preprint Oct 2026

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?

As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenari...

Zhiqing Zhong, Ming-Yu Chen, Ao-Yang Fang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation

When generating verifiable code, natural language requirements are mapped to machine checked code using LLMs and agentic workflows. A crucial component of this pipeline is specification generation (SpecGen), which produces a formal contract against which an agent can prove implementation correctness. Proof generation c...

Srijith Nair, Aditya Vempaty, Jia Liu et al. · 0 citations
Review Open access Oct 2026

Runtime assurance of language-model-based systems: from safety monitors to certification-ready evidence

Language models are entering systems whose failures matter: vehicles, industrial toolchains, and autonomous agents acting on real infrastructure. At the same time, emerging regulation and safety standards demand not merely safe behavior but evidence of it that a third party can re-check. This survey traces a single arc...

Kamal Essam · 0 citations
Open access Oct 2026

CAST: A Compiler-Based Framework for Systematically Testing LLM Compositional Safety

This paper proposes CAST, a systematic testing framework designed to evaluate the compositional safety of LLMs in the domain of malicious code, and demonstrates that CAST systematically exposes severe safety violations in strongly aligned models that resist conventional red-teaming.

Lu Yan, Zhuo Zhang, Xiang-Zhe Xu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.