2027· Proceedings of the 32nd ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1· 0 citations· 18 references
Abstract
Verified code generation, in which an LLM produces both an implementation and a correctness proof, offers a promising path toward correct-by-construction software. Prior evaluations report that LLM capability remains unsatisfactory, but they test models in a bare-model setting that no longer reflects how LLMs are deployed in practice, e.g., as coding agents. We re-evaluate existing benchmarks and find that coding agents solve nearly all tasks derived from well-studied sources such as programming contests. This raises the question of whether such success extends to real-world systems tasks. We present Vero, a benchmark of 103 verified code generation tasks drawn from three security- and correctness-critical projects: the Linux kernel eBPF verifier, the LLVM compiler, and the seL4 microkernel. Each task is real-world, encodes a new specification, and is valuable to solve in that a correct solution directly benefits the upstream project. We evaluate three coding agents; the strongest solves only 3 of the 103 tasks, establishing that real-world verified code generation remains an open problem. Notably, one agent-discovered solution to a Vero task has been merged into the Linux kernel, replacing a 20-line algorithm proposed by domain experts with a 4-line provably correct one. The unsolved tasks provide concrete targets for future work.
As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenari...
Zhiqing Zhong, Ming-Yu Chen, Ao-Yang Fang et al.· 0 citations
Can coding agents restore software that no longer runs while preserving its underlying methods, and reconstruct industrial software engines from open specifications? Here we introduce ReviveBench, a benchmark with two task families evaluated by hidden verifiers calibrated against native execution environments, establis...
Tian-Yu Liu, Ding-Yuan Dai, Yu-Fan Du et al.· arXiv.org· 0 citations
E2E-SWE is introduced, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end and finds substantial variation in end-to-end repository generation ability, providing strong model differentiation while leaving considerable headroom for future progress.
Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable guarantees. Recently, researchers have proposed several benchmarks to evaluate the capabilities of LLMs in generating formally verifiable code, where LLMs need to formulat...
Jia-Ru Qian, Yihong Dong, Yong-Ming Li et al.· arXiv.org· 0 citations
The findings establish environment specification as a distinct, measurable axis of code generation quality that current benchmarks do not capture, and motivate training objectives and evaluation protocols that jointly optimize for functional correctness and environmental portability.
EasyProver is presented, a Lean proving system built from recursive agents with isolated contexts, asynchronous search, parallel tactic probing, and an elastic elaboration service, and Aegis is introduced, which converts competitive programming problems into much harder verification tasks over real implementations.
Debargha Ganguly, Divya Raghunathan, Dejan Jovanovic et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.