Spec-Harness: Measuring and Improving Behavioral Adequacy of LLM-Synthesized Formal Specifications
TL;DR
Comparisons of classical and prompt-based JML synthesis approaches are compared, and it is found that prompt optimization through verification feedback raises pass rates but reaches a clear ceiling, and Spec-Harness is introduced, a framework that measures the behavioral adequacy of a specification along four dimensions of precondition and postcondition correctness and completeness.
Abstract
Formal specifications play a central role in ensuring software reliability, yet automatically synthesizing high-quality specifications remains difficult and often requires domain expertise. Recent work has applied large language models to generate specifications in the Java Modeling Language (JML), reporting high verifier pass rates. But passing a verifier only confirms that an implementation is consistent with a specification, not that the specification is meaningful. A trivial postcondition such as ensures true satisfies any verifier while saying nothing about the code. How much behavior, then, does a verifier-accepted specification actually capture? In this work, we first compare classical and prompt-based JML synthesis approaches under a unified setup, and find that prompt optimization through verification feedback raises pass rates but reaches a clear ceiling. We then introduce Spec-Harness, a framework that measures the behavioral adequacy of a specification along four dimensions of precondition and postcondition correctness and completeness, using Hoare-triple based symbolic verification and input/output mutation. Spec-Harness reveals that many verifier-accepted specifications, including optimized ones, are behaviorally weak, over- or under-constraining inputs and outputs in ways the verifier cannot see. Finally, we show that Spec-Harness works as a feedback signal that helps coding agents synthesize specifications with higher behavioral adequacy, including general-purpose agents such as Codex CLI and Claude Code, as well as VeriAct, a JML-specialized agent we build for this study.
