SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementation satisfies its natural language specification. Reliable validation requires checks that faithfully capture the specification and scenari...