Valid but Not Always Runnable: An Open, Reproducible Benchmark of Large Language Models Drafting Gherkin Scenarios
TL;DR
This work benchmarks eight LLMs generating Gherkin from three requirement corpora, scoring validity, runner acceptance, judged coverage and quality, similarity to gold standard, stability, and cost, and releases the corpora, gold standard, and prototype.
Abstract
Behaviour-Driven Development (BDD) encodes acceptance criteria in Gherkin, but hand-authoring is laborious, and it is unclear which large language model (LLM) drafts it best. We benchmark eight LLMs generating Gherkin from three requirement corpora (requirement lists, user stories, RFP excerpts) over 2960 generations, scoring validity, runner acceptance, judged coverage and quality, similarity to gold standard, stability, and cost. Validity is near the ceiling, yet only 78% of outputs load in the Cucumber runner: a fifth emits several Feature blocks per file. Two student annotators (a small, non-expert panel) calibrate the judge on 72 blinded generations. Human score levels are matched (error 0.33 versus 0.35 between the humans) but outputs are ordered far less reliably (ICC 0.47 versus 0.64; coverage 0.21 versus 0.76): magnitudes hold, but fine rankings do not. Against that gold standard, models span 66–107% of the human–human ceiling, ordering differently again. Pareto analysis leaves three of eight models non-dominated: cost varies 157×, judged quality 0.36 points. Per-model prompt tuning yields no cross-validated gain; a restrictive token budget truncates verbose models. Two newer models displace the low-cost front: tier-level findings transfer, and model names are dated quickly. We release the corpora, gold standard, and prototype. Model choice should weigh cost and runner acceptance over judged quality.
