Evaluating Template- and Model-Guided LLMs for Requirements Specification

Abstract

Requirements specification is a pivotal activity in software development. The use of structured templates for specifying requirements can improve their clarity, consistency, and analyzability, which is particularly important in safety-critical domains such as avionics to support verification and certification activities. However, template-based requirements specification is typically a manual, resource-intensive process that requires domain expertise. Large Language Models (LLMs) provide promising capabilities for automating this task, especially when augmented with domain knowledge, yet their effectiveness in template-based requirement specification remains unexplored. In this paper, we present an empirical evaluation of LLMs for generating template-based functional requirements from raw textual specifications, and we investigate the impact of enriching LLM inputs with explicit domain models compared to unstructured textual context. We conducted 138 experiments evaluating three LLMs with multiple prompting strategies on two avionics datasets containing 146 requirements. The results indicate that simple few-shot prompting is the most effective strategy for this task. Although domain models improve execution efficiency, the gains over unstructured text are not substantial enough to justify the overhead of creating and maintaining those models.