This work presents the first cross-task empirical evaluation of LLMs spanning five RE-related activities, as well as replication materials supporting reproducibility, and a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.
Abstract
Requirements-related information is scattered across heterogeneous artefacts such as user feedback, developer discussions, and software repositories, making the extraction of actionable requirements knowledge labour-intensive and hard to scale. Large Language Models (LLMs) can support many Requirements Engineering (RE) activities, from classification and traceability identification to specification and explanation generation, but existing evidence is fragmented across tasks, artefact types, and evaluation settings, and studies rarely offer cross-task evaluations or replication packages. We present two complementary empirical studies evaluating LLMs across five RE-related activities. The first is a controlled experiment on five lightweight open-source LLMs for feedback-driven requirements classification and specification generation. The second is an exploratory industrial case study on two frontier LLMs for traceability link identification and traceability explanation generation using real project artefacts. Classification and traceability identification were assessed with quantitative metrics, and generation tasks through human evaluation. LLM performance is strongly task-dependent, ranging from moderate to high, and no single model consistently outperformed the others, indicating that effective adoption depends on selecting models and prompting strategies per task. Our contributions are: (i) the first cross-task empirical evaluation of LLMs spanning five RE-related activities, (ii) replication materials supporting reproducibility, and (iii) a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.
Artificial intelligence (AI) is transforming requirements elicitation: machine learning, natural language processing (NLP), and large language models (LLMs) now identify software requirements automatically from the textual data that surrounds every project—user feedback, specifications, regulations, and stakeholder transcripts. This paper presents a systematic mapping study of 74 peer-reviewed primary studies on AI-based automated requirements elicitation published between 2021 and 2025, identified from five databases following PRISMA 2020 and classified by AI technique, textual source, elicitation activity, and application domain. The evidence is divided into two equally sized source families—user feedback and agile artefacts versus formal documentation—each coupled to the AI techniques that suit its signal profile. Fine-tuned transformer encoders set the performance ceiling and, task-for-task, still outperform far larger generative models, while LLMs extend elicitation to long regulatory documents, multilingual feedback, and structured outputs. The central finding concerns automation depth. AI identifies requirements with consistently high accuracy (routinely F1 0.8 and above), but automation thins at every subsequent step: 51% of approaches structure what they identify, 23% consolidate them, and only 8% engineer stakeholder validation into the loop. This leaves the steps that turn candidates into agreed requirements largely manual. Benchmark fragmentation (77% custom datasets), thin industrial validation (14%), and skewed non-functional coverage compound this gap. The resulting map gives researchers an evidence-derived agenda for deepening automation, and practitioners guidance on which techniques the evidence supports for each elicitation task and textual source.
Safaa Eltahier, S. Al-Ghuribi, Mawal A. Mohammed et al.· Information· 0 citations
Interviews with sixteen early-adopter software professionals who integrated LLM-based tools into their day-to-day work in early to mid-2023 offer actionable implications for developers, organizations, educators, and tool designers seeking to integrate LLMs responsibly into professional software practice.
Benyamin T. Tabarsi, Heidi Reichert, Sam Gilson et al.· Empirical Software Engineeri...· 21 citations· ⚡1
: The growing adoption of large language models (LLMs) in software engineering has introduced new opportunities but also risks in the software maintenance lifecycle. While LLMs can generate entire codebases from natural language prompts, such automatically generated or rapidly prototyped code often accumulates structural debt, making systematic refactoring increasingly urgent. This work investigates LLMs as metric-driven refactoring assistants rather than code generators. Six models (ChatGPT, Claude, Gemini, Grok, DeepSeek, and Qwen) were evaluated on two types of Java projects: three controlled applications with manually inflated structural metrics, and three real-world applications from public GitHub repositories. Using MetricsReloaded in IntelliJ IDEA, we measured four CK metrics: complexity (WMC), cohesion (LCOM), coupling (CBO), and inheritance depth (DIT). Results indicate that LLMs significantly reduce complexity and coupling, improving class simplicity and modularity. However, cohesion improvements remained limited, with LCOM proving especially elusive. Inheritance depth showed strong reductions in synthetic high-metric applications but minimal change in real projects. ChatGPT produced the most consistent and structurally stable refactoring outputs in real applications, though occasional cohesion deterioration occurred. These findings suggest that while LLMs are valuable assistants for structural improvement, their interventions require careful monitoring to avoid unintended trade-offs.
Tindwende Sawadogo, Fadel Touré· Proceedings of the 21st Inte...· 0 citations
This paper investigates the reliability of LLMs in evaluating UML diagrams generated through reverse engineering processes (source code) and asks: do LLM assessments align with those of human experts?
Olena Chebanyuk, Carles Sierra· Proceedings of the 21st Inte...· 0 citations
A large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective, providing empirical evidence linking framework design choices to engineering risks and highlighting the need for stronger guidance and support in agent framework development.
Yibo Zhai, Junjun Si, Yan Wang et al.· SIGSOFT FSE Companion· 0 citations
A lifecycle-aware framework that integrates quantitative software quality assessment with Large Language Model (LLM)-based code refinement is proposed and the potential of metric-driven LLM feedback for research software quality improvement is demonstrated while highlighting its inherently multi-objective nature.
Nafis Tanveer Islam, N. Soveizi, Yutong Li et al.· 0 citations