SCLAT: An LLM-Interpretable User Story Quality Evaluation Framework

Abstract

User stories serve as the core requirement carriers in agile development, and their quality directly determines the development team's understanding of requirements, as well as software delivery efficiency and quality. With the expansion of software specifications, user story quality evaluation is constrained by issues such as subjectivity and low efficiency in manual assessment, as well as lack of comprehensiveness and insufficient accuracy in the evaluation dimensions of automated methods, such as Natural Language Processing(NLP), Machine Learning(ML), and Large Language Models (LLM). This paper proposes an LLM-interpretable user story quality evaluation framework. Constructed by integrating the 3C principle, IN-VEST criteria, and IEEE 830 standards, the framework divides 40 core quality attributes into four dimensions: Structural Specification, Content Specification, Logical Soundness, and Actionability & Traceability (collectively SCLAT framework). To complement the framework, an LLM-interpretable application paradigm is proposed (known as K-CoT). This paradigm provides detailed, LLM-tailored designs for core attributes, introduces a negative example guidance mechanism, and a Chain-of-Thought (CoT) prompting strategy. Experimental validation on the NFDI4Cat dataset shows that the SCLAT framework enables LLMs to achieve an average F1-score of 89.9%, representing a significant improvement over traditional frameworks. The experimental results indicate that the interpretability design for LLM can comprehensively improve the dimensions, efficiency, and quality of user story defect detection.