Skip to content
Open access

STLC Manager: Benchmarking LLM-Driven Model-to-Test Transformation and Test Case Optimization

Oct 2026 · Applied Sciences · 0 citations · 40 references

TL;DR

STLC Manager is presented, an LLM-supported framework that transforms requirements, UML/XML models, and source code into structured test scenarios and test cases and supports Serial, Bulk, and Parallel strategies for semantic redundancy reduction and shows greater configuration sensitivity for Parallel under the evaluated protocol.

Abstract

Large Language Models (LLMs) can generate software tests from heterogeneous artifacts, but semantic redundancy and stochastic output variability complicate reliable pre-execution refinement. This paper presents STLC Manager, an LLM-supported framework that transforms requirements, UML/XML models, and source code into structured test scenarios and test cases and supports Serial, Bulk, and Parallel strategies for semantic redundancy reduction. Validation combines archived cross-model evidence with a controlled local study of 60 runs over two fixed 64-case suites, three installed models, partitioned local Bulk and Controlled Local Parallel, and five repetitions per configuration. The evaluation also reconstructs a three-rater human consensus and includes a deterministic TF–IDF reference. In the controlled study, mean Test Redundancy Ratio ranged from 0.0063 to 0.5000 for partitioned Bulk and from 0.1250 to 0.6375 for Parallel. Item-level modal agreement ranged from 98.44% to 100% for Bulk and from 75.31% to 97.19% for Parallel, while mean artifact-scoped Scenario Coverage Preservation ranged from 0.900 to 1.000. The results show greater configuration sensitivity for Parallel under the evaluated protocol. However, the workloads differ and do not support a causal ranking of the strategies. The implementation is publicly available. The conclusions are limited to pre-execution semantic refinement and do not establish executability, coverage, mutation adequacy, or fault detection.

Read PDF

Similar papers

Open access Nov 2026

WHEN CORRECT CODE IS NOT ENOUGH: ASSESSING THE PRODUCTION READINESS OF LLMGENERATED SOFTWARE

Large language models (LLMs) increasingly generate executable software, but functional correctness alone does not establish production readiness. This study evaluates GPT-5.6 Sol, Terra, and Luna across 15 algorithmic and production-oriented tasks using five generations per model-task pair, yielding 225 generated progr...

Ramesh Venkatachalapathy · 0 citations
Book Open access Oct 2026

Beyond Benchmarks: A Case Study of LLM-Generated Verus Specification Failures on Asterinas Vostd

Large language models (LLMs) have recently shown promise in automating formal specification generation, but their effectiveness has mainly been demonstrated on self-contained benchmark tasks. This paper examines the changes that occur when LLM-generated specifications are applied to an industrial verification project....

Du-Gang Liu, Ding Wang, Chao Jiang et al. · 1 citation
Sep 2026

Evaluating the effectiveness of class-level LLM-generated test suites in Python

Reliable assessment of LLM-generated tests should treat executability as a gate and combine coverage with mutation testing and structural quality indicators, and in practice, model selection should precede prompt tuning.

Bilal I. Al-Ahmad, M. Harshvardhan, Khaled El-Fakih et al. · 0 citations
Sep 2026

GRADE-RTL: Evaluating LLM-Generated RTL Beyond Compilation

Large language models (LLMs) can generate register-transfer-level (RTL) code from natural-language specifications, but compilation alone does not establish structural completeness, functional correctness, or implementation efficiency. This paper presents a framework for evaluating LLM-generated RTL beyond compilation,...

Hepziba Susan, R. ShivaranjaniG, Malik Imran et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.