It is found that functional correctness alone is insufficient to assess the operational reliability of LLM-generated software before deployment in continuously running environments, and aging trends can also emerge in manually developed implementations.
Abstract
Large Language Models (LLMs) are increasingly used to generate executable software systems from natural language specifications, accelerating development and reducing manual implementation effort. Although recent studies have investigated the functional correctness, security, maintainability, and robustness of LLM-generated code, little is known about the long-term reliability of such systems under sustained execution. In this paper, we experimentally investigate software aging symptoms in LLM-generated service-based applications across different programming languages. Using backend scenarios derived from BaxBench, we generated applications targeting JavaScript, Python, and Rust through LLM-based generation platforms, validated them with BaxBench-derived tests, and subjected them to 48-hour workload executions. We monitored memory usage, response time, and throughput and analyzed them using the Mann--Kendall test and Sen's slope estimator. We further complemented the runtime evaluation with static analysis of the generated source code and an exploratory comparison with human-written implementations of related backend scenarios. The results show that memory usage is the most consistent indicator of potential software aging, with statistically significant upward trends in most application-language combinations, while response time and throughput exhibit more heterogeneous behavior. Static analysis identified plausible code-level aging mechanisms, and the comparison with human-written systems showed that aging trends can also emerge in manually developed implementations. These findings indicate that functional correctness alone is insufficient to assess the operational reliability of LLM-generated software before deployment in continuously running environments.
Large Language Models (LLMs) are now widely used for code generation, yet even syntactically correct output may contain logical and semantic errors that remain invisible until runtime, particularly in framework-driven applications, where correctness depends on dependency injection, framework conventions, configuration, library compatibility, and database interaction. This paper presents GenTest, an open-source platform for context-aware dynamic generation of Java code using LLMs, runtime compilation and class loading, Spring bean registration, JPA/PostgreSQL execution, and declarative YAML-based test validation. GenTest supports multiple LLM providers through a provider-agnostic interface and evaluates generated components within a live Spring Boot application context. Across 209 test cases and 933 assertions, GenTest achieves a 63.9% assertion pass rate, and 94.2% of failures occur after successful compilation, confirming that execution-centered evaluation reveals failure modes invisible to static analysis.
Muhammed Suphi Şeyhkuruş, Tolga Ovatman· Annual International Compute...· 0 citations
This paper introduces and empirically study the phenomenon of error propagation, where faults in generated code are systematically replicated in associated test artifacts, and examines whether LLM-generated code biases the generation of subsequent tests.
Michael Konstantinou, Florian Tambon, Mike Papadakis· 1 citation
A reproducible benchmarking platform that evaluates open-source LLMs on Verilog RTL generation across 50 curated tasks consisting of combinational, sequential, finite state machine (FSM), and mixed designs, enabling reproducible evaluation of generative AI for hardware design workflows.
This paper investigates how programming language standards influence memory safety by analyzing three distinct approaches: the manual memory management of C++, the ownership-based compile-time model of Rust, and the automatic garbage collection of Python.
Gordon Bednarz, Ludvig Kåhlin, Oliver Andersson· 0 citations
This tutorial introduces a reusable, end-to-end evaluation pipeline grounded in empirical software engineering practices, focusing on post-generation validation rather than prompt design, allowing for validating AI-generated code in modern development workflows.
Large language models (LLMs) are increasingly used to port scientific codes across heterogeneous high-performance computing (HPC) programming models, such as translating CUDA to OpenMP, OpenACC, Kokkos or SYCL. However, current evaluations use compilation success, token-level similarity, or developer-written tests from static benchmarks, which cannot reliably ensure behavioral correctness. We present Kaizen, a metamorphic fuzzing and differential testing framework for evaluating the correctness of LLM-translated HPC code. Kaizen uses metamorphic fuzzing via source-code mutation to generate semantically equivalent programs, grammar-based input fuzzing to explore behavioral diversity, and differential testing to expose semantic divergences between original and translated applications that compile and pass developer-written tests yet produce incorrect scientific results. We evaluate Kaizen on CUDA-to-OpenMP translation of 16 scientific applications from seven domains using three fine-tuned LLMs at kernel-level and full-program granularity. Our evaluation reveals that (1) compilation success is a poor proxy for correctness; (2) LLM-translated programs exhibit systematic compile-time error patterns, with nine categories for kernel-level translation and 27 for full-program translation; (3) semantic errors that survive compilation are often input-dependent and require differential testing to expose; and (4) full-program translation is substantially harder than kernel-level translation. These findings highlight the need for correctness-oriented evaluation of LLM-assisted HPC code translations.
Oscar Ludwig, Ninad Anklesaria, Zheming Jin et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 17, 2026
A USAF cadet and a Lincoln Laboratory researcher found AI chatbots can help nontechnical service members produce viable software applications for their unique problems.
