Jul 2026· Annual International Computer Software and Applications Conference· pp. 1725-1728· 0 citations· 10 references
Abstract
Large Language Models (LLMs) are now widely used for code generation, yet even syntactically correct output may contain logical and semantic errors that remain invisible until runtime, particularly in framework-driven applications, where correctness depends on dependency injection, framework conventions, configuration, library compatibility, and database interaction. This paper presents GenTest, an open-source platform for context-aware dynamic generation of Java code using LLMs, runtime compilation and class loading, Spring bean registration, JPA/PostgreSQL execution, and declarative YAML-based test validation. GenTest supports multiple LLM providers through a provider-agnostic interface and evaluates generated components within a live Spring Boot application context. Across 209 test cases and 933 assertions, GenTest achieves a 63.9% assertion pass rate, and 94.2% of failures occur after successful compilation, confirming that execution-centered evaluation reveals failure modes invisible to static analysis.
This study investigates whether a frontier LLM can generate Dockerfiles and Docker Compose configurations for multi-service applications using repository contents without access to developer-authored deployment artifacts and analytically derives a minimal explicit deployment specification for information that cannot be reliably inferred from repository artifacts.
Oleg Grynets, Kyrylo Fursov, V. Lyashkevych et al.· 0 citations
An empirical benchmark evaluating whether LLMs and SLMs can generate security-compliant AWS Terraform configurations suggests that prompt design is a critical factor, highlighting the need for a proper pipeline for developing and validating LLM-assisted secure IaC generation.
Francis Luis Santos Vargas, R. Mansilha, Diego Kreutz· Anais do I Simpósio de Infra...· 0 citations
Large Language Models (LLMs) have advanced code completion, but their ability to generate API usages compatible with evolving third-party libraries (TPLs) remains uncertain. As TPL APIs frequently change, LLMs risk producing code incompatible with installed library versions, causing build failures or incorrect behaviors. We define such issues as Incompatible Third-party Library API Usage (ITAU) and conduct a systematic study to evaluate how state-of-the-art LLMs handle this challenge. To this end, we propose an automated framework that builds a versioned TPL API Knowledge Base and a large-scale benchmark with 10,867 realistic code completion tasks. Through comprehensive evaluation of six state-of-the-art LLMs, we find that even top-performing models frequently generate incompatible completions. We further propose two lightweight solutions, Real-time Detection and Lightweight Repair, to mitigate ITAUs. This framework and benchmark provide a foundation for more compatibility-aware code generation in evolving software ecosystems.
Li Lin, Yaorui Fei, Yunfeng Shen et al.· ACM Transactions on Software...· 0 citations
This paper formalizes structural coherence as consistency invariants over graph representations of repository artifacts, including import, call, dependency, configuration, schema, resource, control-flow, and routing graphs, and introduces an eight-category failure taxonomy distinguishing defects specific to LLM generation from those merely amplified by it.