This evidence-centered structured survey synthesizes representative work available through May 31, 2026 across software engineering tasks, software security tasks, adaptation mechanisms, artifact granularity, and evaluation design and introduces an assurance framework that separates functional correctness, security, operational reliability, evidence provenance, and agent authority.
Abstract
Large Language Models (LLMs) are moving from code completion toward repository-scale agents that retrieve context, edit files, execute tools, and participate in security-sensitive workflows. The evidence for these systems, however, remains divided between software engineering evaluations centered on functional task completion and software security evaluations centered on vulnerability detection, secure generation, or exploit-oriented validation. This evidence-centered structured survey synthesizes representative work available through May 31, 2026 across software engineering tasks, software security tasks, adaptation mechanisms, artifact granularity, and evaluation design. In addition to a task taxonomy, we introduce an assurance framework that separates functional correctness, security, operational reliability, evidence provenance, and agent authority. The review shows that execution feedback and repository access can substantially improve engineering task completion, but do not by themselves establish security; conversely, static-analysis labels or vulnerability-classification scores rarely establish deployable correctness. We identify recurring validity threats--weak test oracles, duplicated and temporally leaked data, changing agent harnesses, proxy-only security checks, and under-reported budgets and human intervention--and derive a minimum reporting protocol for cross-study comparison. The resulting research agenda prioritizes jointly secure-and-functional benchmarks, repository-scale threat models, calibrated human oversight, longitudinal maintainability evidence, and reproducible agent evaluation. The central conclusion is that model capability should be judged as an assurance case supported by task-appropriate evidence, rather than by a single benchmark score.
Security review of Infrastructure-as-Code (IaC) in enterprise cloud platforms requires understanding not only configuration diffs, but also how those changes interact with existing architecture and risk controls. In financial-engineering environments, IaC changes can directly affect data perimeter boundaries, model-serving paths, entitlement controls, and operational resilience. Current tools often analyze pull requests in isolation, missing security implications that emerge only from cross-resource context. This paper presents SecReviewAgent, an LLM-powered IaC security review system that maintains persistent architecture memory across pull request reviews. On first encounter with a repository, SecReviewAgent scans IaC files to build an architecture model, persists that model to object storage, and reuses it in subsequent reviews to interpret changes in context. We evaluate the system on 847 pull requests from 23 repositories spanning financial services, healthcare, e-commerce, and developer tooling. Results show a $2.4 \times$ improvement in context-dependent issue detection recall over a no-context LLM baseline, precision of 0.89, F1 of 0.83, and a 73% latency reduction on warm reviews by avoiding full repository rescans. A controlled user study with 42 practitioners indicates reduced median review time and improved finding accuracy. The paper contributes a persistent-memory design for LLM-based code review, algorithms for repository context construction and incremental update, and an empirical evaluation of context-aware IaC security review in realistic settings.
Naga Krishna Reddy Muppidi, Veera Ravindra Divi, Sneha Gullapalli et al.· International Conference on...· 0 citations
A large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective, providing empirical evidence linking framework design choices to engineering risks and highlighting the need for stronger guidance and support in agent framework development.
Yibo Zhai, Junjun Si, Yan Wang et al.· SIGSOFT FSE Companion· 0 citations
Empirical evidence is provided that generative AI can effectively support security requirements engineering when embedded within human-centered workflows and organizational governance structures, offering practical insights for adoption in regulated software development contexts.
F. Martins, Elaine Venson· SIGSOFT FSE Companion· 0 citations
User experience (UX) evaluation is central to software quality, yet it remains difficult to integrate into software engineering workflows in a systematic, explainable, and early-stage manner. This paper presents TraceUX, a framework for operationalizing UX heuristics and design guidance into a rule-based static evaluation pipeline that combines machine-interpretable formalization, executability-aware assessment, context-sensitive scoring, and actionable reporting. The framework is instantiated using Apple Human Interface Guidelines, Swift abstract syntax trees, and mobile games, and implemented in a proof-of-concept tool named TraceHIG. Evaluation was conducted in four layers: analysis of the full rule repository, controlled synthetic validation with injected violations, baseline assessment of 12 public Swift game projects, and a case study on one project. The full repository contained 206 rules; after excluding non-iOS yet platform-specific rules, 193 rules were retained for the downstream experiments. In controlled validation, 216 injected violations yielded 99.2% precision, 61.6% recall, and an F1-score of 0.760. In baseline analysis, overall project scores ranged from 41.6 to 88.0, reflecting rule-conformance spread under the instantiated rule base rather than direct measures of UX quality. The case study demonstrated that profile-aware scoring can yield materially different UX assessments for the same codebase under different game configurations, highlighting the importance of app profiling in static UX evaluation. These findings show that a meaningful subset of UX knowledge can be operationalized into explainable, context-aware static analysis that provides structured and actionable decision support while complementing, rather than replacing, manual and empirical UX evaluation.
Large language models are increasingly used to review, clarify, rewrite, and trace software requirements. These applications create a governance problem that output-quality assessment alone cannot resolve: a fluent proposal may rely on inadmissible evidence, alter stakeholder intent, introduce unsupported specificity, or imply an organizational commitment that the model has no authority to make. Existing work on retrieval-augmented generation, controlled natural language, formal verification, human oversight, and AI governance supplies relevant controls, but it does not specify the procedural status of an individual LLM proposal relative to a controlled requirements artifact. This article develops an artifact-centered, human-in-the-loop framework in which the permitted influence of a proposal is the primary object of governance. The framework combines five governance functions—governed evidence, bounded context construction, controlled LLM analysis, pre-commit verification, and accountable human approval—with four artifact-influence states: A0 advisory observation, A1 evidence-linked candidate, A2 verified recommendation, and A3 approved and committed change. Its central theoretical claim is that output quality, evidential legitimacy, verification status, and authority to commit a change are distinct properties and should not be collapsed into a single confidence judgment. Seven falsifiable hypotheses translate the model into measurable comparisons involving source admissibility, context leakage, unsupported specificity, semantic drift, reviewer agreement, unreviewed changes, governance cost, and organizational maturity. Human review is treated as both a necessary decision boundary and a potential source of automation bias, anchoring, and fatigue. The framework is conceptual rather than empirically validated and provides a basis for controlled experiments, field studies, and longitudinal evaluation.
Chuanjin Zhu· Advances in Engineering Inno...· 0 citations
Enterprise software requires specification governance to transform probabilistic AI generation into deterministic, auditable engineering, and the SGRM framework is introduced, which defines four-component specification contracts, constrains stochastic generation via deterministic validation, and integrates generation, verification, and governance into a closed-loop architecture.