TL;DR
ReATest is introduced, an automated approach to enhancing PaC workflows through systematic test case generation from Rego specifications, which achieves an average 35.43% reduction in test suite size and retains 64.57% of the generated test cases.
User experience (UX) evaluation is central to software quality, yet it remains difficult to integrate into software engineering workflows in a systematic, explainable, and early-stage manner. This paper presents TraceUX, a framework for operationalizing UX heuristics and design guidance into a rule-based static evaluation pipeline that combines machine-interpretable formalization, executability-aware assessment, context-sensitive scoring, and actionable reporting. The framework is instantiated using Apple Human Interface Guidelines, Swift abstract syntax trees, and mobile games, and implemented in a proof-of-concept tool named TraceHIG. Evaluation was conducted in four layers: analysis of the full rule repository, controlled synthetic validation with injected violations, baseline assessment of 12 public Swift game projects, and a case study on one project. The full repository contained 206 rules; after excluding non-iOS yet platform-specific rules, 193 rules were retained for the downstream experiments. In controlled validation, 216 injected violations yielded 99.2% precision, 61.6% recall, and an F1-score of 0.760. In baseline analysis, overall project scores ranged from 41.6 to 88.0, reflecting rule-conformance spread under the instantiated rule base rather than direct measures of UX quality. The case study demonstrated that profile-aware scoring can yield materially different UX assessments for the same codebase under different game configurations, highlighting the importance of app profiling in static UX evaluation. These findings show that a meaningful subset of UX knowledge can be operationalized into explainable, context-aware static analysis that provides structured and actionable decision support while complementing, rather than replacing, manual and empirical UX evaluation.
Cloud-native software-delivery pipelines increasingly rely on Policy-as-Code (PaC) to automate security, compliance, and governance enforcement. Although Policy-as-Code is widely adopted within Continuous Integration (CI) pipelines and Kubernetes admission-control frameworks, governance requirements are often implemented independently, potentially increasing maintenance effort and creating opportunities for policy drift. Despite the growing adoption of Policy-as-Code, comparatively little empirical evidence exists regarding the reuse of a shared policy-definition layer across complementary enforcement stages within the software-delivery lifecycle. This paper presents and empirically evaluates a reusable multi-stage Policy-as-Code enforcement model based on a shared policy-definition layer implemented using the Open Policy Agent (OPA) framework and its Rego policy language. Rather than proposing a new Policy-as-Code technology, the study investigates whether a shared policy-definition layer can support consistent policy enforcement across Continuous Integration validation and Kubernetes admission control. The model was evaluated using Conftest and OPA Gatekeeper through a structured experimental study comprising 29 Kubernetes manifests, 37 experimental scenarios, eight Kubernetes resource types, and 261 policy assertions covering representative cloud-native workload-governance requirements. Within the evaluated dataset, all intentionally introduced insecure configurations were correctly identified without observed false positives or false negatives. The shared policy-definition layer was successfully reused across both validation stages, while Kubernetes admission control mitigated all evaluated CI bypass scenarios by providing an independent deployment-time enforcement boundary. The results demonstrate that a shared policy-definition layer can support consistent policy enforcement across complementary enforcement stages while enabling policy reuse without requiring duplicate policy implementations within the evaluated environment. More broadly, the study contributes empirical evidence supporting policy reuse as a governance strategy for cloud-native software delivery and provides a reproducible foundation for future investigations involving larger datasets, broader governance-policy portfolios, alternative Policy-as-Code ecosystems, and production-scale deployments.
Deployed large language model (LLM) agents are now being used to interface with external tools, fetch information, run code, interact with user data and help with decision making at the workflow level. Therefore, their safety issues are not only related to the underlying model, but also to tool permissions, prompt design, retrieval sources, policy constraints, evidence validity, and varying operational contexts. While Goal Structuring Notation (GSN) has become a popular means of structuring safety arguments, many GSN-based safety cases are still reliant on manual inspection and can suffer from unsupported goals, incomplete evidence paths, undischarged assumptions, circular arguments, unaddressed defeaters, or insufficient coverage of hazards. This paper introduces MCSafe-GSN-HOL, a formal assurance framework for creating machine-checkable safety cases for deployed LLM agents that overcomes these disadvantages. The proposed framework translates the structured GSN arguments into Isabelle/HOL proof obligations, and formalizes the typed GSN elements, the hazard contracts of LLM-agents, the evidence-completeness conditions, the assumption-discharge rules, the defeater-resolution rules, and the incremental re-verification rules after deployment changes. The prototype was tested with 25 safety-case fragments from five representative scenarios of LLM-agents. Results demonstrate that MCSafe-GSN-HOL has 97.3% defect detection, 97.2% evidence coverage, 96.4% hazard-control coverage, and 62.7% reduction in review time compared to manual GSN review. The mean Isabelle/HOL checking time was 2.53 s/safety case, showing that theorem-prover supported assurance can still be practical for medium scale safety-case checking. The results overall support the conclusion that formally checked GSN safety cases can help with the traceability, auditability, consistency and maintainability of assurance processes for deployed LLM-agent systems.
Customer-service LLM agents must follow organizational policy when acting on a user's behalf. Compliance failures arise from either forbidden actions, such as granting an ineligible change, or omitted procedural requirements, such as identification or confirmation. Runtime safeguards can intervene on risky actions, but action-local checks do not guide an agent through a multi-step procedure. Workflow-following systems support prescribed process execution, but primarily target workflow completion rather than safeguarding agent behavior. PolicyGuide instead compiles each domain policy into a workflow graph and invokes a proactive verifier at user-turn boundaries. From persisted graph state, the verifier reconciles open requests and returns step-specific remediation along a policy-compliant path. Across the $\tau^2$-bench airline, retail, and telecom domains with a GPT-5.4 agent and verifier, PolicyGuide raises mean $\mathrm{Pass}^4$ from $0.42$ to $0.62$, with the largest gain on telecom ($0.19$ to $0.61$), the most workflow-structured domain. The same workflows transfer to Claude Sonnet 4.6 and Gemini 2.5 Pro agents. Complementary evaluations find the lowest observed attack-success rate under adversarial users and the strongest procedural compliance in an author-designed workflow-level validation.