Skip to content
Open access

Unfulfilled Promises: LLM-Based Detection of OS Compatibility Issues in Infrastructure as Code

Jun 2026 · Proceedings of the ACM on Software Engineering · Vol 3, pp. 1536 - 1559 · 1 citation · 50 references

Abstract

Modern infrastructures rely on Infrastructure as Code (IaC) systems to keep complex deployments consistent, reproducible, and scalable at production scale. The reliability of these infrastructures, however, depends on the correctness of their building blocks, which are reusable components (modules) that each performs a dedicated task, such as installing a package, managing an OS user, or configuring a service, and reconciling its state with the desired specification. A central promise of these components is portability: a specification written once should correctly manage the targeted resource on every OS the IaC component supports. When this property is violated, defects can propagate across entire infrastructures, causing outages, security vulnerabilities, and costly misconfigurations. In this work, we introduce crOSsible, the first automated framework for cross-OS testing of IaC modules. crOSsible leverages large language models (LLMs) to synthesize and repair integration tests from structured module documentation, and executes them across 13 versions of 8 major Linux distributions. While our techniques are generally applicable to different IaC systems, we instantiate and evaluate them on Ansible, the most widely used IaC framework for managing individual servers. Evaluation across 259 popular Ansible modules demonstrates both effectiveness and real-world impact. In just 12 hours of testing, crOSsible uncovered 36 previously unknown bugs, including 22 portability violations. In total, 27 issues have been confirmed by maintainers, with 17 already fixed. The discovered issues range from crashes to dangerous soundness defects where modules reported success despite leaving systems misconfigured. Beyond bug discovery, crOSsible improved the code coverage of Ansible modules by 12.3% on average, systematically exercising OS-specific code paths that existing tests missed.

Read PDF

Similar papers

Preprint Jul 2026

CHARGE: Leveraging CWE Hierarchies for Hardware Security SystemVerilog Assertion Generation

This paper presents CHARGE, an automated framework for generating security properties for unverified RTL modules using CWEs and large language models (LLMs). The hallmark is a reasoning process that leverages the hierarchical nature of CWE entries to improve accuracy when identifying security-critical assets in unverified RTL modules. As a result, the approach can infer expected security behaviors and generate properties from identified assets and CWE semantics, avoiding the need for trusted design specifications and reducing manual engineering effort. We evaluate the framework on the Hack@DAC18, 19, and 21 open source SoC designs using OpenAI's GPT-4.1. CHARGE detects 27 of 42 known bugs in these designs. For Hack@DAC21 OpenPiton SoC, 89% of the generated SVAs run in Cadence JasperGold FPV, and 92.2% are non-vacuous. We compare to an open-source, manually written set of properties for these designs and find that CHARGE correctly writes properties for three bugs in which the manually written properties were incorrect. In addition, CHARGE-generated properties identify a new bug in the Hack@DAC21 OpenPiton SoC that was not previously identified.

Xiao Tan, C. Sturton · 0 citations
Preprint Jul 2026

The Patchwork Problem in LLM-Generated Code

This paper formalizes structural coherence as consistency invariants over graph representations of repository artifacts, including import, call, dependency, configuration, schema, resource, control-flow, and routing graphs, and introduces an eight-category failure taxonomy distinguishing defects specific to LLM generation from those merely amplified by it.

Viraaji Mothukuri, R. Parizi · 0 citations
Preprint Aug 2026

Improving the Security of Containerized Workloads using Transparency and Traceability Services

Containerized workloads are commonly built via CI/CD pipelines, stored in registries, and executed across heterogeneous infrastructures, including cloud and edge environments. A single compromised build step or credential can turn routine automation into large-scale distribution of malicious artifacts, motivating integrity, transparency, and enforceable deployment-time checks. In this paper, we present an architecture for verifiable container image distribution that addresses key-management challenges and enables policy-enforced admission-time verification. A transparency service generates one-time signing keys bound to authenticated identities, records signing events in an append-only transparency registry, and returns cryptographically verifiable proofs of inclusion. These proofs and identity attributes are attached to image metadata and evaluated by policy-as-code at admission time, so only compliant artifacts are deployed. We implement a proof-of-concept integrated with GitHub Actions and GitLab Runners and evaluate how the resulting pipeline mitigates common supply-chain attacks under a realistic threat model.

N. Fotiou, Lefteris Georgiadis, Ignacio Lacalle et al. · 0 citations
Conference Jul 2026

SecReviewAgent: Context-Aware Security Review of Infrastructure-as-Code Using Persistent Architecture Memory

Security review of Infrastructure-as-Code (IaC) in enterprise cloud platforms requires understanding not only configuration diffs, but also how those changes interact with existing architecture and risk controls. In financial-engineering environments, IaC changes can directly affect data perimeter boundaries, model-serving paths, entitlement controls, and operational resilience. Current tools often analyze pull requests in isolation, missing security implications that emerge only from cross-resource context. This paper presents SecReviewAgent, an LLM-powered IaC security review system that maintains persistent architecture memory across pull request reviews. On first encounter with a repository, SecReviewAgent scans IaC files to build an architecture model, persists that model to object storage, and reuses it in subsequent reviews to interpret changes in context. We evaluate the system on 847 pull requests from 23 repositories spanning financial services, healthcare, e-commerce, and developer tooling. Results show a $2.4 \times$ improvement in context-dependent issue detection recall over a no-context LLM baseline, precision of 0.89, F1 of 0.83, and a 73% latency reduction on warm reviews by avoiding full repository rescans. A controlled user study with 42 practitioners indicates reduced median review time and improved finding accuracy. The paper contributes a persistent-memory design for LLM-based code review, algorithms for repository context construction and incremental update, and an empirical evaluation of context-aware IaC security review in realistic settings.

Naga Krishna Reddy Muppidi, Veera Ravindra Divi, Sneha Gullapalli et al. · 0 citations
Open access Jul 2026

Can Language Models Generate Secure Terraform Code? A Security-Focused Benchmark Using Static Analysis

An empirical benchmark evaluating whether LLMs and SLMs can generate security-compliant AWS Terraform configurations suggests that prompt design is a critical factor, highlighting the need for a proper pipeline for developing and validating LLM-assisted secure IaC generation.

Francis Luis Santos Vargas, R. Mansilha, Diego Kreutz · 0 citations
Conference Jul 2026

Domain-Tuned LLMs for Detecting Misconfigurations in Multi-Cloud Architectures

Distributed cloud infrastructure is becoming an essential tool that the modern enterprises are utilizing to implement scalable service and application on the basis of the various service providers. Although multi-clouds lead to increased flexibility and availability of resources, they also pose a major challenge of configuration management. Systems variability in configuration representations, access controls and service dependencies between hosts also enhance the risk of infrastructure misconfiguration, resulting in security weaknesses, unauthorized access, and inconsistent operation. Turning on configuration inconsistencies of large-scale distributed infrastructures is a complicated undertaking to administrators, especially when configuration components interact with non-homogeneous environments. In an attempt to deal with these difficulties, the current paper introduces a Contextual Policy Reasoning Framework (CPRF) that is meant to be used in analyzing infrastructure setups and identifying any inconsistencies that may exist in a multi-cloud system. The suggested CPRF approach understands the configuration structures, analyses the links of infrastructure components as well as measures the dependability of configuration, the exposure of privileges, and the impact of dependencies to detect potential configuration risks. The framework also incorporates analytical modeling in order to measure configuration integrity and approximate a level of risk in infrastructure in distributed environments. CPRF allows better detecting the frameworks of complex configuration inconsistencies within heterogeneous cloud environments by thoroughly analyzing configuration links and operation dependencies. The suggested policy facilitates better configuration management and helps administrators to have uninterrupted and safe infrastructure deployments in the contemporary cloud systems. The suggested approach attains an overall configuration inconsistency detection accuracy of 96.6%, indicating enhanced dependability in the analysis of distributed cloud infrastructure configurations.

Ratna Chaitanya Yarrapothu, Santhosh Reddy BasiReddy, Rajendra Asuri et al. · 0 citations