AIaCGateGuard: A Security-First Pipeline for Benchmarking LLM- and SLM-Generated Infrastructure-as-Code
TL;DR
AIACGATEGUARD is presented, an open-source, fully automated pipeline for evaluating the security compliance of Infrastructure-as-Code (IaC) generated by Large and Small Language Models (LLMs/SLMs), producing permodel, per-scenario security compliance reports.
Abstract
We present AIACGATEGUARD, an open-source, fully automated pipeline for evaluating the security compliance of Infrastructure-as-Code (IaC) generated by Large and Small Language Models (LLMs/SLMs). Unlike existing IaC generation benchmarks, which assess only syntactic validity or functional correctness, AIACGATEGUARD integrates dual-tool static security analysis (Checkov and Trivy) directly into a GitLab CI/CD pipeline, producing permodel, per-scenario security compliance reports. The tool supports plug-and-play integration of new LLMs via REST API and new SLMs via local inference (llama-server), configurable prompt strategies and security specificity levels, optional infrastructure validation and deployment stages (plan-real, apply, destroy), and a Model Context Protocol (MCP) server for conversational interaction with the pipeline. We demonstrate the tool on seven state-of-the-art models across 17 AWS Terraform scenarios, producing 3,570 automated evaluations and exposing security compliance gaps invisible to syntactic-only validation. AIACGATEGUARD is publicly available, including pipeline code, prompt templates, scenario definitions, and a video demonstration of installation and usage.