An Empirical Study and Benchmark of Kubernetes Misconfiguration Scanners

Abstract

Kubernetes is a widely adopted container orchestration framework, yet misconfigurations remain a leading cause of cloud security incidents and a major challenge for practitioners. Automated security scanners are commonly used to detect such misconfigurations, but their effectiveness has not been systematically evaluated. As a result, it remains unclear which tools can be trusted, what misconfigurations they reliably detect, and to what extent they improve Kubernetes security. This paper presents the first systematic investigation of ten prominent Kubernetes security scanners that are actively used in practice. We begin by examining the misconfigurations that each scanner claims to detect. Although scanners provide documentation of their coverage, these descriptions are written in natural language and are often ambiguous, making it unclear what is actually detected. To address this issue, we manually analyze scanner implementations to identify their precise detection targets and validate them through dynamic testing. Our analysis reveals that scanners frequently adopt different criteria for the same documented misconfiguration, exposing significant inconsistencies caused by ambiguous specifications. Building on these results, we introduce the first comprehensive benchmark for Kubernetes misconfigurations, covering all misconfigurations targeted by the ten scanners. The benchmark includes 4,109 misconfiguration files for static and dynamic analysis and 144 shell scripts for dynamic analysis, encompassing 281 unique misconfigurations. Using this benchmark, we conducted an empirical evaluation of ten scanners. Our results show that, in static scanning, Kubescape achieves the highest recall (43.8%), and it also performs best in dynamic analysis with an recall of 62.3%. We further analyze the strengths and limitations of each scanner, identifying coverage gaps that significantly affect detection effectiveness. Our findings provide practical guidance for practitioners selecting Kubernetes security scanners and highlight key challenges that should be addressed by the software engineering community. Moreover, the proposed benchmark establishes a foundation for future research on Kubernetes security.