Two-level Network Bandwidth Allocation for Multi-tenant AI Clouds in Kubernetes

Abstract

Compute and memory resources in cloud environments are strictly managed and isolated by the control plane; in contrast, network resources lack equivalent management and isolation mechanisms. This best-effort treatment of networking leads to significant challenges for modern AI workloads, which have diverse and bandwidth-intensive communication patterns. Without fine-grained network resource control, these workloads suffer from interference, unpredictable throughput, and suboptimal cluster utilization. To address these issues, this paper demonstrates how network bandwidth can be elevated to a first-class, schedulable, and enforceable resource within Kubernetes, the de facto standard for cloud-native orchestration. We introduce a new scheduling capability that models network interfaces as allocatable resources and regulates bandwidth sharing through the Dynamic Resource Allocation (DRA) framework, with enforcement implemented using the Hierarchical Token Bucket (HTB) mechanism. We evaluate the system using multitenant AI workloads derived from real-world communication characteristics with a simulation-based approach and validate the proposed enforcement strategy in a real cluster. Results show that the proposed two-level bandwidth allocation improves tenant performance predictability and satisfaction while maintaining packed cluster utilization.