FaultSense: Fault Localization in Large-Scale Mixture-of-Experts Model Serving Infrastructure

Abstract

While Mixture-of-Experts (MoE) models scale LLM serving across hundreds of GPUs, their reliance on all-to-all communication makes them susceptible to gray failures (e.g., GPU stragglers) which inflate serving latency without explicit errors, complicating fault localization. We present FaultSense, an application-layer approach that localizes faulty GPUs and communication paths in MoE model serving replicas without host instrumentation. FaultSense introduces a lightweight MoE probe model prototype alongside a two-phase hierarchical fault localization algorithm. Modeling the cluster as a GPU communication graph, it performs: (1) a global coverage that tests all GPUs and communication paths with a small set of probes to identify suspicious components, and then (2) a drill-down into suspicious components to localize faulty GPUs or communication paths. Our design reduces the number of diagnostic tests compared to individually testing all components (up to 20×) while maintaining low memory overheads.