Weng-Fai Wong

FairCache: Demystifying Cache-Induced Unfairness in Multi-Tenant Large Language Model Serving

Large language models (LLMs) increasingly rely on context caching to enhance serving efficiency. However, this optimization inadvertently compromises fairness in multi-tenant LLM serving systems. Existing fair schedulers, which account only for compute resources, are unable to handle the multi-dimensional resource dema...