Huai-Zhi Zhang

MCF: A Memory- and Communication-Aware Co-Designed Framework for Long-Context LLM Inference

This paper presents Memory- and Communication-Aware Framework for Long-Context LLM Inference (MCF) that optimizes long-context inference through communication-aware and stage-adaptive KV-cache layout transitions. At the core of MCF is {Memory-Aware Replication and Scheduling} (MARS), a dynamic orchestration mechanism f...