Yi Kang

FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference

Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights are transferred across memory hierarchies during autoregressive decoding. In this setting, reducing the number of executed layers can lower per-token latency while als...