Chih-Jen Lin

TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference

Activation sparsity speeds up large language model (LLM) inference by setting unimportant activations to zero so that the corresponding computations can be skipped. Existing training-free methods, however, make different trade-offs: threshold-based methods such as TEAL adapt the sparsity level to each token but do not...