AutoEMatic: Runtime-Adaptive Mixed-Precision LLM Inference via Automatic Exponent-Mantissa Refinement
AutoEMatic, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime, is presented, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime.
Akshat Ramachandran, Reena Elangovan, Rangharajan Venkatesan et al.
· 0 citations