EPTQ: Fast and Accurate 2-bit Post-Training Quantization via Factored E8 Lattice
Abstract
Two-bit Post-Training Quantization (PTQ) offers an attractive path to deploying Large Language Models under strict memory budgets, yet scalar quantization suffers from catastrophic performance collapse at this extreme bit-width. While Vector Quantization (VQ) methods resolve the accuracy issue, existing approaches suffer from practical inefficiencies: complex optimization pipelines result in hours-long quantization, and some methods exhibit severely limited inference throughput. We propose EPTQ, a 2-bit PTQ framework available in two complementary variants (E8 and FE8), achieving high accuracy and practical efficiency through three contributions: (1) Factored E8 (FE8) lattice quantization, which decomposes the E8 codebook into a compact 4, KB structure that fits in the GPU L1 cache, enabling fast codebook lookup at inference time; (2) Lightweight scale normalization via row/column scaling vectors (g, h), which aligns weight distributions to maximize E8 quantization efficiency, requiring only element-wise per-token scaling at inference; and (3) Adaptive critical weight preservation via knee-point threshold detection, which more accurately identifies Hessian-sensitive weights than approaches relying on fixed global criteria, consistently improving perplexity across models, with zero bit-overhead by encoding the preservation mask into the existing codebook index space. Evaluated across multiple LLMs including Llama-2 and Llama-3 families, EPTQ (E8) achieves accuracy competitive with or surpassing QTIP on Llama-2 models, while EPTQ (FE8) surpasses competing VQ methods in decode throughput on most models, achieving up to 38% faster inference than FP16, with both variants reducing quantization time by over 80% compared to competing approaches.