Training large language models directly in 4-bit floating-point (FP) formats promises substantial improvements in throughput and energy efficiency. While some recipe design choices have been validated at scale, many promising approaches remain untested beyond small models and short token horizons, leaving open the ques...
A. Agrusa, A. Panferov, Elizabeth Wei et al.· 0 citations
This work proposes GA-SVD, a generation-aware low-rank compression framework for LLMs, and introduces Generation-Aware Weight Whitening (GAWW), which leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior, and develops an evolutionary global ran...
Sheng-Kun Tang, Cong Zeng, Zong-Fang Liu et al.· 0 citations
A new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients, and its stability during training allows for infrequent, cheap recalibration.
Alexandra Volkova, Matin Ansaripour, Erik Schultheis et al.· 0 citations
This work presents an end-to-end hardware-software co-design framework that compresses expert weights into hardware-native, low-precision sparse representations and accelerates their execution on SpTCs, establishing hardware-software co-design as a practical path toward scalable and efficient MoE deployment.
Kwanhee Lee, Namhoon Lee, Martin Jaggi· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.