SteerQuant, a 4-bit quantization framework for WAMs that steers errors toward computations with less influence on final actions, is introduced and Rudder, a 4-bit inference engine for WAMs that fuses scaling and output compensation into low-bit kernels is developed.
Yun-Han Wang, Hao-Dong Wang, Zhi-Ming Liu et al.· arXiv.org· 0 citations
Pilot, an efficient engine that reduces sparse inference overhead through lightweight scoring and cross-step reuse of token selections, is introduced, an efficient engine that reduces sparse inference overhead through lightweight scoring and cross-step reuse of token selections.
Xin-Ling Xie, Hao-Dong Wang, Jia-Zhi Mi et al.· arXiv.org· 0 citations
HATCH (Hint-Annealed Self-Teaching), an online single-policy framework that learns from both generating and using its own hints to improve reasoning without assistance, is proposed and gradient projection is used to remove the opposing component of hint-generation updates.
Zi-Le Wang, Zi-Jian Li, Hao-Dong Wang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.