Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations
Analysis shows that Lngram v2 substantially reduces both total and activated memory parameters while maintaining or improving language modeling performance, and shows that its discrete IDs preserve substantial semantic structure of continuous hidden states, enabling semantic recovery from IDs alone and stable ID--seman...
Yun Zheng, Bin Wen, Xiao-Jie Wang et al.
· 0 citations