Parameter-efficient linearization turns a pretrained autoregressive Transformer into an efficient model by hybridizing sliding-window attention (SWA) with a constant-size recurrent memory. But does every layer's recurrent state contribute to long-context retrieval? Investigating an SWA and Test-Time Training (TTT) hybr...
Vimal William, Yuan-Zhe Li, Danella Zhao et al.· 0 citations
Emerging long-horizon agentic tasks require repeated model calls, worsening the inference cost of already-costly language models. While narrow agentic tasks suggest potential for aggressive model pruning without performance drop, empirical results show existing methods proposed for question answering tasks severely deg...
Yuan-Zhe Li, Peng-Xin Wang, Yuxin Ren et al.· 0 citations
Stable Perturbation-Robust Policy Optimization (SPrPO) is introduced, which applies adaptive and sensitivity-aware perturbations during RL training and shows improved perturbation robustness while maintaining stable policy optimization.
Peng-Xin Wang, Yuan-Zhe Li, Yuxin Ren et al.· 0 citations
A quantization-robust unlearning framework that makes forgetting robust to quantization while maintaining overall model utility is proposed, and a curvature-based criteria that pinpoints sensitive weights in the unlearned model that leads to both non-robust forgetting and reduced utility is revealed.
Jia-Lu Wang, Jia-Ning Deng, Shu-Qing Luo et al.· 1 citation
Temporal-Adaptive Bit Sparsification Quantization (TASQ) achieves quality comparable to static quantization with less computation, and together with the Temporal-Precision Engine, it reduces execution cycles by 25 to 50 percent over static quantization.
Seokho Han, Dong-Wei Wang, Jinhee Kim et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.