Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of e...
Jelena Markovic-Voronov, P. Prastakos, Kayhan Behdin et al.· 0 citations
Language-model agents revise their working context as information becomes obsolete, but deleting text does not erase its influence on cached computation. Later key-value (KV) states remain conditioned on the deleted content, while recomputing the entire retained suffix can be costly. We introduce budgeted KV-cache repa...
Han Yu, Wen-Hui Zhu, Jincheng Cao et al.· 0 citations
PR-SDBPG, a penalty-regularized variant that eliminates the rare-visit assumption, and VR-PR-SDBPG, which improves the resulting sample complexities entirely through variance reduction, are developed, believed to be the first explicit stochastic nonconvex-nonconvex simple bilevel optimization guarantees.
Mohammad Mahdi Ahmadi, Jincheng Cao, Aryan Mokhtari et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.