Reinforcement learning is crucial for improving large language models'reasoning and generalization. It relies on massive rollouts whose lengths become increasingly long-tailed as context windows grow. In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL...
Yong-Qiang Yao, Jing-Ru Tan, Kai-Huan Liang et al.· 0 citations
ACache is presented, a cross-request cache reuse mechanism for shared spans, or affixes, at any position: prefix, infix, or suffix, and co-designs it with a modern inference engine, whose attention reads each request's recomputed Anchor KV states alongside one affix cache shared across concurrent requests.
Kai-Huan Liang, An Zhong, Xin Tan et al.· 0 citations
This work compositions existing eviction methods across Transformer layers and systematically exploring the resulting layer-wise design space establish layer-wise method composition as an exploitable, placement-sensitive design dimension for KV cache compression.
Chao Fei, Kai-Huan Liang, Han-Zhi Hu et al.· 1 citation
Results show that EMAS can turn experience from new samples into reusable updates to MAS topology and prompts, and is best or tied in six of eight model--benchmark settings.
Chao Fei, Qing-Yi Si, Kai-Huan Liang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
