ACache is presented, a cross-request cache reuse mechanism for shared spans, or affixes, at any position: prefix, infix, or suffix, and co-designs it with a modern inference engine, whose attention reads each request's recomputed Anchor KV states alongside one affix cache shared across concurrent requests.
Kai-Huan Liang, An Zhong, Xin Tan et al.· 0 citations
This work compositions existing eviction methods across Transformer layers and systematically exploring the resulting layer-wise design space establish layer-wise method composition as an exploitable, placement-sensitive design dimension for KV cache compression.
Chao Fei, Kai-Huan Liang, Han-Zhi Hu et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.