MiniCache is a reusable program caching framework that transforms Program-of-Thought programs into parameterized cache objects, enabling reusable computation across structurally similar requests, and consistently improves the trade-off between inference latency, cache reuse, and task quality.
Results show that separating reusable schema encoding from selective resource access substantially reduces agentic inference costs with limited effectiveness loss.
Yichu Fang, Si-Tong Wei, Hao-Zhe Hu et al.· 3 citations
Large language model (LLM) agents repeatedly load reusable content, such as skills, documents, and memory entries, into the current context. Re-encoding this content for every request wastes computation. Position-independent caching (PIC) alleviates this by encoding each artifact independently and reusing its key-value...
Xing-Hao Chen, Jun-Nan Dong, Cai Ke et al.· 0 citations
Complex reasoning queries can be decomposed into directed acyclic task graphs and distributed across heterogeneous LLMs, reducing latency through parallelism and enabling smaller models to solve complex tasks. In practice, however, the suitability of an LLM for a given subtask may be a priori unknown, and execution alo...
Hao-Ran Zhang, Dongjun Kim, Seohyeon Cha et al.· 0 citations
This work studies KV cache reuse for RAG under a strictly plug-and-play regime, where the base language model is left unchanged and no retraining, prompt redefinition, or selective recomputation is performed, and proposes a fine-tuning-free approach that reuses KV states across queries through an anchor-block and posit...
Ye Yue, Zhe-Wei Wang, Fabio Agosto et al.· 0 citations
Modern AI backend services have evolved from single-turn prompt completion into complex workflows that interleave retrieval, task planning, external tool invocation, and multi-step generation. Existing caching mechanisms—KV-cache reuse, RAG document caching, and model cold-start reduction—optimise individual pipeline c...
Feng Wang, Mu-Zi Li, Feng-Jiao Sheng et al.· 2026 6th International Confe...· 0 citations
This work presents vla.simd, a CPU inference engine that combines shared SIMD micro-kernels, reusable computation, and target-specific optimization, and introduces IMPACT, an ACT-based policy with cached text representations and language-modulated visual features.
Khanh Duy Nguyen, Hoang M. Truong, A. T. Le· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
