Looped language models apply one block of layers many times per token, so a serving engine pays every per-layer overhead once per loop; we measure serving for Ouro and Huginn-0125 in an open-source looped-model engine and grade our optimizations from bitwise-identical changes to those that require an accuracy check. At...
De-Zhen Lu, Run-Guo Li, Zi-Xiang Ni et al.· 0 citations
This work presents InSituANN, an IVF-based ANNS engine that enables billion-scale vector search on a single commodity GPU, and keeps original base vectors in host memory, performs fine search in situ, and uses the GPU for compact routing and optional pruning.
Yue-Meng Xu, Zongxi Liu, Junyu Long et al.· 0 citations
Vertumnus is presented, an adaptive CP serving system designed for heterogeneous and evolving workloads that reduces mean TTFT and improves token-weighted SLO attainment and introduces a global prefix-cache management policy that coordinates cache placement and replication among workers with the same or different CP de...
Jiarui Guo, Rong-Le Wang, Pei-Jun Huang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.