Statically scheduled accelerators such as Groq's Tensor Streaming Processor (TSP) achieve low-latency large language model inference by stripping dynamic synchronization hardware from the chip. With no caches, arbiters, or hardware schedulers, the compiler must fix the exact slice and issue cycle of every operation, so...
Xu-Xiao Yang, Jun-Chi Wu, Yu-Qi Peng et al.· 0 citations
KV cache reuse accelerates LLM serving, but existing systems manage host DRAM and NVMe SSD as a tiered hierarchy where DRAM acts as an intermediate cache. Production LLM workloads break both premises of this hierarchy. First, bimodal reuse defeats DRAM temporal filtering, since workloads are dominated either by one-sho...
Xin Huang, Teng Ma, Zhuo-Ran Li et al.· 0 citations
HBF sucks as an SSD replacement for transient KV, but earns its place in LLM serving when used selectively with reuse-aware placement, write budgeting, and thermal coordination.
Zhuo-Ran Li, Zhuo-Hang Bian, Xin Huang et al.· 5 citations
DegreeSpar is introduced, which formulates secure Transformer compression as structured sparsification over nonlinear polynomial degrees, establishing structured polynomial degree as an effective shared optimization space for secure Transformer compression.
Yi-Fei Cai, Zhuo-Ran Li, Xiao-Zuo Shen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.