Concurrent local LLM serving on unified-memory desktops must preserve memory headroom and output fidelity, which speed-only rankings overlook. We introduce SiliconBench, which evaluates nine Apple Silicon serving engines through three lenses: speed, memory, and fidelity. We evaluate chat and agent serving on Qwen3, Qwe...
Ran-Ran Zhang, A. X. Fan, David Munhá Correia et al.· 0 citations
Model-as-a-service platforms call for continuous optimization as complex serving conditions expose inefficiencies missed before deployment. Detailed always-on profiling can incur substantial overhead, while lightweight collection omits information needed for diagnosis. We present Herschel, a continuous optimization sys...
Lu-Ping Wang, Wei-Gao Chen, Yi-Fei Wu et al.· 0 citations
CacheScout is presented, an agent-aware KV-cache runtime layer for multi-agent LLM serving that improves KV-cache hit rate, reduces mean TTFT, lowers mean per-turn latency, and increases peak throughput by up to 57%.
Rui Zhang, Chaeeun Kim, Shaoting Feng et al.· 5 citations
Atrex-Bench is presented, a benchmark whose 30 operators and 440 shapes are sampled directly from full-cluster production inference traces of compute-limited, memory-rich GPUs, and a profile-driven kernel-optimization agent that combines iterative measure-revise search, optimization dropout for escaping stalled search...
