Efficient serving of long-context reasoning-and-acting (ReAct) agents relies on KV cache reuse to reduce large language model (LLM) prefill latency and monetary cost. However, a semantic gap exists between agent harnesses and the underlying serving stack. Through context mutation, tool execution, and subagent coordinat...
Kai-Hua Fu, Yu-Kun Zhou, Chaokun Chang et al.· 0 citations
As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure alternative to public cloud services. However, these resource-constrained servers struggle to guarantee heterogeneous Service Level Objective...
Ze-Shen Zhang, Han Zhao, Wei-Hao Cui et al.· 0 citations
Microservice architecture has become the dominant paradigm for cloud-native applications, yet is often subject to resource fragmentation when Horizontal Pod Autoscaler (HPA) is used. While Vertical Pod Autoscaler (VPA) addresses this by adjusting pod resources, the Re-Create approach causes downtime, impacting availabi...
Zi-Chen Xu, Zi-Jun Li, Quan Chen et al.· Proceedings of the Internati...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
