Efficient serving of long-context reasoning-and-acting (ReAct) agents relies on KV cache reuse to reduce large language model (LLM) prefill latency and monetary cost. However, a semantic gap exists between agent harnesses and the underlying serving stack. Through context mutation, tool execution, and subagent coordinat...
Kai-Hua Fu, Yu-Kun Zhou, Chaokun Chang et al.· 0 citations
As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure alternative to public cloud services. However, these resource-constrained servers struggle to guarantee heterogeneous Service Level Objective...
Ze-Shen Zhang, Han Zhao, Wei-Hao Cui et al.· 0 citations
Microservice architecture has become the dominant paradigm for cloud-native applications, yet is often subject to resource fragmentation when Horizontal Pod Autoscaler (HPA) is used. While Vertical Pod Autoscaler (VPA) addresses this by adjusting pod resources, the Re-Create approach causes downtime, impacting availabi...
Zi-Chen Xu, Zi-Jun Li, Quan Chen et al.· Proceedings of the Internati...· 0 citations
Aethon is presented, a memory offloading system for public clouds that maximizes offloaded data while ensuring each application satisfies the service-level agreement (SLA).
Guang-Qiang Luan, Pu Pang, Quan Chen et al.· Proceedings of the Internati...· 0 citations
As applications demand increasing memory capacity in clouds, memory pooling provides a cost-effective way to improve utilization and expand capacity. Compute Express Link (CXL), which enables high-performance direct access to remote memory, makes this approach increasingly practical. However, existing studies of memory...
Guangqiang Luan, Pu Pang, Quan Chen et al.· Proceedings of the Internati...· 0 citations
Serverless edge computing, despite its flexibility and efficiency, is hindered by high startup latency during peak load. Remote fork, employing either Checkpoint/Restore (C/R) or Remote Direct Memory Access (RDMA), offers a potential solution for function scaling acceleration. Although RDMA fork is faster, the opportun...
Zhe-Xiong Li, De-Ze Zeng, Lin Gu et al.· IEEE Transactions on Mobile...· 0 citations
DSTAR, a software-hardware co-design framework that accelerates DiT inference by reducing spatial and temporal redundancy and incorporates a sparse attention reuse mechanism to minimize redundant computation in attention layers, and design a specialized hardware accelerator which achieves high efficiency in both latenc...
Chi Zhang, Jieru Zhao, Yu Feng et al.· arXiv.org· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
