Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload. These limi...
Hao Kang, Wei-Li Xu, Zi-Yang Li et al.· 0 citations
AutoEMatic, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime, is presented, a full-stack co-design that adapts the precision of weights, activations, and the KV cache at runtime.
Akshat Ramachandran, Reena Elangovan, Rangharajan Venkatesan et al.· 0 citations
Using neuro-symbolic AI as an end-to-end case study, it is shown that co-designing the full software-to-silicon stack yields orders-of-magnitude efficiency gains while preserving the accuracy and interpretability that make cognitive AI compelling.
Zi-Shen Wan, Yu Cao, S. K. Gupta et al.· IEEE Micro· 0 citations
This paper introduces 𝑆 2 Loop, a framework that introduces a flexible, all-level partitioning based AuthBlocks for ensuring full coverage of the entire design space, a realistic layout-based memory model, and an Authentication-Layout-Mapping co-search algorithm to explore the drastic combinatorial design space to figu...
J. Strzeszynski, Jianming Tong, Kyungmi Lee et al.· 0 citations
DynamoServe is presented, a multi-tenant LLM serving framework that addresses challenges through three key innovations: leveraging stranded GPU memory to offload model weights and KV caches, mitigating resource fragmentation in multi-workload environments, and improving memory locality through coordinated data placemen...
Diman Zad Tootaghaj, Khaled Diab, Bob Lantz et al.· Conference on Applications,...· 0 citations
TRACE-Router is presented, a task-level routing framework that aligns routing with the unit of supervision, and learns routing policies that adapt to the workload while avoiding explicit task-complexity estimation.