Agentic search and automated optimization of GPU kernels are powerful tools for large language model inference. Their effectiveness, however, depends not on the sophistication of the search itself, but on the clarity of the optimization problem being solved. We provide an application-first approach that drives a hierar...
This work introduces MINCE (Monte Carlo Informed N-sizing for Compact Evaluation), which uses Monte Carlo simulation over per-item logs from a small set of calibration models to find the minimum subset size that bounds accuracy drift and then fixes a randomly sampled subset at that size, with no prediction layer needed...
This work frames 4-bit KV caching around multi-round agent workloads where task quality, cache residency, and serving throughput must be measured jointly, and presents serving optimizations on AMD GPUs, including optimized decode-attention kernels and UltraQuant.
Inesh Chakrabarti, David Limpus, Aditi Rana et al.· arXiv.org· 0 citations
This paper presents FastTPS, a high performance and low-precision loss method for accelerating the token-phase in LLM inference on general AI accelerators which includes three key components: AI accelerator-enabled reloading-free KV Cache concatenation which decreases memory access overhead as well as enables full fusi...
Wenzong Yang, Danyang Zhang, Kunteng Cao et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
