Stable Token-level Scheduling for Serverless LLM Inference
Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, improve GPU utilization through resource sharing, and reduce infrastructure costs. However, existing approaches primarily optimize time to first token (TTFT) while overlo...
Hardi Ashishbhai Desai, K. BalajiN, Sourav Kanti Addya
· 0 citations