ALPS: An Application-Aware, Length-Predictive Scheduler for Efficient LLM Inference at Edge
The deployment of Large Language Models (LLMs) on resource-constrained edge server clusters introduces a critical systems challenge: achieving high throughput and low latency under strict GPU memory and bandwidth limits. Current schedulers are ill-suited for this environment, as they treat requests independently, incur...
Chen Chen, Lei Jiao, Richard Mortier
· IEEE Transactions on Paralle... · 1 citation