TL;DR
TAILOR is presented, a token-aware inference framework that jointly optimizes offline partitioning and online routing for autoregressive transformer serving that reduces end-to-end latency and lowers OOM-induced fallback compared with static partitioning baselines, demonstrating robust inference under long-tailed token workloads.
This paper employs five lightweight machine learning models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution.
KernelFlume is presented, a decode-centric architecture that disaggregates the stable projection/FFN path from core-attention computation: weight nodes execute dense projection/FFN kernels, while weightless attention nodes store token-range KV partitions and scale with request-state demand.
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
This work presents AsymSpec, which addresses uplink-gated verification and invalid dependent work with two corresponding mechanisms, and shows that AsymSpec delivers 2.82-28.03$\times the output-token throughput of the strongest baseline.
PRISM, a prediction-guided runtime framework that jointly selects model variants and CPU allocations for containerized edge microservices, and adapts each pipeline stage in place and minimizes predicted CPU-package energy under deadline, resource, and offline model-level Quality of Result constraints is presented.