Many Tenants, One Cache: Per-Tenant Rotation of the Attention Cache in a Production Serving Engine

Abstract

Language model serving engines put many customers into one process on one accelerator, sharing one attention cache. Per-tenant rotation stores each tenant’s cached rows under a tenant-specific orthogonal transform and rotates the tenant’s query the same way, so attention is unchanged for the tenant and a read under any other key returns nothing usable. We measure what this costs, and whether it survives the features that make modern serving dense, in a production engine (vLLM) on one confidential H100, across five models and a set of pre-registered phases. Three results stand out. First, rotation extends to multi-head latent attention: on Moonlight 16B, a memory-bound ceiling of 793 tenants against 822 without rotation at 2,048 tokens, a prefill gap of 14.3 ms and a decode gap of 0.74 to 1.35 ms per token, with every isolation gate passed. Second, one H100 holds 4,000 registered Moonlight tenants at a 2-second service target, with or without rotation, by parking idle tenants off the card and reviving them in 35.3 ms against 32.8 ms without rotation. The ceiling is host memory for the parked tenants, not the GPU. Third, a shared-prefix tier that holds a common system prompt once halves a new tenant’s first-request time (24.9 against 48.0 ms) and makes a parked tenant about 17 times smaller (4.0 against 67.7 MB). On the key path, fusion and CUDA graphs bring rotation’s decode cost to 0.8 to 2.2 ms per token, and an FP8 cache doubles the tenant ceiling under rotation. The overhead targets we registered failed in every boot, and the Moonlight density figures rest on 64 keys. Both are reported in full. What rotation does not hide is measured in the companion Paper B. Companion paper: “What Per-Tenant Rotation Leaks” (doi:10.5281/zenodo.23452292).