Luis Carranza

Many Tenants, One Cache: Per-Tenant Rotation of the Attention Cache in a Production Serving Engine

Language model serving engines put many customers into one process on one accelerator, sharing one attention cache. Per-tenant rotation stores each tenant’s cached rows under a tenant-specific orthogonal transform and rotates the tenant’s query the same way, so attention is unchanged for the tenant and a read under any...