M. Loretz

Accelerating LLM Inference via Vector Index Based Output Embeddings

This work reformulates the output projection followed by top-k token selection as a maximum inner product search over token embeddings and replaces the dense vocabulary projection with an HNSW-based vector index, suggesting approximate retrieval is a practical alternative to dense output projections in latency-sensitive small-batch decoding.