MATRIX AND VECTOR METHODS FOR LANGUAGE-MODEL EFFICIENCY: DISTINGUISHING EMBEDDING COMPRESSION, TOKEN REDUCTION, AND RAG
TL;DR
A structured critical review of matrix and vector methods used to reduce storage and inference costs in language models and retrieval-augmented generation (RAG) systems shows that matrix factorization, adaptive embeddings, and Matryoshka representations compress parameters or vectors but do not automatically decrease the number of input tokens.
Abstract
Language-model efficiency is often discussed through concepts that are related but not equivalent: token count, embedding dimensionality, parameter count, key-value cache size, and retrieved-context length. This article presents a structured critical review, supported by document-level traceability, of matrix and vector methods used to reduce storage and inference costs in language models and retrieval-augmented generation (RAG) systems. The analysis shows that matrix factorization, adaptive embeddings, and Matryoshka representations compress parameters or vectors but do not automatically decrease the number of input tokens. Direct token reduction requires sequence-level interventions, including appropriate tokenization, selection, prompt compression, gist tokens, and retrieved-context control. KV-cache compression reduces memory and latency without necessarily changing the tokens submitted to the model. We propose a mathematical framework that separates V (vocabulary size), d (embedding dimension), n (sequence length), k (number of retrieved documents or passages), and nKV (cache positions retained).