Ameerah Alsulami

Hashrope: A Persistent Data Structure for Mechanisms of LLM Context Management

Serving stacks for large language models manage context through four recurring mechanisms: incremental edits to a growing context, branching and snapshotting for tree search, identification of reusable prefixes for KV-cache reuse, and encoding of repeated segments. Each mechanism is served today by a different speciali...