AI Networking Cookbook: Practical recipes for AI-assisted network automation and development
TL;DR
LLM serving systems cache prompt KV state is serving systems cache prompt KV state, yet most front ends still re-tokenize the full request text on every call, leading to a mismatch that is most costly for coding agents.
GraphTok is presented, a GPU tokenizer that runs the entire encoding pipeline, from pre-tokenization through byte-pair encoding (BPE) merging to token-ID assembly, as one replayable CUDA graph.
TokenCast is proposed, which learns a composable cost representation for each execution segment, recording its own consumption and the context growth it introduces, and captures the extra input cost incurred when context from earlier segments is re-read by every later call.
It is found that fewer tokens need not mean faster or cheaper execution: on Terminal-Bench with Qwen, policies using roughly one-third as many tokens can take 20-80% longer than the uncompressed agent.
The experiments show that token-wise replay in AgentReplay effectively eliminates workload variation that greedy decoding and length-wise replay cannot avoid, enabling fairer performance comparisons across serving configurations.
EfficientAgent is presented, which sizes and manages the host tier by this working set, which estimates the working set from agent histories to size the host tier and its predictions, made before the experiments, located the capacity at which offloading starts to pay.
JustFit is presented, an MLX runtime that coordinates their overlapping allocations: KVExec executes and checkpoints four-bit KV with bounded workspace, PhaseSwap loads phase-dependent components, and StateTrans preserves history across execution modes.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.