Hierarchical Context Compression for Efficient Long-Sequence Diffusion Language Modeling

Abstract

Diffusion language models (DLMs) generate text by iteratively denoising an entire sequence in parallel, an appealing alternative to token-by-token autoregressive decoding. On long inputs, however, conditioning every denoising step on the full history costs time and memory that grow linearly with sequence length, and recent segmented decoders that compress each completed window into a small, fixed set of summary tokens still incur a total injected context that scales linearly with the number of windows. We propose Hierarchical Context Compression (HCC), which organizes per-window summaries into a suffix-dyadic tree of “summaries of summaries”: recent windows are kept at fine resolution while progressively older windows are covered by recursively merged, coarser summaries. HCC uses only two learned operators—a leaf compressor and a shared binary MERGE—applied through a segment-tree decomposition of the window history. We prove that HCC reduces the active context injected at each denoising step from O(n) to O(log n) summary vectors while still covering the entire history (no window is dropped), and we give a rate–distortion bound predicting graceful rather than abrupt degradation of far-context fidelity. As full-scale DLM training exceeds our single-CPU budget, we validate the mechanism with controlled and real-text information-retention studies and with direct latency and memory measurements. On histories of 64 windows, HCC retains 5–6× more recoverable information about past windows than a fixed recent-window budget of equal asymptotic footprint, and its measured per-step latency is up to 240× lower than uncompressed full-history attention at 4096 windows while remaining essentially flat as the history grows. HCC directly realizes the hierarchical-compression direction identified as future work by recent segmented-decoding methods.