Nov 2026· Proceedings of the 35th ACM International Conference on Information and Knowledge Management· 0 citations· 5 references
TL;DR
GIGO is introduced, a meta-optimizer that estimates a sparse directed graph of cross-block gradient dependencies online, partitions each step into contiguous sequential stages along this graph, and refreshes downstream gradients using a stale-activation approximation, and is compatible with mixed precision and data parallelism.
Abstract
Training large Transformer language models is dominated by the cost of synchronous parameter updates: every block of weights is moved at every step using a gradient computed before any block has moved. The resulting trajectory is wasteful, because per-block updates are evaluated on a frozen snapshot of the rest of the network that the next step will already invalidate. Second-order methods such as Shampoo and K-FAC address this spatially through preconditioning, and sharpness-aware methods such as SAM trade compute for sharper minima, but both leave the temporal coupling between blocks untouched. We introduce Gradient Interaction Graph Optimization (GIGO), a meta-optimizer that estimates a sparse directed graph of cross-block gradient dependencies online, partitions each step into contiguous sequential stages along this graph, and refreshes downstream gradients using a stale-activation approximation. The framework wraps any base optimizer, adds one cached boundary activation per stage (~32 MB at Tseq= 1024, scaling linearly with sequence length), and is compatible with mixed precision and data parallelism. On GPT-2 Small through XL (124M-1.5B), GIGO-Adam reaches the same OpenWebText/C4 perplexity in 32.0-34.0% fewer optimizer steps than synchronous AdamW --- a lead of 8-22 percentage points over the strongest 2024-vintage baselines (SOAP, Muon, Sophia, Lion, GaLore) on the Small comparison cohort --- with a per-step overhead of 32-38% and a net wall-clock benefit ranging from +5.2% (Small) to +16.2% (XL) across the four sizes tested. The benefit vanishes or reverses where the method's preconditions do not hold: a synthetic block-diagonal control (SRR=0.0%), the overhead-dominated Penn Treebank stress test (-18.5% wall-clock), and a weak-coupling small-vision-Transformer diagnostic (-12.4% wall-clock on CIFAR-100). Ablations decompose the mechanism: the partial-re-forward refresh primitive alone (random partition) accounts for 9.4% SRR, the interaction-graph schedule contributes a further ~22pp, and the layer-contiguity constraint controls overhead. Our convergence theorem matches the standard Adam O(T-1/4) rate uniformly in graph-estimation quality, providing a safety guarantee; the empirical step-reduction is reported as an empirical regularity not formally predicted by the theory.
Multimodal large language models process the complete visual token sequence, increasing prefill cost even when many tokens are irrelevant to the instruction. Token-wise saliency captures limited inter-token dependence; graph-based selection instead depends on how relations govern importance propagation and ranking. We...
Gated Delta Networks (GDNs) offer linear-time sequence modeling, yet long-context prefill remains costly due to dense intra-chunk computation, recurrent-state updates, and intermediate state storage. We introduce SparseGDN, a hardware-aware framework that combines structured intra-chunk approximation with input-depende...
Jie-Dong Zhuang, Peng Zhao, Jia-Lin Tang et al.· 0 citations
Graph embedding represents nodes in a continuous, low-dimensional latent space while preserving the graph’sunderlying structure. In practice, however, most real-world graphs are far from static — nodes and edges appear, vanish,and shift in weight as time passes. Despite this, a large portion of the embedding literature...
Experiments on five dynamic graph benchmarks show that CacheDyG adopts substantially fewer trainable parameters and lower runtime to obtain more competitive predictive performance than baselines, demonstrating that cache-based decoupling provides an effective principle for scalable dynamic graph learning.
Semantic-Geometric Decoupled Routing is proposed, a training-free block routing framework that shifts semantic aggregation to the pre-RoPE space and reconstructs geometric bias with an offline structural prior and relative block distances and yields an explicit closed-form block routing score without token-level search...
CommunityKV, a framework that formulates sparse attention as a community detection problem, is introduced, a framework that formulates sparse attention as a community detection problem and is evaluated on Qwen3 and Llama-3.1 models across three long-context benchmarks.
J. McKenna, Anastasios Alexandridis, Nathan Susanj et al.· 0 citations