Evaluations demonstrate that MegaGraph enables training on large-scale graphs where state-of-the-art baselines fail due to out-of-memory (OOM) errors, and achieves up to 4.51 times training speedup while maintaining model accuracy.
Abstract
Graph Transformers (GTs) offer superior representation capabilities by overcoming the depth limitations and over-smoothing issues of traditional Graph Neural Networks (GNNs). However, scaling GTs to large graphs poses critical bottlenecks. Specifically, the attention score matrix and its associated topology-aware bias matrix jointly incur significant per-layer memory overhead, and heavy graph embedding layers result in severe workload imbalances. These characteristics are unique to GT training and are not addressed by parallelism techniques designed for either conventional GNNs or Transformers, making a dedicated solution necessary. This paper introduces MegaGraph, the first automated hybrid parallelism framework designed for efficient GT training. MegaGraph designs three specialized strategies, namely graph-aware context parallelism, heterogeneous pipeline parallelism, and hybrid data parallelism, to support efficient training on large-scale graphs. However, coordinating these three parallelism strategies yields an exponentially large configuration space. To address this complexity, an automatic search engine leverages precise cost models via a Profile - Model - Search workflow to identify the optimal parallelism configuration. Evaluations demonstrate that MegaGraph enables training on large-scale graphs where state-of-the-art baselines fail due to out-of-memory (OOM) errors. The framework reduces per-device peak memory by up to 77.8\% and achieves up to 4.51$\times$ training speedup while maintaining model accuracy.
Poseidon is an efficient and scalable LLM training framework designed with heterogeneity awareness, which employs two efficient, theoretically grounded strategies: stage-level pruning via early stopping with partial estimation, and layer-to-stage mapping exploiting a ridge-like distribution pattern.
Xiao-Song Chen, Shao Nie, Zhong-Min Zhao et al.· 0 citations
The rapid growth of large-scale deep neural networks has pushed inference workloads toward increasingly diverse shapes. In practical inference, these models are often invoked with tensors of varying shapes rather than a single fixed shape. Such shape variability reshapes operator loops and intermediate tensors, leading...
Hon-Gen Shao, Cen Chen, Xin Fang et al.· Journal of systems architect...· 0 citations
DeepSeq3 is introduced, a novel hierarchical framework that abstracts circuits into a two-level representation: fine-grained combinational subgraphs partitioned by flip-flops (FFs) and a high-level Super-Node Graph (SNG) that models the register-transfer structure.
Jing-Yi Zhou, Zhengyuan Shi, Jiaying Zhu et al.· 0 citations
This work introduces serial and parallel algorithms for multi-core CPUs, as well as the first GPU-based algorithm for multi-core CPUs, and introduces a grid structure the authors call FC-Grid, which is exploited to distribute work among threads.
Cheng Huang, Davide Mottin, Ira Assent· Proceedings of the VLDB Endo...· 0 citations
OmniPipe is proposed, a flexible bidirectional multi-pipeline parallelism scheme for unified dense and MoE LLM training that minimizes the pipeline bubble ratio while effectively overlapping EP communication with computation, enabled by the flexible and scalable parallelism scheme of bidirectional pipelines.
Jun Li, Zhi Ma, Shi-Gang Li· International Conference on...· 0 citations
QUARTET is an expressive graph transformer architecture that applies full self-attention on local subgraphs while enriching global context through cross-attention branches, and consistently matches or outperforms the current state-of-the-art graph transformer baselines (HGT and RelGT).
K. Myint, Nan Jiang, Xiang Li et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026