Skip to content
#graph neural networks Book Open access

Memory-Aware Joint Optimisation of Partitioning and Scheduling for Pipeline-Parallel Training

Sep 2026 · 0 citations · 22 references

Abstract

Pipeline parallelism (PP) is widely adopted for distributed training of neural networks, but its efficiency is limited by pipeline bubbles that leave devices idle. A variety of schedules have been proposed to reduce these bubbles, yet most are hand-crafted, rely on idealised timing assumptions as well as rigid structural constraints, and cannot adjust to memory limitations. Moreover, stage partitioning, task scheduling, and memory management are often optimised in isolation, despite being tightly interdependent. In this paper, we present OptPipe, a unified framework that jointly optimises partitioning and scheduling for pipeline parallelism. We introduce a memory-aware directed acyclic graph (DAG) that captures both task dependencies and the lifetime of intermediate tensors, enabling explicit reasoning about the trade-off between execution efficiency and memory usage. Building on this representation, we present a mathematical formulation for pipeline parallelism that simultaneously determines stage partitioning, device assignment, task ordering, and memory feasibility. Unlike prior methods, our approach imposes no structural constraints on pipeline configurations and adapts naturally to a wide range of memory budgets and workload characteristics. Experiments on GPT-3 and LLaMA-3.2 show that OptPipe improves training throughput by up to 72.5%, 37.8%, and 38.0% over 1F1B, Chimera, and ZB-V, while remaining feasible under memory budgets where these baselines fail.

Read PDF

Similar papers

#computer vision Review Open access Mar 2016

Determinants of overweight or obesity among ever-married adult women in Bangladesh

The findings of the study indicate that a large number of women in Bangladesh are suffering from being overweight or obese, and multiple factors are responsible for this including, older age,being from wealthy households, higher education, being from food-secured households, watching TV at least once a week, and being...

Haribondhu Sarma, N. Saquib, M. Hasan et al. · 75 citations · ⚡5
#artificial intelligence Open access Sep 2026

Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaders

The rapid growth of scientific literature makes it increasingly challenging for researchers to identify novel and impactful ideas, especially across disciplines. Modern artificial intelligence (AI) systems offer new opportunities for scientific ideation, but how compelling are AI-generated ideas, and how can their qual...

Xuemei Gu, Mario Krenn · 65 citations
#computer vision Conference Aug 2008

Scrum in a Multiproject Environment: An Ethnographically-Inspired Case Study on the Adoption Challenges

Agile methods continue to gain popularity. In particular, the Scrum method appears to be on the verge of becoming a de-facto standard in the industry, leading the so called Agile movement. While there are success stories and recommendations, there is little scientifically valid evidence of the challenges in the adoptio...

A. Marchenko, P. Abrahamsson · 59 citations · ⚡11
#data science Review Open access Apr 2022

Association between Mother’s Education and Infant and Young Child Feeding Practices in South Asia

Findings demonstrate a strong positive association between mother’s education and IYCF indicators and suggest strengthening national policies to educate women at least to the secondary level in SACs might be a cost-effective intervention for improving IY CF practices.

M. Tariqujjaman, M. Hasan, M. Mahfuz et al. · 45 citations · ⚡4
#computer vision Open access Sep 2012

Making the leap to a software platform strategy: Issues and challenges

A comprehensive taxonomy of the challenges faced when a medium-scale organization decided to adopt software platforms is provided, namely: business challenges, organizational challenges, technical challenges, and people challenges.

Yaser Ghanam, F. Maurer, P. Abrahamsson · 41 citations · ⚡3

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.