The detachment mechanism of Forward-Forward (FF) is reinterpreted as a scheduling primitive: given a local objective, detaching a block's output removes downstream gradient dependencies, making its backward pass ready when its forward pass finishes.
Abstract
Lightweight proxy models enable rapid experimentation without repeatedly training frontier-scale systems, but their small kernels often leave modern accelerators underutilized. Conventional training compounds this inefficiency by scheduling the forward and backward passes as disjoint phases, so spare capacity in one cannot be filled by work from the other. We reinterpret the detachment mechanism of Forward-Forward (FF) as a scheduling primitive: given a local objective, detaching a block's output removes downstream gradient dependencies, making its backward pass ready when its forward pass finishes. ERASE launches each detached subgraph's backward pass early on a separate CUDA stream, overlapping it with subsequent forward work. Execution trace on a lightweight transformer demonstrates this overlap and its limit: a kernel that saturates the device leaves no capacity for concurrency. On a large-scale click-through-rate model, detaching six dense subarchitectures improves training throughput by up to $9.51\%$ while keeping normalized entropy close to the baseline.
This work presents LongStraw, an objective-aware, architecture-aware system for resident-state virtualization, response replay, and distributed-gradient execution that bounds the live training graph by the response suffix while reusing the expensive prompt computation across the complete GRPO group.
Changhai Zhou, Kieran Liu, Yuhua Zhou et al.· 2 citations
Prism abstracts the highly dynamic diffusion workload into a predictable, static execution flow transparent to the compiler, and achieves this via three techniques: spatial regularization, temporal stabilization, and specialized kernels that selectively bypass padding data.
Jianian Zhu, Hang Wu, Yinghui Li et al.· International Conference on...· 0 citations
SemaCredit is presented, a receiver controller that admits each remote-memory operation against a vector of target-resource demands and returns each component when its corresponding HBM, Atomic, or response stage completes, reducing small-operation P99 latency by 52.4% under Atomic contention and 10.2% under response incast.
Fan Yang, Jiaqi Liu, Tao Jiang et al.· 0 citations
Agora combines bandwidth-efficient pipeline-parallel model sharding over internet-grade links with multi-party, fault-tolerant collective operations, and enables collectively trained, collectively owned models, opening a path to open-source frontier training with economic sustainability.
Gil Avraham, Violetta Shevchenko, H. M. Dolatabadi et al.· 0 citations
Deep neural networks often achieve high accuracy, but their compute, memory, and power demands can hinder deployment on resource-constrained edge devices. This paper proposes an iterative, constraint-aware knowledge distillation framework that compresses a large model into a small one under an explicit model-size budget and a maximum allowable accuracy drop. Starting from user-specified constraints, the method trains candidate students, checks feasibility, and adjusts capacity across generations; oversized students may be promoted to intermediate teachers for subsequent distillation stages. Experiments on MNIST and Speech Commands validate the approach: on MNIST, a 100 KB target with a 2-point accuracy-drop budget yields a 55 KB student achieving 97.28% accuracy; on Speech Commands, a 4-point drop yields a 197.5 KB student reaching 94.08% accuracy. These results demonstrate predictable tradeoffs between compactness and accuracy, and show that multistage teacher promotion can bridge large capacity gaps while preserving performance within user-defined tolerances.
Mohamed Echchidmi, Anas Bouayad· IEEE International Conferenc...· 0 citations
This work proposes a BP-free algorithm, called ZeroLock, that decouples the model updates into independent chunk updates by local objective construction and provides the first theoretical framework for such local objective construction-based approaches under general model chunk division by mapping local objectives to the global objective.
Wentao Dai, Xuanran Li, Yuxiang Zhang et al.· 0 citations
