A Flexible Framework for Layer-Parallel CNN Training on FPGA Clusters
TL;DR
A flexible and scalable hardware/software framework for training CNNs on Ethernet-connected FPGA clusters using tightly pipelined layer parallelism, highlighting the potential of FPGA-based, layer-parallel training for separable-convolution-dominated CNNs.
Abstract
We present a flexible and scalable hardware/software framework for training CNNs on Ethernet-connected FPGA clusters using tightly pipelined layer parallelism. Starting from a high-level CNN and cluster description, the system automatically maps layers onto (potentially heterogeneous) devices, generates FPGA-specific bitstreams, and orchestrates fully streaming forward and backward passes while keeping most parameters and gradients in on-chip memory. We demonstrate support for general DAG-style CNNs, including MobileNetV2, MnasNet, and ResNet18, and replace batch normalization with online normalization to enable normalization in this streaming setting while achieving ImageNet validation accuracies comparable to PyTorch baselines with batch normalization for all three networks. A CP-SAT-based planner, driven by implementation-level resource estimates from SpinalHDL, performs resource-aware placement under constraints on DSPs, on-chip memory, DRAM bandwidth, and network bandwidth, and exposes FPGA-specific optimizations such as multipumped Matrix Multiplication engines, fabric-aware memory tiling, and activation recomputation. We evaluate throughput, resource utilization, and energy efficiency on an eight-board Altera Agilex 7 cluster and show that, for both MobileNetV2 and MnasNet, the framework achieves more than \(2\times\) lower energy per frame than Nvidia V100, A100, and H100 GPU baselines, highlighting the potential of FPGA-based, layer-parallel training for separable-convolution-dominated CNNs.