A Suitability Score to Optimize CNNs on an FPGA Accelerator

Abstract

This paper proposes an optimization methodology for convolutional neural networks (CNNs) on field programmable gate arrays (FPGAs) under strict latency and hardware constraints. The approach integrates CNN model restructuring, hardware-aware scheduling, and FPGA-based hardware-software co-design to reduce computational overhead, while preserving accuracy. A suitability score is introduced to identify the CNN layers best suited for hardware-efficient transformation. Selective architectural modifications and layer-specific pipelining based on this score improve energy efficiency and inference latency. The optimized CNN model is deployed on an Advanced Micro Devices (AMD) Kintex UltraScale+ FPGA and evaluated using an esophageal cancer detection task. Compared to graphics processing unit (GPU)-based inference, the design achieves up to 47.6% higher throughput and more than twice the energy efficiency. Compared with prior FPGA implementations, it delivers up to $7.8\times $ speedup. The FPGA accelerator processes 76.19 images per second with 97.45% accuracy, demonstrating its suitability for real-time, resource-constrained medical and other image processing applications.