Automatic Model Compression and Quantized Deployment of Convolutional Neural Networks on Programmable Data Planes

Abstract

The rapid development of programmable network devices and the widespread adoption of machine learning (ML) in networking have facilitated efficient research into intelligent data planes (IDPs). Offloading ML to programmable data planes (PDPs) enables quick analysis and responses to network traffic dynamics, and efficient management of network links. Compared to using an external low-cost board with sufficient memory and a general-purpose CPU, IDP deployment keeps inference inside the switch forwarding pipeline, avoiding inter-device transfer and coordination overhead. This enables line-rate processing and faster response for real-time network control. However, the hardware pipeline presents significant resource limitations. For instance, Intel Tofino ASIC has only 10Mb SRAM in each stage, and lacks support for multiplication, division, and floating-point operations. These constraints significantly hinder the development of IDP. This paper presents Quark, a framework that automatically compresses the convolutional neural network (CNN) and fully offloads quantized inference onto PDP. Quark employs model pruning to simplify the CNN model, uses quantization to support floating-point operations, and utilizes neural architecture search to balance accuracy and PDP resource constraints. Additionally, Quark divides the CNN into smaller units to improve resource utilization on the PDP. We have implemented a testbed prototype of Quark on both P4 hardware switch (Intel Tofino ASIC) and software switch (i.e., BMv2). Extensive evaluation results on the ISCX Botnet dataset demonstrate that Quark achieves 97.3% accuracy while using only 24.27% of the SRAM resources on the Intel Tofino ASIC switch, completing inference tasks at line rate with an average latency of $42.66\mu s$ .