Practical Modeling for Split DNN Inference on Near-Edge Accelerators
Splitting complex model inference between multiple computing devices can overcome latency and energy constraints at the edge. Newer edge accelerator devices with higher computing capacity and energy efficiency, enable more fine-grained offload throughout layers of the network, leading to the potential for multiple split configurations. However, optimizing a DNN for inference across networked devices requires a precise performance model that can guide design choices. In this paper, we demonstrate that existing models for computation and communication latency are inaccurate due to system considerations and propose a new empirical model based on structured benchmarking, considering data ingestion overhead due to transfers between devices as well as within a device for data to reach the GPU. We validate our split inference performance model using VGG16 and ResNet50 networks on two heterogeneous platforms, showing it achieves a mean absolute error of no more than 4% for both DNNs, significantly outperforming previous models with errors of more than 15%. We also validate the practical utility of our model by incorporating it into existing split inference search algorithms under multi-split, dynamic bandwidth, and multi-tenant scenarios, demonstrating its effectiveness in navigating the split inference search space.