A Survey on Lightweight Deep Neural Network Architecture Design

Abstract

Despite the remarkable achievements of deep neural networks (DNNs) in numerous fields, the growing number of parameters and computational complexity severely limit their deployment feasibility on edge devices. Against this backdrop, lightweight DNNs have not only become a hot topic in academic research but also a key technological pathway to promote the democratization and implementation of AI. This article focuses on reviewing the methods of designing lightweight DNN architectures to achieve model lightweighting, aiming to provide researchers with effective solutions for designing lightweight model architectures. The article distinguishes between convolutional-based and Transformer-based frameworks for model design and delves into several typical lightweight model structural designs, development paths, and their pros and cons. By experimentally comparing the lightweighting metrics of different models, this article points out that model selection needs to be closely integrated with the constraints of specific application scenarios. Finally, the article observes that future breakthroughs may lie in exploring the lightweighting of hybrid architectures that combine convolution and Transformer, to integrate the advantages of local perception and global modeling, and further enhance model expressiveness while maintaining efficiency. In summary, this article not only provides a comprehensive review of lightweight model structural design but also emphasizes its practical guidance and development direction in promoting the implementation of edge intelligence.