Systematic review of training-free neural architecture search: research progress, core challenges, and future directions

Abstract

Neural Architecture Search (NAS) serves as a performance engine for automated machine learning, driving intelligent advancements across various domains. However, its development has been significantly constrained by the prohibitively high computational costs involved. Training-Free Neural Architecture Search (TFNAS) addresses this bottleneck by enabling rapid architecture evaluation through extremely low-cost proxy metrics, where “training-free" refers to the absence of candidate network training during the search loop, and its broader interpretation permits offline pre-training of lightweight predictors that do not constitute search-stage training. This paper presents a systematic, critical review of TFNAS. First, the core concepts and practical advantages of TFNAS are revisited, defining it as a distinct and emerging research paradigm. Second, a two-dimensional taxonomy is proposed based on the source of information and the construction methodology of proxy metrics, aiming to clarify the developmental trajectory of state-of-the-art techniques in this field. Furthermore, this work delves into the evolutionary mechanism involving the co-development of evaluation metrics and search strategies, positing that the joint optimization characterized by dynamic feedback and mutual reinforcement is key to maximizing efficacy. Finally, this review reveals four core contradictions that persist as the technology progresses toward practical application and outlines future directions. This review offers valuable references and conceptual inspiration for the maturation of TFNAS, laying a solid foundation for its reliable and widely applicable real-world deployment.