Local Search Hyper-heuristics For Neural Architecture Search
TL;DR
The proposed algorithm, Local Search Hyper-Heuristic NAS (LSHH-NAS) works by exploring the neighborhood of the current solution through applying low-level heuristics to an alternative space called the operator space, which maps onto the architecture search space.
Abstract
Designing neural network architectures remains a resource intensive process that typically relies on repeated expert decisions and costly experimentation. Neural Architecture Search (NAS) addresses this challenge by automating architecture design. In this study we investigate the potential of a local search hyper-heuristic for NAS. The proposed algorithm, Local Search Hyper-Heuristic NAS (LSHH-NAS) works by exploring the neighborhood of the current solution through applying low-level heuristics to an alternative space called the operator space, which maps onto the architecture search space. To mitigate cycling induced by the many-to-one mapping from operator strings to architectures, the local-search procedure is augmented with a tabu memory of recently visited architectures. We evaluate LSHH-NAS on three established benchmarks: NAS-Bench-101, NAS-Bench-201, and NAS-Bench-301. Our results show that LSHH-NAS attains state-of-the-art or competitive performance, by achieving 94.22% and 94.92% test accuracy on NAS-Bench-101 and NAS-Bench-301 respectively. While also managing 94.37%, 73.51% and 47.31% test accuracy on all three datasets of NAS-Bench-201. These results were achieved with at least 22% and at most 65% fewer queries than competing methods, indicating a favorable effectiveness-efficiency trade-off.