Regularity-Driven Multi-objective Neural Architecture Search

Abstract

Deep neural networks can achieve very high accuracy, but their architectures are most often designed by hand and tuned for a single goal, such as accuracy. In practice, we often care about several goals at once, for example, accuracy, model size, and computational cost. Multi-objective neural architecture search (MONAS) can find a set of architectures that balance these goals, but the resulting models are usually very different from each other, which makes them hard to maintain and deploy as a family. In this work, we study how to make the diverse trade-off architectures to possess a regularity, so they can be better understood, maintained, and deployed with confidence. To this effort, we first run a standard multi-objective NAS to obtain a set of Pareto neural architectures, then analyze the common structural patterns that appear frequently among them. We then fix these common patterns and run a second MONAS in the reduced search space to obtain a set of regularized neural architectures. Across NAS-Bench-101, NAS-Bench-201, and an AG News text classification task, we show that this regularity-driven search can produce families of architectures that remain competitive in performance while being structurally simpler and having a regular pattern.