This work proposes NeurGO, a generative MetaBBO framework that directly synthesizes elite candidates from historical population states and designs a quality-diversity loss to maintain solution quality and population diversity throughout the search.
Abstract
Expensive black-box optimization is ubiquitous in science and engineering, where function evaluations are costly and the evaluation budget is limited. Traditional evolutionary algorithms and Meta-BlackBox Optimization (MetaBBO) approaches typically consume most evaluations on candidate selection, often wasting precious budget on inferior solutions. Although surrogate-assisted evolution and Bayesian optimization aim to reduce evaluations through surrogate models, constructing an accurate global model from limited data remains challenging, and model bias can easily trap the search in local optima. To overcome these limitations, we propose NeurGO, a generative MetaBBO framework that directly synthesizes elite candidates from historical population states. Specifically, we employ an attention-based encoder to capture the population-level search trend and condition a decoder on this representation to generate high-quality candidates, avoiding the expensive evaluation of large offspring pools. We then design a quality-diversity loss to maintain solution quality and population diversity throughout the search. Through extensive benchmarking on CEC 2008 and the COCO BBOB test suites, our method achieves better optimization performance under the same evaluation budget and exhibits faster convergence.
Evolutionary algorithms are well suited to neural architecture search and other combinatorial design problems, but their scalability is often limited by the high cost of fitness evaluation. This paper studies evolutionary multi-objective optimisation in a regime where fitness evaluations are effectively free, enabled by a training-free proxy for neural network expressivity. We propose SWAP-Lite, a Pareto-guided evolutionary algorithm that maintains an explicit archive of non-dominated solutions over representational capacity and deployment cost, yielding an anytime optimiser that exposes budget-feasible solutions throughout the search. Using a MobileNet-style architecture space as a case study, we instantiate the fitness function with a sample-wise activation pattern proxy and perform large-scale evolutionary searches with up to 105 architecture evaluations. Experiments on CIFAR-10 and ImageNet show that SWAP-Lite discovers compact architectures that are competitive with state-of-the-art training-based and zero-shot baselines, while reducing search cost by one to four orders of magnitude. Analysis of the evolutionary dynamics demonstrates that explicit bi-objective optimisation produces higher-quality constrained Pareto fronts and superior anytime hypervolume compared with random search, greedy local search, and single-objective evolutionary baselines.
Jingyue Cong, Kai Wei, Yameng Peng et al.· Annual Conference on Genetic...· 0 citations
LLM-driven evolutionary search can discover algorithm designs that achieve Pareto-efficient trade-offs difficult to reach through manual design, with SMAC hyperparameter optimization integrated into the evolutionary loop.
G. Laskaris, R. Brasher, Niki van Stein et al.· 0 citations
Large Language Models (LLMs) are opening new directions for automated heuristic design (AHD), allowing evolutionary methods to create and enhance heuristics for constrained optimization problems (COPs). However, most existing approaches face the challenge of the exploration-exploitation balance, where the evolution needs to escape convergence to homogeneous populations and discover as large a heuristic landscape as possible. To address this challenge, we introduce Quality-Diversity Evolution (QDEvo), a multi-objective framework that integrates Quality-Diversity optimization with LLM-based AHD. At its core, QDEvo employs a semantic survival selection mechanism that clusters algorithms by functional similarity, then applies local Pareto competition. Evaluation on well-known COPs benchmarks and real-world problems shows that our method consistently outperforms the state-of-the-art baseline in both Hypervolume and Inverted Generational Distance metrics. These results facilitate further exploration of the algorithmic design space, while ensuring competitive solution quality and efficiency.
Nam Do Khanh, Nhat Nguyen Tran Minh, Dat Pham Vu Tuan et al.· Proceedings of the Genetic a...· 1 citation
Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers (LOs), their meta-training remains biased toward short-unroll learning on particular tasks, resulting in redundant computation and leaving LOs often unable to compete with hand-designed optimizers. We introduce Efficient Long-hOrizon (ELO) learning, an efficient meta-training algorithm that (1) reallocates wasted meta-training compute to longer failure regimes, achieving efficient long-horizon learning, and (2) enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs. Our empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs. Across downstream language modeling (GPT-2-124M/350M on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K) tasks, ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. In particular, ELO-Celo2 consistently outperforms well-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling. \textit{Notably, all ELO baselines require less than 7 H100 GPU-hours for meta-training.}
Xiaolong Huang, Benjamin Thérien, James Harrison et al.· 0 citations
Finding the right balance between exploration and exploitation is a crucial challenge in evolutionary algorithms (EAs), as it strongly impacts solution quality and the ability to escape local optima. While diversity-preserving mechanisms are commonly employed, high population diversity does not necessarily translate into improved performance across problem landscapes. Recent empirical evidence suggests that effective EA optimization largely depends on predominantly exploitative search dynamics, with exploration being most beneficial during the early iterations. Nevertheless, sporadic exploratory bursts may help reintroduce diversity and enhance solution quality. In this study, inspired by dropout regularization in neural networks (NNs), we introduce three mechanisms—and their variants— that stochastically deactivate entire individuals or subsets of decision variables with the expectation of enhancing exploration. We assess the effectiveness of dropout in genetic algorithms (GAs) on twelve single-objective and five multi-objective benchmark functions. We empirically demonstrate that per-variable stochastic deactivation, termed Individual Dropout (IDrop), significantly outperforms a standard GA on single-objective problems with exploitable global structure and improves convergence toward the theoretical Pareto front in multi-objective settings. We also show that, while dropout alone is not exempt from the curse of dimensionality, combinations of dropout mechanisms can partially alleviate this limitation in most of the tested functions.
Daniele Ganci, L. Correia· Annual Conference on Genetic...· 0 citations
Multi-objective optimization problems (MOPs) are prevalent in numerous real-world applications. Recently, Pareto Set Learning (PSL) has emerged as a powerful paradigm for solving MOPs. PSL can produce a neural network for modeling the set of all Pareto optimal solutions. However, applying PSL to black-box objectives, particularly those exhibiting non-separability, high dimensionality, and/or other complex properties, remains very challenging. To address this issue, we propose leveraging evolution strategies (ESs), a class of specialized black-box optimization algorithms, within the PSL paradigm. Traditional ESs capture the complex dimensional dependencies less efficiently, which can significantly hinder their performance in PSL. To tackle this issue, we suggest encapsulating the dependencies within a neural network, which is then trained using a novel gradient estimation method. The proposed method, termed Neural-ES, is evaluated using a bespoke benchmark suite for black-box PSL. Experimental comparisons with other methods demonstrate the efficiency of Neural-ES, underscoring its ability to learn the Pareto sets of challenging black-box MOPs.
Chengyu Lu, Zhenhua Li, Xi Lin et al.· Neural Information Processin...· 0 citations