An Adaptive Neuro-Metaheuristic Framework Integrating Transformer-Based Surrogate Modeling with Differential Evolution for High-Dimensional Black-Box Optimization
TL;DR
The Adaptive Neuro-Metaheuristic with Transformer-based Transformer-based Differential Evolution (ANM-TDE), which combines a Transformer-based surrogate model with a self-adaptive Differential Evolution (DE) algorithm to greatly reduce the number of real function evaluations while maintaining a strong global search performance.
Abstract
High-dimensional black-box optimization is a highly challenging problem, especially when each function evaluation is expensive, the objective is non-differentiable, and we know almost nothing about its structure. In this case, traditional gradient-based methods are not available, and most existing metaheuristics either stagnate far away from the truly competitive solutions or require an impractically large number of evaluations once the dimension goes beyond about d=100. This paper directly addresses this challenge, and we propose the Adaptive Neuro-Metaheuristic with Transformer-based Differential Evolution (ANM-TDE), which combines a Transformer-based surrogate model with a self-adaptive Differential Evolution (DE) algorithm to greatly reduce the number of real function evaluations while maintaining a strong global search performance. The surrogate model is incrementally trained on all previously evaluated candidates, and uses multi-head self-attention to predict the fitness of candidates, while the DE engine consults the surrogate to pre-screen the trial solutions before paying the cost of real evaluations. We prove that the expected difference between the surrogate-predicted fitness and the true fitness converges with rate O(N^{-1/2}) as the size N of the archive increases, and that the overall method converges to a stationary point with rate O(1/K) in the nonconvex setting, assuming an L smooth bound on the surrogate model’s approximation error. On 12 benchmark problems with dimensions from d=30 to d=500, including Sphere, Rosenbrock, Rastrigin, Ackley, Schwefel, and Levy, ANM-TDE reduces the number of real function evaluations by 67-73% compared with standard DE, while achieving a mean best fitness of 1.47 x 10^{-12} on F1 (Sphere, d=100), versus 8.93 x 10^{-8} for CMA-ES. Wilcoxon signed-rank tests are applied, and statistically significant improvements are reported on 10 out of 12 benchmarks. The results show that ANM TDE is a competitive algorithm for expensive black box optimization problems such as engineering design, hyperparameter tuning and scientific simulation.