This work evaluates Constrained Bayesian Optimization with the primary objective of minimizing energy consumption and subject to the constraint that the generalization performance is above some threshold and demonstrates that CBO achieves lower energy consumption without compromising the predictive performance of ML models.
Abstract
Bayesian optimization (BO) is an efficient framework for optimization of black-box objectives when function evaluations are costly and gradient information is not easily accessible. BO has been successfully applied to automate the task of hyperparameter optimization (HPO) in machine learning (ML) models with the primary objective of optimizing predictive performance on held-out data. In recent years, however, with ever-growing model sizes, the energy cost associated with model training has become an important factor for ML applications. Here we evaluate Constrained Bayesian Optimization (CBO) with the primary objective of minimizing energy consumption and subject to the constraint that the generalization performance is above some threshold. We evaluate our approach on regression and classification tasks and demonstrate that CBO achieves lower energy consumption without compromising the predictive performance of ML models.
A unified framework, KENDO (Kernel ENsemble Disagreement-aware Operator), is proposed that integrates Ensemble Gaussian Processes (EGP) with disagreement-aware acquisition strategies and extends the approach to multi-objective optimization via random scalarization that preserves the single-optimizer conditioning structure.
Heng Zhang, Haotian Xiang, Qin Lu et al.· 0 citations
A surrogate-recommendation framework is introduced that predicts the most suitable BO surrogate from inexpensive dataset characteristics and establishes FruBO as a reproducible, compute-aware baseline for Bayesian Optimization and provides practical guidance for surrogate selection under limited computational and experimental budgets.
P. Krokidas, C. Rekatsinas, Vassilis Sioros et al.· 1 citation
Many design tasks can be cast as black-box function optimization, enabling use of Bayesian optimization to find an ideal design with minimal number of trials. However, often we do not actually need the optimum but instead a sufficiently good solution is enough, for instance a material that is durable enough for its intended use. In most cases there are multiple satisfactory solutions, forming a superlevel set of the function, raising a key question of which one to prefer. We answer this by explaining why robustness to input perturbations that may occur when the solution is deployed is a good criterion and by introduce a Bayesian optimization method that efficiently finds satisficing solutions that are robust to maximally large perturbations. In contrast to previous works, we assume the inputs can be accurately controlled during optimization, but will be perturbed after the deployment.
Samuli Kinnunen, P. Mikkola, Antti Niskanen et al.· 0 citations
This study proposes an adaptive hyperparameter optimization framework based on the Bobcat Optimization Algorithm for tuning the learning rate and momentum of an ANN, and demonstrates that BOA-ANN achieves a test accuracy of 98.30%, outperforming Bayesian optimization, the Secretary Algorithm, and Automated Model Compression under identical settings.
Mohammed Majid Abdulrazzaq, Nehad T. A. Ramaha, A. A. Hameed et al.· PeerJ Computer Science· 0 citations
Gradient-free post-training has emerged as a compelling alternative to gradient-based optimization for large language models (LLMs), but existing approaches remain costly. We ask whether structured search can identify a strong single expert under a modest evaluation budget. Motivated by evidence that useful weight updates lie in low-dimensional subspaces, we apply Bayesian optimization within a random linear embedding of weight space. Our method requires no backpropagation and uses a Gaussian process surrogate to guide candidate evaluations efficiently. Across several reasoning benchmarks with Qwen2.5-Instruct models from 0.5B to 3B parameters, Bayesian optimization using five times less candidate evaluations matches or exceeds RandOpt. These results show that surrogate-guided search can substantially reduce the evaluation cost of gradient-free post-training while producing stronger deployable single experts.
Nigel Bastian Cendra, Abdelhamid Ezzerg, Fernando Julio Cendra et al.· 0 citations
A generalizable transformation methodology is developed that converts compatible single- or multi-objective sequential Bayesian optimization methods into B-MOBO methods that are as effective as existing methods at solving optimization problems in terms of solution quality and show improvements when real-time is considered.
Adelle Holder, H. Debruin, Jesse M. Sestito· PLoS ONE· 0 citations
