Results show that multi-objective Bayesian optimization is valuable as a search layer for expressive merge parameterizations in model merging and introduce MOBO-Merge, a merge-operator agnostic framework that uses multi-objective Bayesian optimization to approximate the Pareto front under a limited evaluation budget.
Abstract
Model merging combines trained models directly in weight space, offering a compute-efficient alternative to additional fine-tuning. Selecting merge parameters is nevertheless difficult because downstream evaluations are expensive, gradients are unavailable, and source capabilities can conflict. We formulate merge-parameter selection as a black-box multi-objective optimization problem and introduce MOBO-Merge, a merge-operator agnostic framework that uses multi-objective Bayesian optimization to approximate the Pareto front under a limited evaluation budget. We evaluate Qwen3-4B and Llama-3.1-8B in two-model instruction-math and three-model instruction-math-code settings using Linear, SLERP, TIES, and block-wise merge operators. On held-out benchmark partitions, MOBO-Merge obtains higher mean hypervolume than random search in 11 of 12 reported comparisons. The gain is small for one-dimensional Linear interpolation but substantially larger for several TIES, block-wise, and three-objective searches. No merge operator is uniformly best: TIES leads in three of four family-setting combinations, whereas Block-Linear 4x is strongest for the Llama three-model merge. These results show that multi-objective Bayesian optimization is valuable as a search layer for expressive merge parameterizations.
OptiMerge is proposed, a novel multi-objective evolutionary framework tailored for LLM model merging that discovers well-structured Pareto fronts and achieves improved performance trade-offs compared to base models and grid-search baselines.
Sri Srinivasa Raju Modampuri, Aojun Lu, Yanan Sun· Proceedings of the Genetic a...· 0 citations
LLM-driven evolutionary search can discover algorithm designs that achieve Pareto-efficient trade-offs difficult to reach through manual design, with SMAC hyperparameter optimization integrated into the evolutionary loop.
G. Laskaris, R. Brasher, Niki van Stein et al.· 0 citations
Multi-objective Bayesian optimization (MOBO) is effective in identifying the Pareto fronts for expensive black-box problems. However, most current MOBO approaches are limited to low-dimensional decision space due to its exponential sampling complexity. This paper presents decision variable interaction analysis-based MOBO, ViaMOBO, a generic framework for expensive multi-objective problems with high-dimensional decision space. The key idea of ViaMOBO is that it utilizes a variable interaction analysis model to determine whether the decision space can be completely or partially divided, and then performs local Bayesian optimization in the divided decision subspaces. Through the variable analysis model, it can be derived whether the objectives in black-box problems are separable, partially separable, or non-separable based on the potential independent or interdependent relationships among decision variables without any strong assumptions. We compare ViaMOBO with the state-of-the-art MOBO methods on both synthetic and real-world benchmarks. The experimental results demonstrate that ViaMOBO outperforms other related MOBO baselines in approximating the Pareto front of high-dimensional expensive multi-objective problems.
A unified framework, KENDO (Kernel ENsemble Disagreement-aware Operator), is proposed that integrates Ensemble Gaussian Processes (EGP) with disagreement-aware acquisition strategies and extends the approach to multi-objective optimization via random scalarization that preserves the single-optimizer conditioning structure.
Heng Zhang, Haotian Xiang, Qin Lu et al.· 0 citations
Ranking models are repeatedly refined within established model families, yet the choice of which component to scale, replace, or retire is often guided by expensive trial-and-error. We present Model Internal State Optimization (MISO), a systems workflow that uses model internal states (MIS), including parameters, activations, gradients, and normalization statistics, to prioritize such local optimization decisions. MISO extracts MIS from a trained ranking model, aggregates them into ranking, alignment, and comparison signals, and converts those signals into a small set of interpretable candidate edits. Because MIS are re-extracted after each retraining cycle, MISO naturally supports an adaptive optimization workflow that tracks evolving model behavior as data distributions and system requirements shift over time. In an ads ranking case study, MISO improves normalized entropy while requiring substantially fewer validation runs than expert-driven and black-box scaling workflows, offering a practical middle ground between manual tuning and opaque automated search.
Yongzhen Zhang, Xiaoyu Deng, Yifan He et al.· 0 citations
Expensive multiobjective optimization requires a diverse Pareto approximation under a severely limited evaluation budget. Existing surrogate-assisted algorithms often treat decision variables homogeneously and rely on either global or local models, causing inaccurate regional prediction or premature search concentration. This paper proposes HDS-LGSC, a hierarchical decision-space modeling and local–global surrogate coevolutionary algorithm. A lightweight graph-attention encoder learns variable interactions and partitions variables into global-dominant, locally coupled, and weakly related groups. A global radial-basis-function ensemble and local Gaussian processes guide cooperating populations, while an adaptive infill criterion combines hypervolume improvement, uncertainty, and decision-space novelty. Experiments on four benchmarks and a building energy-efficiency case demonstrate lower IGD, higher hypervolume, improved scalability, and robust prediction. Ablation results verify the complementary contributions of hierarchical modeling, dual surrogates, and composite infill selection.
Wenchao Pan, Junhan Li· 2026 8th International Confe...· 0 citations