Adaptive hyperparameter optimization in artificial neural networks using the bobcat optimization algorithm (BOA)

Optimization of Neural Networks Using Advanced Hyperparameter Tuning

Neural networks have become an essential tool for solving complex tasks in domains such as image recognition, natural language processing, and autonomous systems. However, the performance of neural networks heavily depends on the choice of hyperparameters such as learning rate, batch size, activation functions, and the number of hidden layers. Hyperparameter optimization, therefore, plays a critical role in enhancing model accuracy, convergence speed, and generalization capabilities. Traditional tuning methods like manual selection and grid search are often computationally expensive and suboptimal for large-scale networks. In this study, we explore advanced hyperparameter tuning strategies including Random Search, Bayesian Optimization, Hyperband, and Genetic Algorithms, evaluating their efficiency and effectiveness on various benchmark datasets. The research presents a comprehensive methodology integrating automated hyperparameter selection with neural network training, highlighting the trade-offs between computational cost and model performance. Experimental results demonstrate that optimized hyperparameters significantly improve the accuracy and stability of neural networks, reducing overfitting and training time. This paper also proposes a systematic framework for hyperparameter optimization that can guide practitioners and researchers in selecting optimal configurations tailored to their specific problem domains. By comparing the performance across different tuning strategies, we offer practical insights into the scalability and adaptability of neural network optimization techniques. The findings underscore the importance of leveraging intelligent hyperparameter optimization methods to advance deep learning applications and achieve superior performance in real-world scenarios.

Optimal tuning of Kolmogorov Arnold Network hyperparameters using Bayesian optimization

Evolutionary Training of Neural Networks: The Role of Crossover Operators in Genetic Algorithms Compared with Backpropagation

Training neural networks with gradient-based methods such as backpropagation is the dominant paradigm, but it depends on differentiable loss functions and is sensitive to initialization and local minima. Evolutionary algorithms offer a gradient-free alternative, yet the influence of their internal operators on training quality remains insufficiently characterized. This study presents a systematic comparison of backpropagation and ten variants of a genetic algorithm (GA) for training multi-layer perceptrons (MLPs), with particular focus on the role of crossover operators. The evaluation covers four MLP architectures and ten classification datasets from the UCI Machine Learning Repository, differing in sample size, dimensionality, and number of classes. Each configuration was assessed using stratified 4-fold cross-validation with 30 independent repetitions, and accuracy served as the primary performance metric, with macro-F1 reported to assess classifier behavior on class-imbalanced datasets. Backpropagation achieved higher mean accuracy than every GA variant on nine of the ten datasets, with the largest margins on high-dimensional problems. The genetic algorithm proved competitive on simpler, class-balanced datasets, where its better-performing variants matched the gradient-based baseline within one to two percentage points, and, on the Heart disease dataset, every GA variant reached a higher mean accuracy than backpropagation across all four architectures, though absolute performance remained modest on this five-class problem. Among crossover operators, BLX-α and BLX-α-β combined with tournament selection and a high crossover probability yielded the strongest configurations, while averaging crossover performed worst, as it restricts offspring to the midpoint of the parents and cannot explore beyond the range already present in the population. Tournament selection consistently led to higher mean accuracy than roulette-wheel selection, and shallow but moderately wide architectures, which encode fewer trainable parameters and thus a shorter chromosome, proved more amenable to evolutionary training than the two-layer alternative. These findings clarify when gradient-free training is competitive and which evolutionary operators drive its effectiveness.

Research on Neural Network Training Mechanism Integrating Convex Optimization and Backpropagation

This paper deeply integrates convex optimization theory with the backpropagation algorithm and constructs a novel stable and efficient training mechanism for neural networks that achieves favorable adaptability to both shallow fully connected networks and deep convolutional networks.

A Comparative Study of Bayesian Regularization Neural Network and Multilayer Perceptron Neural Network in the Hybridization of Self-Exciting Threshold Autoregressive Models

Artificial Neural Networks (ANNs), particularly Bayesian Regularization Neural Networks (BRNNs), have demonstrated strong capabilities for modeling complex nonlinear patterns in time-series data. This study explores the hybridization of Self-Exciting Threshold Autoregressive (SETAR) models with BRNN and Multilayer Perceptron Neural Networks (MLPNNs) to improve forecasting accuracy of nonlinear cyclical data, using the Canadian Lynx dataset. The hybrid models aim to capture regime shifts and nonlinear dynamics more effectively. Results show that the SETAR-BRNN hybrid outperforms individual models and other hybrids, achieving a Mean Absolute Percentage Error (MAPE) of 2.261%, representing a 45.8% reduction compared to the standalone SETAR model (MAPE = 4.17%) and a 14.7% reduction compared to BRNN alone (MAPE = 2.65%). Additionally, the SETAR-BRNN model reduces the Root Mean Square Error (RMSE) by 65.3% relative to the MLPNN-SETAR hybrid and suggest a forecasting performance across multiple horizons. The findings indicate that integrating BRNN into the SETAR framework significantly enhances predictive accuracy and model robustness for nonlinear, regime-dependent time series. This highlights the effectiveness of the SETAR-BRNN hybrid approach in accurately modeling complex cyclical behaviors and regime shifts in real-world data.

An Adaptive PSO–GA Hybrid Optimization with Diversity-Aware and Stagnation-Resistant Mathematical Framework for SVR

Support Vector Regression (SVR) is a reliable and generalizable model for non-linear regression, but its performance depends strongly on selecting proper hyperparameters such as the penalty parameter $C$, kernel width $\gamma$, and the $\epsilon$-insensitive loss. Because the SVR objective is highly nonconvex with a multimodal error surface, manual tuning and grid search become inefficient. Metaheuristic optimizers therefore offer a more practical alternative through population-based search. Although Particle Swarm Optimization (PSO) and Genetic Algorithms (GA) have both shown promise, each has clear limitations: PSO converges quickly but often stagnates in local optima due to loss of diversity, whereas GA preserves diversity but converges slowly because of excessive random exploration. To address these complementary weaknesses, this work introduces a mathematically formulated hybrid framework called Adaptive PSO-GA (A-PSO-GA). Experiments on the Housing, Concrete, and Abalone benchmarks show that A-PSO-GA consistently outperforms PSO-only, GA-only, and non-adaptive hybrid baselines in RMSE, MAE, MAPE, and $R^{2}$. It also achieves faster and more stable convergence across multiple seeds, demonstrating improved prediction accuracy, robustness, and optimization reliability. These results confirm that mathematically justified adaptive mechanisms are important for effective evolutionary hyperparameter tuning of SVR.