TL;DR
It is demonstrated that network architecture and its coeficients can be learned together by unifying concepts of evolutionary search within a population based traditional training process.
Common practice when training Convolutional Neural Networks (CNNs) is to utilize randomly shuffled mini-batches. During this stage of the training process, two limitations are faced. Firstly the slowing down of convergence, and secondly the limited learning signal contribution due to large quantities of samples that are considered easy while training CNNs.In our work, we aim to address these inefficiencies by proposing A*-Inspired Batch Selection (A*-BS). This is a lightweight and model-agnostic training strategy that formulates mini-batch scheduling as a heuristic search problem. In this type of batch selection process each batch is treated as a node in a search space and ranked using an A*-like score by combining a loss-based difficulty measure with a reuse penalty. The two main benefits of this method are: informative gradient updates encouragement and batch diversity selection throughout the training process. The proposed method does not modify network architectures or optimization algorithms and can be seamlessly integrated into existing training pipelines. We evaluate the approach on the twelve 2D classification tasks of the MedMNIST-v2 benchmark, using a deliberately simple architecture of approximately 2.25 × 105 parameters and comparing against the ResNet-18 and ResNet-50 baselines reported by the benchmark.On half of these tasks, the lightweight model combined with A*-BS reaches higher accuracy and AUC than both ResNet baselines, with relative gains of up to 15%. Moreover, an ablation under identical CNN architecture and hyper-parameters shows that A*-BS outperforms random batch shuffling on all twelve MedMNIST tasks. Wall-clock measurements further show that the lightweight CNN with A*-BS trains substantially faster than ResNet-18 and ResNet-50 on identical hardware.Indicative in these results is that intelligent batch ordering can partially compensate for reduced architectural complexity, thus offering a computationally efficient alternative to deeper models. Furthermore, the reliability of the proposed strategy is reinforced by its strong performance with a simple CNN, even when compared to deeper and more sophisticated architectures.
Hyperparameter optimization is a critical determinant of the performance and generalizability of convolutional neural networks (CNNs), yet selecting optimal configurations remains a complex challenge due to the large, non-linear search space. While numerous optimization techniques have been explored in the literature, most comparative studies have relied on benchmark datasets, limiting insights into their practical effectiveness under real-world constraints. This study evaluates four widely used hyperparameter optimization methods (i.e., Grid Search, Optuna, Hyperopt, and DEAP) across five CNN architectures (i.e., AlexNet, GoogLeNet, LeNet, MobileNet, and ResNet18) under two distinct data regimes: the Modified National Institute of Standards and Technology (MNIST) benchmark and a custom SkinCancer image dataset. On MNIST, all configurations achieved strong classification performance, with test accuracies ranging from 95.45% to 97.61%, led by ResNet18 optimized via Grid Search (97.61%). In this simpler setting, DEAP provided the shortest optimization runtimes, while Optuna and Hyperopt offered an effective balance between performance and computational cost. In contrast, the SkinCancer dataset exhibited substantially greater variability in performance, highlighting the stronger influence of both model architecture and optimization strategy. ResNet18 achieved the highest overall test accuracy (81.50%) when optimized with Hyperopt. Although no single optimization method consistently outperformed all others, adaptive approaches frequently achieved competitive or superior predictive performance while requiring considerably less computation than exhaustive search. These findings demonstrate that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.
This work proposes an NC-inspired training framework for simplifying deep networks during training, monitoring representation dynamics through the Inverse Fisher Criterion to identify both the split point between feature extraction and classification and the training stage at which simplification becomes viable.
The results demonstrate that transfer learning significantly improves convergence speed, generalization, and computational efficiency, making it a promising approach for AI applications across domains such as healthcare, NLP, and autonomous systems.
Deep Neural Networks (DNNs) have achieved remarkable success across diverse machine learning applications, yet designing effective architectures remains a laborious, expert-driven process. Neural Architecture Search (NAS) was introduced to automate this process, with Evolutionary NAS (ENAS) emerging as one of the most effective and widely adopted NAS paradigms. This survey provides a comprehensive and systematic review of 164 ENAS studies published between 2020 and 2024, categorized according to the specific evolutionary algorithm employed as the search strategy. Unlike prior surveys—which either treat evolutionary methods at a high level or focus on general NAS pipelines—this study is, to the best of our knowledge, the first to perform an in-depth, algorithm-level breakdown of ENAS research, covering evolutionary strategies (ES), genetic algorithms (GA) and their derivatives, Particle Swarm Optimization (PSO), Differential Evolution (DE), Quantum-Inspired Evolutionary Algorithms (QIEA), and other heuristics. Our quantitative analysis reveals that ES is the dominant paradigm (45.7% of studies), followed by GA (29.9%), with annual publication counts growing 6.6-fold from 2020 to 2024—reflecting a rapidly expanding field. Key findings include: (i) surrogate-assisted methods have grown substantially in 2023--2024, significantly reducing search cost; (ii) GA-based multiobjective approaches (NSGA-II, NSGA-III) dominate Pareto-optimal architecture search; (iii) hybridization of evolutionary search with gradient descent and predictor networks is an accelerating trend; and (iv) image classification on CIFAR-10/100 and NAS-Bench-201 benchmarks accounts for the majority of evaluations, highlighting opportunities in underexplored domains. This survey also provides a critical comparison of encoding strategies (fixed-length vs. variable-length) and an analysis of search space complexity across reviewed methods. We believe this work serves as a practical guide for researchers seeking to understand, select, and advance evolutionary strategies for automated neural architecture design.
This work proposes a mechanism that bridges these two paradigms: an LLM produces a high-quality seed architecture, then decomposes it into a "slotted architecture", a scaffold with named, interchangeable module slots that automatically defines a bounded, task-specific search space for conventional NAS to explore, without manual engineering.