EFE demonstrates that LLM-based evolution can improve both accuracy and interpretability when automatically tackling structured data, and is found to be particularly effective on classical decision trees.
Abstract
Large language models are increasingly used as open-ended search operators in evolutionary optimization. We introduce Evolutionary Feature Engineering (EFE), a framework for using LLM-based evolution to discover preprocessing transformations for structured data. EFE represents transformations as Python programs with a standardized fit/transform interface, allowing them to be inserted directly into existing machine learning pipelines. During evolution, candidate programs are refined using dataset context, summary statistics, and downstream performance feedback on validation set. We instantiate EFE in two settings. For time-series forecasting, EFE-Time learns invertible, dataset-specific normalizations that improve off-the-shelf time-series foundation models. It reduces forecasting errors (MASE, WQL, MAE) 3% or more when averaged across datasets and improvements are as much as 19% on the COVID-Deaths dataset. Notably, these improvements occur with recent TSFMs such as Chronos-2. For tabular prediction, EFE-Tab evolves compact feature programs that add useful interpretable features and remove redundant ones, improving or matching existing LLM-based feature-engineering methods. We found EFE-Tab to be particularly effective on classical decision trees, where small sets of evolved features yield competitive accuracy while preserving interpretability. Overall, EFE demonstrates that LLM-based evolution can improve both accuracy and interpretability when automatically tackling structured data.
Experimental evaluation on the Madelon dataset indicates that the automated and interpretable pipeline performs comparably to, and in some respects favourably against, baseline feature engineering approaches, demonstrating the practical effectiveness of combining scalable feature generation with interpretable AutoML.
Ms. CH. Vasavi, Ms. SK. Raqeeba· International Journal for Re...· 0 citations
Structured data drives enterprise decision-making, yet building predictive pipelines for time-series and tabular modalities requires intensive feature engineering, model selection, and other ''tricks of the trade''. AutoGluon, an open source AutoML system, automates this through multi-layer stack ensembling, providing a unified API that reliably achieves state-of-the-art accuracy across time series and tabular data. Concurrently, Foundation Models for Structured Data (FM4SD) have emerged to push the boundaries of predictive performance, enabling both powerful zero-shot inference and efficient fine-tuning. Specifically, Chronos-2 delivers state-of-the-art multivariate forecasting for time series, while a rapidly growing ecosystem of tabular foundation models (e.g., TabPFN, Mitra, TabICL) transforms how practitioners execute classification and regression tasks. This system-focused tutorial teaches attendees how to operationalize FM4SDs using AutoML across temporal and tabular modalities, with an additional highlight on the newly released Chronos-2. We guide participants through a technical progression: establishing AutoGluon as the orchestration framework, detailing the internal mechanics of Chronos-2 and various tabular FMs, and unifying them into production-ready pipelines.
Boran Han, Nick Erickson, Abdul Fatir Ansari et al.· Proceedings of the 32nd ACM...· 0 citations
This work presents LACE, an AutoML framework that instead searches over complete executable pipeline programs: an evolutionary loop maintains a population of scikit-learn-compatible Python classes, and a large language model acts as the variation operator.
Sofoklis Kitharidis, C. Veenman, J. V. Rijn et al.· 0 citations
Evaluating the performance of machine learning workflows is a major computational bottleneck in automated machine learning (AutoML), particularly for complex pipelines involving preprocessing, model selection, and hyperparameter optimization. This work aims to develop an efficient performance prediction framework that estimates the expected accuracy of candidate machine learning workflows on unseen datasets without requiring explicit model training. We formulate performance prediction as a meta-learning regression problem that leverages historical experimental results from the OpenML platform. Machine learning workflows are represented as structured pipelines and encoded using text-based vectorization techniques, including TF-IDF, count-based, and hashing vectorizers, as well as LLM BERT embeddings. These workflow descriptors are combined with dataset-level meta-features capturing basic statistical and structural properties. Several regression models are evaluated as meta-learners, including linear models, decision trees, random forests, Gaussian processes, and gradient-boosted decision trees. The approach is systematically evaluated on the OpenML-CC18 benchmark suite using cross-validation over more than 100,000 workflow-dataset evaluations. The proposed framework achieves strong predictive performance across a wide range of workflows and datasets. In particular, gradient-boosted decision tree regressors combined with standard TF-IDF representations of workflows consistently yield the best results, reaching an average coefficient of determination \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R^2$$\end{document} of approximately 0.8 on unseen test data. While transformer-based MiniLM embeddings were evaluated, they did not consistently outperform sparse TF-IDF representations and incurred higher computational cost. Feature ablation studies indicate that restricting vocabulary size degrades performance, while extending representations with bigrams provides only marginal gains at substantially higher computational cost. The results demonstrate robust generalization across heterogeneous workflows and dataset characteristics. This study shows that accurate performance prediction for classical machine learning workflows can be achieved through meta-learning using readily available OpenML meta-data. The proposed approach enables rapid and computationally efficient estimation of workflow performance, making it well suited for accelerating AutoML search and model selection. The results indicate that careful selection of regression models is more critical than increased representational complexity, with simple and scalable workflow encodings yielding the most robust performance. Given its scalability and flexibility, the framework provides a strong foundation for future extensions incorporating richer dataset descriptors, larger meta-datasets, and more expressive embedding and regression models.
Roman Neruda, Juan Carlos Figueroa–García, Carlos Franco· SN Computer Science· 0 citations
This work proposes a general LES framework that incorporates feature-assisted niche construction within abstract search spaces, enabling the seamless integration of niche-based search strategies from evolutionary computation and introduces PartEvo (Partition to Evolve), an LES method that combines niche collaborative search and advanced prompting strategies to improve algorithm discovery efficiency.
Qinglong Hu, Qingfu Zhang· Neural Information Processin...· 10 citations· ⚡3
EvoMem is introduced, a persistent memory architecture for LLM-based evolutionary program search that captures and reuses candidate mutation knowledge and provides evidence that persistent memory can reduce some redundant exploration and improve the reuse and adaptation of successful strategies in LLM-driven evolutionary search.
Viktor Volkov, V. Khrulkov, Andrey V. Galichin et al.· 0 citations