LLM-MetaAS: A Semantic-Statistical Policy Routing Framework for AutoML Execution Strategy Selection

Abstract

Automated machine learning (AutoML) aims to reduce manual intervention in machine learning pipeline construction. However, AutoML performance is affected not only by the searched pipeline but also by high-level execution policies, such as validation protocols and budget allocation strategies. Auto-sklearn 2.0 has shown that these policies can be selected automatically, but its lightweight task representation mainly captures dataset scale and may not distinguish tasks with similar sizes but different semantic and structural characteristics. To address this limitation, this article proposes LLM-MetaAS, a semantic-statistical framework for AutoML execution policy selection. Building on the established pairwise policy-selection paradigm, LLM-MetaAS focuses on improving task representation rather than introducing pairwise decomposition itself. It constructs a policy-oriented fingerprint by combining LLM-assisted semantic profiling with lightweight statistical descriptors related to validation reliability and computational demand. Policy quality is evaluated using an explicit regret-based criterion relative to the empirical oracle, and vote margins are used to analyze routing uncertainty. Experiments on 39 benchmark tabular classification datasets show that LLM-MetaAS improves overall AutoML performance and selects policies closer to the oracle than fixed strategies, random selection, and the native Auto-sklearn 2.0 selector. Ablation and robustness analyses further support the utility of the complete semantic-statistical representation within the evaluated framework.