MOCHA introduces two key ideas: a validity prediction model to filter out invalid configurations prior to expensive hardware profiling, and an advanced performance prediction model that leverages hidden features extracted during the compilation process to enhance autotuning efficiency and robustness.
Abstract
The growing complexity of deep learning models necessitates specialized hardware and software optimizations, across diverse computing platforms, spanning from general-purpose CPUs and GPUs to domain-specific accelerators. While machine learning-based autotuning methods have emerged as a promising solution to reduce manual effort, existing template-based and template-free approaches often suffer from prolonged tuning times due to the extensive profiling of invalid configurations, which may result in runtime errors. To address this inefficiency, we propose MOCHA, a multi-level optimization with compiler for hardware acceleration framework designed for robustness and generality. MOCHA introduces two key ideas: (1) a validity prediction model to filter out invalid configurations prior to expensive hardware profiling, and (2) an advanced performance prediction model that leverages hidden features extracted during the compilation process. We evaluated MOCHA on two distinct hardware backends: a customized VTA accelerator, and an NVIDIA T4 GPU. Experimental results show that MOCHA on the VTA achieves equivalent performance improvements using only 12.3% of the samples required by a TVM-like approach and reduces the invalidity ratio during autotuning by an average of 66.4%. Similarly, on the NVIDIA T4 GPU, MOCHA demonstrates its superior efficiency by reaching the performance plateau using only 87.1% of the samples required by the baseline TVM autotuner while reducing the invalidity ratio during autotuning by 62.2%. These results highlight the potential of MOCHA to enhance autotuning efficiency and robustness across diverse hardware platforms by effectively filtering out invalid configurations and focusing the search on high-performance regions.
The rise of heterogeneous computing systems has intensified the need for performance-portable programming models and effective autotuning methodologies. Although compiler and runtime tuning are known to significantly influence application performance, it remains unclear how such optimizations transfer across different hardware architectures, particularly within single-source models such as SYCL. This work investigates the transferability of compile-time and runtime autotuning decisions across CPUs and GPUs, focusing on AdaptiveCpp, a SYCL implementation built on LLVM. We introduce an automated framework that jointly explores compiler flags and runtime parameters using both Bayesian optimization and a tabu-search–based strategy. The tool orchestrates compilation, execution, and measurement while also providing statistical attribution via ridge regression to quantify the impact of individual tuning parameters. Through an extensive evaluation of CPUs and GPUs from multiple vendors, we demonstrate that autotuning can deliver substantial performance gains—up to 3 × on CPUs—yet the influence of specific compiler flags often diverges across different architectures. For example, flags such as -fno-builtin yield large improvements on CPUs but have negligible effect on GPUs. We also demonstrate that runtime-level choices, such as thread-placement policies, can significantly affect performance on CPUs. Our findings highlight the challenges and opportunities of autotuning in heterogeneous, single-source programming ecosystems. They also underline the importance of architecture-aware autotuning strategies and motivate further exploration of cross-device performance modeling.
Hari Abram, Nikela Papadopoulou, Jens Domke et al.· International Conference on...· 0 citations
The next iteration of Nova is presented, an automated end-to-end JIT compiler that achieves absolute control over hardware mapping by synthesizing fine-grained kernels directly from the computation's structure by extending Nova's compilation pipeline to natively support full Transformer architectures.
Adwaid Suresh, Aparna A. Harshini, Jona Delcy et al.· 0 citations
Domain-Specific Accelerators (DSAs) have become a key driver of performance and efficiency improvements in the post-Moore's Law era. These improvements stem from specializing the hardware for domain workloads and exploiting the workloads' inherent parallelism. In the domain of Deep Learning (DL), workloads (models) comprise multiple operations, known as layers, that exhibit parallelism opportunities and diverse computational characteristics. Consequently, DSAs with multiple computational units (engines) provide a natural architectural paradigm to fully exploit the specialization and parallelism potential inherent in such multi-layered models.Multi-engine DL accelerators generally fall into two categories: model-specific and flexible. Model-specific accelerators are co-designed to efficiently execute one or a few closely related models. Flexible accelerators, by contrast, are designed to support a broad range of DL workloads. Designing and implementing accelerators in either category that fully exploit specialization and parallelism, and thus optimize performance and efficiency, requires systematic exploration based on quantitative evaluation of design alternatives. Existing multi-engine DL accelerator design approaches range from intuition-driven to exploration-based methodologies. However, even the latter typically leave key architectural parameters unexplored by fixing them a priori based on expert knowledge and intuition. In many cases, the fixed parameters are more consequential for accelerator specialization and parallelism than the explored parameters. Consequently, the full potential of the multi-engine paradigm is often left unexploited.To fully exploit the potential of multi-engine DL accelerators, this thesis presents design methodologies that increase the extent to which key design choices are based on exploration and quantitative evaluation of design alternatives. The first contribution of this thesis is the Fixed Budget Hybrid CNN Accelerator (FiBHA). FiBHA proposes a hybrid, model-specific, multi-engine architecture and an accompanying design methodology. FiBHA targets a specific class of DL models and relies primarily on empirical analysis to exploit opportunities for specialization and parallelism. To expand the scope and co-design accelerators for a broader class of models, the work moved to a more analytical approach. The second contribution of this thesis comprises MCCM, a fast analytical cost model for evaluating model-specific multi-engine accelerators, and MCExplorer, a design space exploration framework built upon it. Together, they enable orders-of-magnitude faster evaluation and systematic exploration of model-specific multi-engine accelerator designs. Unlike existing approaches that rely on predefined design choices, MCExplorer quantitatively evaluates alternative architectural configurations across a broader design space. To further expand the scope, the work extends to flexible, in addition to model-specific, multi-engine accelerators. The third contribution of the thesis is a design methodology for flexible multi-engine DL accelerators, termed MEDEM. To support a wide range of diverse DL workloads, a flexible multi-engine accelerator must have an engine combination with complementary capabilities to ensure that different layers across these diverse workloads are processed efficiently. Existing work builds flexible multi-engine accelerators by combining expert-selected, independently optimized engines. However, independently optimized engines may perform best on largely overlapping subsets of workloads, and thus their combination does not necessarily improve overall workload coverage. MEDEM presents an alternative design methodology where the engines are co-designed, then curated to find a combination that maximizes the coverage of diverse workloads.Using a systematic approach based on modeling and quantitative evaluation of a wider space of design alternatives, the proposed methodologies identify accelerator architectures that better exploit the specialization and parallelism inherent in DL workloads. This applies to both model-specific accelerators, as FiBHA, MCCM, and MCExplorer demonstrate, and to flexible ones, as MEDEM shows. As a result, these methodologies identify architectures that consistently outperform the state-of-the-art, achieving considerable improvements in latency, throughput, energy, and energy-delay product (EDP).
Traditionally, GPU kernels have been developed and optimized within vendor-specific programming models to achieve high performance, resulting in software that is difficult to optimize and adapt across increasingly heterogeneous computing systems. Hardware-agnostic programming models offer a more sustainable approach to GPU software development by improving portability and maintainability, but achieving efficient execution across diverse architectures remains challenging. We address this challenge by integrating auto-tuning into hardware-agnostic GPU kernels written in Julia. We rebuild the established Kernel Tuner auto-tuning framework with Julia support, enabling systematic exploration of kernel configurations for hardware-agnostic GPU kernels targeting NVIDIA, AMD, Intel, and Apple GPUs. We demonstrate this approach on hardware-agnostic singular value decomposition (SVD) as implemented in the NextLA.jl linear algebra library. The results show that auto-tuning is essential for creating resource-efficient hardware-agnostic GPU kernels across a variety of hardware. Optimal configurations improve kernel performance by a factor of 3x to 7x compared to median parameter configurations, demonstrating the substantial impact of tuning on efficient hardware utilization.
Floris-Jan Willemsen, Evelyne Ringoot, Alan Edelman· 0 citations
Deploying deep neural networks on resource-constrained hardware relies on mixed-precision quantisation (MPQ). current deployment toolchains severely fragment this process. Quantisation typically occurs as a hardware-agnostic preprocessing step in front-end frameworks, disconnected from the downstream compilers that generate the physical machine code. This separation leads to suboptimal configurations where assigned bit-widths map poorly to the target machine's heterogeneous hardware execution blocks such as tensor cores and variable-width vector units, incurring severe runtime execution penalties. Furthermore, evaluating these configurations via exhaustive hardware-in-the-loop (HIL) testing is intractable due to the exponentially large search space. We present CONQuER, a unified compiler-integrated infrastructure for hardware-aware MPQ. CONQuER shifts quantisation into the compiler pipeline at the TOSA level, enabling intelligent configuration handling based on compiler support. To evaluate this combinatorial search space of different of model layers within practical compilation budgets, CONQuER couples an NSGA-II evolutionary algorithm with a dual-surrogate prescreening engine. This engine evaluates theoretical cache memory bounds and feature space isotropy to discard non-viable configurations. CONQuER then executes only the strongest candidate policies on hardware via IREE, feeding the execution metrics into an online calibrator. This calibrator aligns the surrogate models with the true hardware behaviour during an NSGA-II evolutionary search. Evaluation across mobile and laptop CPUs, and server GPUs demonstrates that optimal quantisation policies are hardware-dependent. By coupling quantisation with compiler lowering and physical execution, CONQuER discovers Pareto-optimal configurations up to 12.19x faster inference with top-1 accuracy within 1.44% of the unquantised baseline.
PTXBench provides an auditable testbed for measuring and improving LLMs'ability to exploit evolving GPU architectures, and shows that architecture-specific PTX capability remains uneven.
Genghan Zhang, Yixin Dong, Chengze Fan et al.· 0 citations