The next iteration of Nova is presented, an automated end-to-end JIT compiler that achieves absolute control over hardware mapping by synthesizing fine-grained kernels directly from the computation's structure by extending Nova's compilation pipeline to natively support full Transformer architectures.
Abstract
The performance of deep learning models at scale relies heavily on how effectively high-level mathematical operations are mapped to underlying physical hardware. While high-level tensor frameworks provide flexible abstractions, their execution models inherently lack the whole-graph visibility required to maximize hardware utilization, often forcing a reliance on opaque, hand-written kernel libraries for complex operations like Attention. To bridge this gap, we present the next iteration of Nova, an automated end-to-end JIT compiler that achieves absolute control over hardware mapping by synthesizing fine-grained kernels directly from the computation's structure. In this work, we extend Nova's compilation pipeline to natively support full Transformer architectures. By capturing eager executions and unifying forward and backward passes into a single value-semantic dialect, Nova unlocks aggressive whole-graph optimizations. Rather than relying on rigid, pre-compiled library calls, Nova focuses on extensive cross-operator fusions, collapsing complex causal attention sub-graphs, element-wise operations, and memory-bound normalizations directly into single fused kernels to drastically reduce global memory roundtrips. In our evaluations training a full GPT-2 architecture on Ada 6000 GPUs, Nova demonstrates superior end-to-end throughput, averaging 441K tokens/second compared to 406K for our own eager execution and 405K for torch.compile. By drastically reducing memory-bound overheads through compiler-native fusion, Nova enables efficient full LLM compilation on modern hardware while strictly maintaining numerical parity.
MOCHA introduces two key ideas: a validity prediction model to filter out invalid configurations prior to expensive hardware profiling, and an advanced performance prediction model that leverages hidden features extracted during the compilation process to enhance autotuning efficiency and robustness.
Joohyoung Cha, Munyoung Lee, Jinse Kwon et al.· ACM Transactions on Embedded...· 0 citations
Apple's M5 generation introduces a redesigned GPU architecture in which every core carries a dedicated Neural Accelerator: on-die matrix units exposed through the Metal~4 tensor API. We show that BaseRT, our native Metal inference runtime for large language models on Apple Silicon, exploits these units to push inference throughput on Apple hardware substantially beyond both llama.cpp and MLX. Building on BaseRT's framework-free design, we add a family of hand-written Metal~4 tensor-core kernels (including dense and mixture-of-experts GEMM and flash-attention prefill kernels) that route the compute-bound matrix multiplications of inference through the M5 Neural Accelerators while leaving the memory-bound decode path on our existing specialised kernels. On an Apple M5 Pro, across fifteen model configurations spanning the Qwen3, Qwen3.5/3.6, Llama~3.2, and Gemma~4 families from sub-1B to 35B parameters, BaseRT delivers up to $6.4\times$ higher prompt-processing throughput than llama.cpp and $3.9\times$ higher than MLX, with the largest margins on the mixture-of-experts models where matrix multiplication dominates, while maintaining its lead on decode of up to $1.75\times$ over llama.cpp and $1.33\times$ over MLX. These results establish a new performance ceiling for on-device LLM inference and show that the M5's tensor cores are the decisive lever for prompt processing on Apple Silicon. BaseRT is publicly available at https://github.com/basecompute/baseRT.
Deep learning (DL) workloads span multiple layers of abstraction, from high-level Python scripts to intermediate C/C++ tensor libraries and low-level GPU compute kernels. While this layered design enables flexibility, it also obscures the root causes of inefficiencies, leaving GPUs underutilized. Profilers are essential for exposing such inefficiencies, yet existing tools fall short of explaining how high-level DL semantics translate into low-level GPU kernel and hardware behavior. They either offer DL operator-level statistics (e.g., for a convolution operator) with coarse-grained metrics such as execution time and memory usage, or focus exclusively on individual GPU kernels without linking them to DL semantics. As a result, true performance bottlenecks remain hard to diagnose and fix. We present TenProf, the first tensor-centric GPU profiler for PyTorch-based DL workloads. TenProf systematically bridges low-level hardware stall metrics with high-level DL semantics (particularly tensor transformations) through three key tensor-centric analyses: (i) tracing tensor transformations and modeling them as a forest, (ii) attributing GPU memory accesses to both base and view tensors, and (iii) assessing tensor access efficiency via hardware stall attribution. By unifying insights across layers, TenProf reveals how tensor accesses and transformations directly impact performance. It operates on unmodified program binaries, introduces moderate overhead, and generates interpretable profiles that guide actionable optimizations. Evaluation across several real-world DL applications shows that TenProf uncovers inefficiencies such as non-contiguous memory accesses, where resolving them requires only minor code changes yet yields non-trivial speedups. Several optimizations guided by TenProf have already been validated and upstreamed by application developers, demonstrating its practical value in production settings.
Xingjian Ding, Keren Zhou, Yueming Hao et al.· International Conference on...· 0 citations
The design of transformer-based Large Language Models (LLMs) is being radically changed through new architectures that are able to overcome scalability limitations of previous designs, including Mixture-of-Experts (MoE), Multi-Head Latent Attention (MLA), and Multi-Token Prediction (MTP). As an open-weighted model released at the end of 2024, which has both state of the art architectural transparency and production scale efficiency, DeepSeeek-V3 represents the ultimate testing ground for investigating these modern technologies. This paper provides a comprehensive analysis of the architectural structure of DeepSeek-V3 based upon information from the DeepSeek-V3 Technical Report, industry benchmarking data and independent latency testing, to demonstrate how various techniques can be used to optimize training while still providing competitive performance in code generation and mathematical reasoning. In addition, latency testing conducted on a Distilled version of DeepSeek-V3, with approximately 14 billion parameters, running on a T4 GPU, reveals that although significant improvements have been made in optimizing latency there remains substantial barriers to deploying these models. Through this context, this research will serve as a reference document for practitioners and researchers who wish to understand current trends and challenges in increasing accessibility to high performance AI models.
Yassine Zouhdi, B. Hdioud· EPJ Web of Conferences· 0 citations
An MLIR (Multi-Level Intermediate Representation) based compilation method for large language models, illustrated using two dialects of operators, TopOp and TpuOp, supporting a variety of generative models including the Qwen, Llama, InternVL, and MiniCPM-V series.
Pengchao Hu, Zhibin Xin, Yifan Chen et al.· 0 citations
Deep learning (DL) compilers such as Apache TVM translate high-level models into optimized low-level code through multi-stage compilation pipelines. While recent testing efforts have improved fuzzing of optimization stages, they still face two key challenges: (i) the lack of semantics-preserving test models, leading to low validity, and (ii) coarse-grained input generation that fails to trigger hard-to-reach compiler components. To address these limitations, we propose CovCraft, a unified testing framework that integrates constraint-driven model generation with large language model (LLM)-guided input adaptation. CovCraft constructs diverse and valid ONNX models via symbolic constraint encoding and SMT solving, and then iteratively refines inputs using LLM-guided prompts to target uncovered functions, enabling the activation of rarely executed code paths. We evaluate CovCraft on TVM and observe consistent improvements over state-of-the-art techniques: it increases branch and function coverage by 8.9% and 7.0%, respectively, and detects 8 bugs. Moreover, the LLM-guided component achieves an 83.75% success rate in covering designated target functions, demonstrating the effectiveness of combining constraint-based generation with adaptive LLM reasoning for DL compiler testing. The prototype implementation of CovCraft is publicly available at: https://github.com/duduhedangdang/CovCraft.
Yifei He, Fangyu Yang, Ting Su et al.· Annual International Compute...· 0 citations
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
MIT News · Artificial Intelligence· news.mit.eduAug 24, 2026
