Recent advances in generative AI have significantly increased the demand for high-resolution image and video generation, positioning diffusion models as a core technology. Among them, Diffusion Transformers (DiTs) have emerged as the state-of-the-art (SOTA) models due to their scalability and output quality. However, self-attention in DiTs incurs significant computational overhead, leading to excessively long latency as the complexity grows with the fourth power of the output resolution. While prior works have attempted to mitigate this cost using sparsity and quantization techniques, they fall short of effectively reducing the computational cost in high-resolution DiTs. In this paper, we present APT, a software-hardware co-designed accelerator for high-resolution DiTs. APT leverages attention probabilities as a unified importance metric to jointly optimize computation through fine-grained pruning and adaptive precision scaling. At the algorithm level, we propose Attention Probability-guided Adaptive Dual Thresholding (APDT), which dynamically performs element selection and precision assignment using dual thresholds. To ensure compatibility with memory-efficient FlashAttention, we introduce Timestep-Aware FlashAttention (TAFA), which predicts attention probabilities across timesteps by exploiting temporal similarity. At the architecture level, we co-design a specialized accelerator that efficiently supports irregular sparsity and dual-precision execution, featuring dynamic mask management, address translation, dual-precision compute units, and a tile-based dataflow. Finally, we evaluate APT on SOTA DiT models, including PixArt-$\alpha$, Stable Diffusion 3, and FLUX. APT achieves up to 8.16$\times$ speedup and 14.98$\times$ higher energy efficiency over NVIDIA A100, and up to 3.01$\times$ speedup and 2.04$\times$ higher energy efficiency over EXION, a SOTA diffusion model accelerator.
Sungyeob Yoo, Seeyeon Kim, Joonyong Park et al.· 0 citations
On edge devices, convolutional neural network (CNN) inference is bottlenecked mainly by memory bandwidth, owing to the frequent memory accesses to feature maps and parameters. To address this challenge, we propose a memory-efficient hardware accelerator for depthwise separable convolution that minimizes off-chip memory traffic and parameter storage. The proposed architecture employs three key techniques: (1) a 64-bit run-length coding (RLC) packet compression that exploits feature-map sparsity after ReLU, (2) a mixed-precision scheme that represents feature maps and weights at different precisions, and (3) separated depthwise and pointwise convolution units. In particular, feature maps are transferred in RLC-compressed form, which reduces the amount of data exchanged with the host. The compressed data are decoded row by row, so the on-chip buffers hold only the rows required for computation rather than a complete feature map. In software simulation on ImageNet, the mixed-precision scheme reduced the parameter storage by 49.22% at a cost of 6.24 percentage points (pp) in Top-1 accuracy, and the RLC reduced the data by up to 56.07% in the deeper layers. Implemented on a Xilinx ZCU-104 FPGA, the proposed accelerator performs the depthwise separable convolution with a small number of logic resources and on-chip memory, confirming its feasibility for resource-constrained edge devices.
Jaeseong Kim, Taehong Min, Chaebin Lee et al.· Electronics· 0 citations
Diffusion Transformers (DiTs) have been widely used in many tasks, including image synthesis, video generation, and content editing. However, their multi-iteration inference process leads to performance inefficiency and high energy consumption. Existing acceleration methods primarily focus on reducing temporal redundancy between adjacent timesteps, but often overlook the specific features of DiTs. As a result, these approaches either suffer from great accuracy degradation or fail to achieve high efficiency. We present DSTAR, a software-hardware co-design framework that accelerates DiT inference by reducing spatial and temporal redundancy. At the algorithmic level, DSTAR introduces a fine-grained mixed-precision quantization method for differential activations in linear operations, significantly increasing the proportion of low-bit computations. Additionally, DSTAR incorporates a sparse attention reuse mechanism to minimize redundant computation in attention layers. For architectural support, we design a specialized hardware accelerator which achieves high efficiency in both latency and energy consumption. Evaluation on seven typical DiTs demonstrates that DSTAR achieves up to 7.33x latency speedup and 41.89x energy savings compared to an NVIDIA A100 GPU, and achieves up to 2.54x latency speedup and 3.68x energy savings compared to SOTA accelerators, without accuracy degradation.
Chi Zhang, Jieru Zhao, Yu Feng et al.· 0 citations
Deep learning for solving partial differential equations (PDEs) has become increasingly prominent. The Fourier Neural Operator (FNO) architecture has been proven to be an efficient and high-precision method that is widely used in scientific research. However, FNO incurs significant overhead by increasing the scale and dimensionality of practical problems. The insufficient utilization of hardware resources in its key operations reduces the computational efficiency of FNO solvers in high-resolution and time-sensitive problem scenarios, and cannot provide effective solution capabilities. To address the latency induced by low computational resource utilization and large-scale data access and computation, we propose FNO-Speed, an integrated solution incorporating the multi-level parallel FNO-aware mapping and tiling GEMM optimization strategy and the custom-sized high-frequency signal filtering scheme. FNO-Speed effectively leverages the data characteristics of FNO layers and the GPU hierarchical structure to adopt a data tiling and partitioning strategy, implementing matrix multiplication based on vector outer products and operator fusion to replace convolution. It also adopts a data reorganization scheme and computation restructuring to address fragmented memory access operations and serial einsum in frequency-domain. The FNO-Speed optimization strategy enhances the utilization of device memory bandwidth and computational efficiency and achieves significant acceleration in both 2D and 3D problem scenarios while maintaining nearly identical accuracy. The model achieves up to 1.4 × end-to-end training speedup, and the parallel efficiency achieves around 70% on 4 GPUs, fully demonstrating the effectiveness of the FNO-Speed optimization strategy in improving FNO performance.
Tao Song, Ouyang Ying, Xiangyu Meng et al.· ACM Transactions on Architec...· 0 citations
Diffusion models are widely used to generate high-quality images and videos, but their iterative denoising process remains computationally intensive. A growing class of training-free accelerators reduces this cost by reusing cached intermediate features or forecasting future ones. To control draft drift, these methods sometimes compute an exact block feature for verification. Yet the resulting exact feature is typically used only to measure discrepancy or guide a later decision and is then discarded. We find that this previously computed feature can instead be reused for correction. Forwarding it at the verification site resets the local draft residual and reduces downstream feature error. Based on this observation, we introduce FeatFix, a local exact-feature correction method for cached diffusion inference. FeatFix operates at a fixed sparse set of layer--timestep sites. At each selected site, it replaces the complete draft block output with the exact output computed from the same incoming state, avoiding token- or channel-level partial replacement and full-timestep recomputation. Experiments across four image and video backbones show that FeatFix consistently accelerates generation, achieving a speedup of up to $6.70\times$ over Vanilla while maintaining competitive output quality.
Hanshuai Cui, Zhiqing Tang, Z. Yao et al.· 0 citations
Wavelet convolution (WTConv) has emerged as an increasingly popular drop-in replacement for standard convolutions, expanding a network's receptive field exponentially with the number of decomposition levels while keeping the parameter count linear. However, its reference implementation is severely memory-bound due to excessive data movement through high-bandwidth memory (HBM). We develop an I/O model of WTConv to characterize this bottleneck and use it to guide three algebraic reformulations: (1) recomputing the inexpensive Haar analysis butterfly on chip, (2) collapsing the multi-level synthesis cascade into a single closed-form pass indexed by output-coordinate bits, and (3) folding learned per-channel scales into the convolution weights. Together, these reformulations enable an I/O-aware fused implementation that substantially reduces HBM traffic. We evaluate the WTConvNeXt configuration across decomposition levels and a broad range of tensor shapes. Despite performing comparable arithmetic, the reference WTConv is substantially slower than the depthwise convolution it replaces. Our reformulation reduces modeled HBM traffic by approximately $2.55\times$, yielding up to a $4.35\times$ training speedup over the reference while roughly halving peak memory usage. Thus, our reformulation preserves the benefits of WTConv while substantially reducing its execution time and memory footprint, removing the systems overhead that previously limited its practical efficiency.
Amit Aflalo, Shahaf E. Finder, Roy Amoyal et al.· 0 citations