This work shows how to efficiently perform arbitrary contractions of the derivative tensor of arbitrary order for multivariate functions by properly constructing the input tangents to univariate high-order AD, which can be used to randomize any differential operator efficiently.
Tensor networks are powerful formats for compressing large-scale data. However, their application to general data processing has been limited by the difficulty of performing nonlinear operations. Here, we introduce iterative tensor network transformations (ITNTs), a general algorithmic framework for the element-wise evaluation of elementary and nonlinear filtering functions on data encoded as tensor trains (TTs), a class of tensor networks. Our approach operates entirely in the compressed domain, enabling efficient computation on exponentially large datasets while maintaining a controlled computational cost. We demonstrate its power in two key areas: (I) evaluating highly nonlinear elementary and filtering functions on a 3D reactive flow field, enabling high-fidelity reaction rate computation and region filtering, and (II) finding extrema in complex optimization problems, such as solving Max-SAT instances on spaces up to $2^{70}$ configurations. These results establish ITNT as a foundational tool that provides tensor network methods with the capability for general-purpose data science and large-scale optimization.
Xiao Wang, Tomohiro Hashizume, Pia Siegl et al.· 2 citations
BaKron is an efficient solver that combines anti-diagonal parallelism with a recursive divide-and-conquer construction that matches the cubic scaling of GPTQ while exploiting richer curvature information.
Differential equations arise in numerous applications, particularly within scientific and technical contexts. Systems of stiff, time-dependent ordinary differential equations constitute the focus of this work. Exponential integrators are designed to solve such equations by integrating the linear part exactly, while simultaneously approximating the nonlinear part through a linear combination of $\varphi$-functions. By utilizing an augmented stiffness matrix, state-of-the-art methods like KIOPS and RK2EXPINT solve the linear part and evaluate linear combinations of $\varphi$-functions for the nonlinear part in a single step, effectively reducing the computational effort to a single matrix exponential evaluation. However, these classical approaches assume that the system is represented by matrices and vectors, potentially not utilizing the underlying high-dimensional structure. Tensors address this limitation and offer significant storage efficiency through well-established decompositions like the Tensor Train (TT) format. This work provides a general framework for solving stiff, time-dependent systems directly within the TT format. Specifically, KIOPS-TT and RK2EXPINT-TT are developed as extensions of the original KIOPS and RK2EXPINT algorithms. This involves reformulating the scheme of explicit exponential Runge-Kutta integrators for tensors and augmenting the stiffness tensor to compute linear combinations of $\varphi$-functions acting on tensors through a single evaluation of the exponential function using Krylov subspace methods. Furthermore, it is shown that the underlying theory of the matrix methods remains valid, thereby enabling the transfer of key theorems to the tensor case. Numerical experiments confirm significant speed-ups for KIOPS-TT and RK2EXPINT-TT in low-rank scenarios compared to their classical counterparts.
R. Weigel, Tom-Christian Riemer, Martin Stoll· 0 citations
This work presents a method to accelerate the optimization of learning high dimensional functions using deep neural network (DNN), and studies the effect of adding features which distill pretrained DNN into TNs using a discretize and decompose strategy.
Deep learning for solving partial differential equations (PDEs) has become increasingly prominent. The Fourier Neural Operator (FNO) architecture has been proven to be an efficient and high-precision method that is widely used in scientific research. However, FNO incurs significant overhead by increasing the scale and dimensionality of practical problems. The insufficient utilization of hardware resources in its key operations reduces the computational efficiency of FNO solvers in high-resolution and time-sensitive problem scenarios, and cannot provide effective solution capabilities. To address the latency induced by low computational resource utilization and large-scale data access and computation, we propose FNO-Speed, an integrated solution incorporating the multi-level parallel FNO-aware mapping and tiling GEMM optimization strategy and the custom-sized high-frequency signal filtering scheme. FNO-Speed effectively leverages the data characteristics of FNO layers and the GPU hierarchical structure to adopt a data tiling and partitioning strategy, implementing matrix multiplication based on vector outer products and operator fusion to replace convolution. It also adopts a data reorganization scheme and computation restructuring to address fragmented memory access operations and serial einsum in frequency-domain. The FNO-Speed optimization strategy enhances the utilization of device memory bandwidth and computational efficiency and achieves significant acceleration in both 2D and 3D problem scenarios while maintaining nearly identical accuracy. The model achieves up to 1.4 × end-to-end training speedup, and the parallel efficiency achieves around 70% on 4 GPUs, fully demonstrating the effectiveness of the FNO-Speed optimization strategy in improving FNO performance.
Tao Song, Ouyang Ying, Xiangyu Meng et al.· ACM Transactions on Architec...· 0 citations