Numerical experiments show that the proposed online-learning-enhanced nonlinear elimination preconditioner produces more reliable and coherent bad subsets, is robust with respect to both NE and learning parameters, and outperforms the baseline NE preconditioner in terms of the convergence.
Abstract
Nonlinearly preconditioned inexact Newton methods form an effective class of solvers for large-scale nonlinear algebraic systems arising from the discretization of partial differential equations. A central challenge in nonlinear elimination (NE) preconditioning is the reliable identification of the slowly converging components to be eliminated. Existing selection strategies often rely on problem-specific physical information or user-tuned thresholds applied directly to the raw nonlinear residual, which may contain irregular oscillatory structures near stagnation regions, making the selected bad subset highly sensitive to threshold parameters. In this work, we propose an online-learning-enhanced NE preconditioner that identifies the bad subset from the dominant structure of the nonlinear residual rather than from the raw residual itself. Residual snapshots are collected online during the stagnation phase of the current Newton solve, and an unsupervised extraction model is trained to capture the principal nonlinear imbalance. We consider both a linear extractor based on principal component analysis and nonlinear extractors based on autoencoder neural networks. Moreover, we integrate the approach into a parallel domain decomposition framework, which trains a local extraction model independently on each subdomain. The learned residual reconstruction is then used to define the bad subset and guide the nonlinear elimination process. Numerical experiments on lid-driven cavity flows at Reynolds numbers up to 10,000 show that the proposed method produces more reliable and coherent bad subsets, is robust with respect to both NE and learning parameters, and outperforms the baseline NE preconditioner in terms of the convergence.
Domain decomposition methods have been established as efficient and parallel scalable iterative solvers and preconditioners for the solution of large-scale systems arising from the discretization of partial differential equations. In particular, overlapping Schwarz methods have been successfully applied to a wide range of linear and nonlinear problems. However, for problems with highly heterogeneous coefficients, standard domain decomposition methods typically suffer from deteriorating convergence rates. Robustness with respect to the coefficient contrast can be achieved by enriching the coarse space with adaptively selected constraints obtained from local generalized eigenvalue problems. The construction of these adaptive coarse spaces, however, can account for a significant part of the overall computing time. In the present work, machine learning techniques are employed to reduce this part of the computing time in the context of the adaptive Generalized Dryja-Smith-Widlund (AGDSW) coarse space. A two-stage approach is proposed in which regression neural networks are used to predict the adaptive coarse basis functions, while a classification neural network is employed to predict the number of basis functions required to ensure robustness. As a consequence, adaptive coarse spaces can be set up in the online phase without solving any eigenvalue problem. Particular attention is paid to problem-specific aspects, including sign-invariant loss functions and post-processing strategies to significantly improve the predicted constraints. The proposed approach is first investigated for scalar diffusion problems with high coefficient contrasts and is subsequently transferred, without retraining, to problems of linear elasticity and to nonlinear $p$-Laplace problems, also within a nonlinear Schwarz framework.
A. Klawonn, M. Lanser, Janine Weber-Hamacher· 0 citations
Differential equations arise in numerous applications, particularly within scientific and technical contexts. Systems of stiff, time-dependent ordinary differential equations constitute the focus of this work. Exponential integrators are designed to solve such equations by integrating the linear part exactly, while simultaneously approximating the nonlinear part through a linear combination of $\varphi$-functions. By utilizing an augmented stiffness matrix, state-of-the-art methods like KIOPS and RK2EXPINT solve the linear part and evaluate linear combinations of $\varphi$-functions for the nonlinear part in a single step, effectively reducing the computational effort to a single matrix exponential evaluation. However, these classical approaches assume that the system is represented by matrices and vectors, potentially not utilizing the underlying high-dimensional structure. Tensors address this limitation and offer significant storage efficiency through well-established decompositions like the Tensor Train (TT) format. This work provides a general framework for solving stiff, time-dependent systems directly within the TT format. Specifically, KIOPS-TT and RK2EXPINT-TT are developed as extensions of the original KIOPS and RK2EXPINT algorithms. This involves reformulating the scheme of explicit exponential Runge-Kutta integrators for tensors and augmenting the stiffness tensor to compute linear combinations of $\varphi$-functions acting on tensors through a single evaluation of the exponential function using Krylov subspace methods. Furthermore, it is shown that the underlying theory of the matrix methods remains valid, thereby enabling the transfer of key theorems to the tensor case. Numerical experiments confirm significant speed-ups for KIOPS-TT and RK2EXPINT-TT in low-rank scenarios compared to their classical counterparts.
R. Weigel, Tom-Christian Riemer, Martin Stoll· 0 citations
We introduce a residual-driven procedure for training nonlinear-manifold reduced-order models for parametrized linear partial differential equations that, given prescribed latent and lifting spaces, identifies the nonlinear lifting without high-fidelity solution snapshots. The approximation is represented by a low-dimensional latent coordinate together with a nonlinear lifting into a richer reduced space. Rather than fitting the lifting to snapshot data, we determine it by minimizing a computable residual-based upper bound for the state error. For affinely parametrized operators, the resulting training objective admits an efficient offline--online decomposition, and the lifting update reduces to a sequence of low-dimensional weighted least-squares problems. Numerical evaluations on an advection--diffusion problem and a plane-strain elasticity benchmark show that the proposed approach substantially improves accuracy over linear subspaces. The resulting nonlinear models achieve accuracy comparable to snapshot-driven nonlinear-manifold training while avoiding high-fidelity snapshots in the lifting-identification stage.
Francesco A. B. Silva, J. Ragusa, T. Guo et al.· 0 citations
Differentiable nonlinear model predictive control (NMPC) provides a principled way to embed optimal control structure into end-to-end learning paradigms, but its practical use is often limited by the computational and memory costs of both forward optimization and backward sensitivity propagation. This brief proposes PANDA, a matrix-free solver for differentiable NMPC. In the forward pass, PANDA combines proximal-gradient iterations with quasi-Newton acceleration and introduces an adaptive stepsize enlargement mechanism to mitigate the conservativeness of monotone stepsize reduction. The resulting stepsize behavior and its effect on local convergence are theoretically analyzed. In the backward pass, PANDA performs implicit differentiation from the residual equation and computes adjoint sensitivities using Krylov-subspace iterative methods together with automatic-differentiation-based Matrix-Vector product operators, thereby avoiding explicit Hessian and Jacobian construction. The method is evaluated on a nonconvex trailer NMPC problem embedded in an imitation learning task. The results show that PANDA achieves much faster forward and backward computation and lower memory overhead than representative differentiable optimization solvers, while maintaining effective imitation learning performance.
Yuankun Chen, Zifei Nie, Xun Gong et al.· 0 citations
The Levenberg-Marquardt (LM) algorithm is the most widely used method for solving nonlinear least-squares problems, as it combines the robustness of steepest descent with the fast local convergence of the Gauss-Newton method. However, its computational cost can become prohibitive for large-scale problems because each iteration requires solving a large damped linear system, and conventional step acceptance strategies may require repeated solves as the damping parameter is adjusted. Despite this computational challenge, many large-scale least-squares problems exhibit effective low-dimensional structure, with only a small number of parameter-space directions strongly informed by the data. We propose an adaptive hybrid subspace Levenberg-Marquardt (HSLM) algorithm that constructs a low-dimensional subspace from complementary sources of gradient, memory, Krylov-subspace, and randomized curvature information and computes a spectrally damped LM step within this subspace. A distinguishing feature of the method is a deterministic adequacy monitor that quantifies how much descent information is captured by the reduced space and adaptively enriches the subspace when necessary. Step acceptance is decoupled from damping adjustment: Armijo backtracking determines the accepted step length, while the ratio of actual to predicted reduction is used solely to update the damping parameter, thereby avoiding repeated damped-system solves during step acceptance. For the HSLM algorithm, we establish global convergence to stationarity and prove local linear and superlinear convergence. Numerical experiments on neural-network training problems show that HSLM achieves convergence behavior comparable to classical and Krylov subspace LM (KSLM) while substantially reducing per-iteration computational cost, with increasing advantages observed as the parameter dimension grows.
Solving large, sparse linear systems is a core task in scientific computing, and efficient iterative solvers rely critically on effective and robust preconditioning. While classical methods such as algebraic multigrid (AMG) are highly scalable, their robustness can degrade on indefinite or nonsymmetric systems where heuristics originally developed for elliptic PDEs are less reliable. Recently, Graph Neural Networks (GNNs) have emerged as data-driven preconditioners; yet, the practical impact of imposing an AMG-style hierarchy remains underexplored for general sparse matrices. In this work, we propose a Graph Neural Multilevel Preconditioner (GMP) that adopts an AMG hierarchy as a structural prior and learns smoothing, restriction, and interpolation operators in a unified framework. Our method targets general sparse systems and is instantiated as a drop-in preconditioner for standard Krylov solvers. On a benchmark of over 800 sparse matrices, we compare against classical AMG, single-level ILUT, and state-of-the-art GNN preconditioners, and characterize the regimes where multilevel graph neural preconditioning improves convergence or, conversely, introduces overhead relative to strong single-level baselines. These results highlight both the promise and the limitations of enforcing AMG-style multilevel structure in learned preconditioners for large-scale scientific simulations.
Zecheng Zhang, R. Li, Yousef Saad· Proceedings of the 32nd ACM...· 0 citations