Inductive bias and out-of-distribution generalization in deep learning surrogates for 2D PDE dynamics

Abstract

Partial differential equations (PDEs) are a foundational tool for modeling physical systems. However, high-fidelity numerical simulations are often computationally intensive, which has prompted the development of surrogate models in the field of scientific machine learning. While recent approaches have achieved high predictive accuracy by employing complex architectures and extensive datasets, many scientific applications offer only a limited number of simulation trajectories. In this study, we examine the impact of architectural inductive biases on the learning of two-dimensional PDE in data-limited settings. Instead of proposing a highly expressive model, we deliberately adopt a simple multi-channel U-Net modified to enforce periodic boundary conditions. In this study, we compare this architecture with transformer-based and UNet variants such as vision transformer (ViT), adaptive fourier neural operator (AFNO), PDE-Transformer (PDE-T), and UNet with Kolmogorov-Arnold Layers (KAN-UNet) under a common training setup across five PDE families. To keep comparisons uniform and to mimic partial observability in real measurements, we train all models on rollouts of a single representative scalar field per system. Despite its simplicity, the adapted U-Net consistently matches or outperforms these more complex models in forecasting accuracy, spectral similarity, and physics-based metrics while requiring substantially less training time. The findings of this study demonstrate that, in scenarios where data is limited, the judicious selection of inductive biases can often prove to be more advantageous than increasing the model complexity for the purpose of learning the dynamics of PDEs.