Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations
TL;DR
The result shows that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget increases, and it is proved that the optimal worst-case uniform approximation error over the unit ball ofolder functions on $[0,1]^d$ has the sharp order.
Abstract
Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture $\Phi$ with $P_\Phi$ parameter slots, we write $\boldsymbol{\theta}_f=\mathcal{G}(\boldsymbol{\xi}_f)$, where $\mathcal{G}\colon\mathbb{R}^M\to\mathbb{R}^{P_\Phi}$ is a parameter generator and $\boldsymbol{\xi}_f\in\mathbb{R}^M$ is a latent representation of the target function $f$. The architecture $\Phi$ and the generator $\mathcal{G}$ are shared across the entire target class, while each target $f$ is represented by its own latent vector $\boldsymbol{\xi}_f$, with $\Phi_{\mathcal{G}(\boldsymbol{\xi}_f)}$ approximating $f$. This framework encompasses hypernetworks, low-dimensional parameterizations, parameter-efficient adaptation, and model compression. Understanding the tradeoff between the latent dimension $M$ and the network budget $P$ is therefore fundamental to characterizing the expressive efficiency of these methods. We study this tradeoff for affine generators and fully connected ReLU architectures. More precisely, optimizing jointly over architectures $\Phi$ satisfying $P_\Phi\leq P$ and affine generators $\mathcal{G}:\mathbb{R}^M\to \mathbb{R}^{P_\Phi}$, we prove that the optimal worst-case uniform approximation error over the unit ball of $\alpha$-H\"older functions on $[0,1]^d$, where $0<\alpha\leq1$, has the sharp order $ \bigl(P\min\{M,P\}\bigr)^{-\alpha/d}. $ In particular, our result shows that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget increases.
