Spectral Bounds for Kernel Quadrature
TL;DR
It is demonstrated that the eigenvalues of the kernel obtained by discretizing the integral using suitable quadrature formulas instead are approximated much better by the eigenvalues of the kernel obtained by discretizing the integral using suitable quadrature formulas instead.
Abstract
A bottleneck in the theory of kernel methods in machine learning is the storage requirement. To ameliorate this, a standard trick is to replace the kernel with an explicit feature map. Perhaps, the most well known example is the Gaussian kernel which can be expressed in terms of the Fourier features. Analytically, the kernel $K$ can be expressed in terms of an integral expression that involves a possibly asymmetric kernel $G$ representing the feature map. Numerically, one needs to approximate this integral by a suitable numerical integration scheme, typically Monte Carlo. In this paper, we demonstrate that the eigenvalues of $K$ are approximated much better by the eigenvalues of the kernel obtained by discretizing the integral using suitable quadrature formulas instead. We illustrate this fact in the case of the Gaussian kernel and neural tangent kernels on the unit sphere of a four-dimensional Euclidean space corresponding to the sigmoid and ReLU activation functions.