A Compositional Kernel Model for Feature Learning
TL;DR
This model provides a tractable setting for studying feature learning in compositional architectures and shows how relevant variables are recovered while noise variables are eliminated, and proves that both global minimizers and stationary points discard noise coordinates when the noise variables are Gaussian distributed.
Abstract
We study a compositional variant of kernel ridge regression in which the predictor is applied to a coordinate-wise reweighting of the inputs. Formulated as a variational problem, this model provides a tractable setting for studying feature learning in compositional architectures. From the perspective of variable selection, we show how relevant variables are recovered while noise variables are eliminated. We prove that both global minimizers and stationary points discard noise coordinates when the noise variables are Gaussian distributed. A central finding is that $\ell_1$-type kernels, such as the Laplace kernel, succeed in recovering features contributing to nonlinear effects at stationary points, whereas Gaussian kernels recover only linear ones.
