A Framework for Supervised and Unsupervised Learning via Reproducing Kernel Hilbert Spaces

Abstract

We introduce a unified framework for supervised and unsupervised learning through density estimation using reproducing kernel Hilbert spaces (RKHSs). To this end, we introduce the notion of a nested RKHS, that is, an increasing sequence of RKHSs whose union is dense in an ambient $L^2$-space. This construction allows us to exploit the computational structure of RKHS methods at each finite approximation level while preserving the approximation power of the entire $L^2$-space. We first study statistical learning in a fixed RKHS. In both unsupervised and supervised settings, we establish strong convergence results for empirical averages of random elements taking values in an RKHS continuously embedded in $L^2$. We then use the nested RKHS structure to combine these statistical convergence results with the density of the union of the RKHSs in the ambient $L^2$-space. We apply this construction to probability density estimation and binary Bayesian classification. For density estimation, we construct explicit empirical estimators of the orthogonal projections of the target density onto the RKHSs and establish their almost sure convergence. A diagonal selection procedure then yields a sequence of estimators converging almost surely to the target density in $L^2$. For binary Bayesian classification, we formulate the approximation of the Bayes target as the minimization of suitable population and empirical risk functionals. We prove consistency of the empirical minimizers and use the nested structure to obtain convergence towards the Bayes target in $L^2$. We further provide a geometric interpretation in terms of hyperplanes and weighted distances in the ambient $L^2$-space. Finally, we show that the population risk admits a direct probabilistic interpretation in terms of the probability of misclassification.