Evidence that influential samples tend to lie close to the decision boundary, thereby making contact with a standard data selection heuristic in active learning is provided.
Abstract
The impact of a given training point on a statistical model is classically measured through its leave-one-out influence, which quantifies the effect of its removal from the training set on the model accuracy. While the statistics of leave-one-out influences are well understood in the low-dimensional, large sample limit $n\to \infty, d=O(1)$, they become more intricate in high dimensions, as the influence of a given sample develops non-trivial dependencies on all other training samples. For convex M-estimation under Gaussian design, in the high-dimensional limit $n\asymp d$, we show that the distribution of the influences across the training set converges to a limiting measure which we sharply characterize. Building on these results, we provide evidence that influential samples tend to lie close to the decision boundary, thereby making contact with a standard data selection heuristic in active learning.
In survey sampling, the goal is to estimate finite population parameters such as totals, means, and proportions. At the estimation stage, it is common to have access to auxiliary information in the form of covariates known either in aggregate form or for each population unit. These covariates are often used, through models relating them to the variable of interest, to improve efficiency; this approach is known as model-assisted estimation. Modern applications increasingly involve settings where a large number of covariates are observed, sometimes of the same order as the sample size. While this setting offers greater modeling flexibility, it also creates important challenges for inference. In this article, we study variance estimation for the generalized regression (GREG) estimator in high-dimensional regimes. We derive new theoretical results that characterize the high-dimensional asymptotic bias of commonly used variance estimators, including those based on Taylor linearization. Furthermore, under suitable distributional assumptions on the covariates, we show that a cross-validated variance estimator is naturally asymptotically unbiased.
High-dimensional change-point analysis is essential in modern statistical inference. However, existing methods are often designed either for specific parameters (e.g., mean or variance) or for particular tasks (e.g., testing or estimation), making them difficult to generalize. Moreover, they typically rely on restrictive distributional assumptions, limiting their robustness to heavy-tailed data. We propose a unified framework for testing, estimating, and inferring multiple change points in high-dimensional data. Our approach leverages a two-sample U-statistic within a moving window, allowing flexible kernel function selection to accommodate structural changes in general parameters such as variance changes or robust statistics. For testing, we develop an L-infinity norm-based statistic with a high-dimensional multiplier bootstrap procedure, achieving minimax-optimal power under sparse alternatives. For estimation, we construct an initial estimator for the change-point number and locations and refine it using the U-statistic Projection Refinement Algorithm (U-PRA), attaining minimax-optimal localization rates. We further derive the asymptotic distribution of refined estimators, enabling valid confidence interval construction. Extensive numerical experiments demonstrate the better performance of our method across various settings, including heavy-tailed distributions. Applications to genomic copy number variation data highlight its practical utility. An R package implementing the proposed method, U-PRA, is publicly available at https://github.com/liubin0145/R-codes-UPRA/.
Efron's bootstrap is the default tool for estimating the sampling distribution of a statistic, yet it is provably inconsistent for maxima of bounded-support distributions, means under infinite variance, extreme quantiles, and tail-index estimators. The classical remedies, the m-out-of-n bootstrap and subsampling, require rate corrections that depend on unknown parameters and behave erratically at realistic sample sizes. We propose an amortized alternative: a neural network is trained on simulated datasets drawn from a prior over a distribution family, using single independent draws of the root T_n - T(F) scored by the pinball loss, a proper scoring rule whose population minimizer is the posterior-predictive law of the root. At test time, a single forward pass maps one dataset of n = 200 observations to its full sampling-distribution estimate, from which confidence intervals follow directly. On four canonical bootstrap-failure problems (bounded-support maximum, alpha-stable mean, Pareto tail index, and 99% value-at-risk under tempered stable returns), the method attains nominal 95% coverage, beats every feasible classical method in Wasserstein distance to the true sampling distribution, and captures over 97% of the achievable improvement where the exact Bayes-optimal answer is computable. For the value-at-risk problem no distribution-free method can reach nominal coverage at all; the learned method attains 94.7%. A single universal network with a statistic token matches all four specialists, and on real daily market returns the unchanged model averages 0.87 coverage against 0.73 for the bootstrap, as predicted by our out-of-family analysis.
Classical canonical correlation analysis becomes numerically unstable when the number of variables is large relative to the sample size and is sensitive to contamination in observations or individual cells. This study develops an integrated robust and regularized procedure that combines bounded cellwise wrapping, shrinkage estimation of the joint correlation matrix, and robust reweighting in a low-dimensional canonical score space. The resulting observation weights enter a second regularized canonical correlation fit, so the final estimator remains well defined when the combined number of variables exceeds the sample size. The simulation study shows that relative estimation accuracy depends on the signal strength, contamination mechanism, and dimensional configuration. The proposed estimator is competitive in several moderate-signal settings and has a clear computational advantage, whereas the minimum regularized covariance determinant plug-in estimator provides lower estimation error in many high-signal configurations. An additional ultra-high-dimensional experiment demonstrates numerical feasibility with modest memory use but also reveals substantial attenuation, identifying a limitation of the present dense estimator. The results therefore support a regime-dependent interpretation rather than a claim of uniform superiority. The complete reproducible simulation workflow is provided.
Hasan Bulut, Müjgan Zobu, V. Saglam· Mathematics· 0 citations
Training neural networks requires balancing the trade-off between fitting the training data and achieving robust performance on unseen inputs. This ability, commonly referred to as generalizability, is determined by the gap between the empirical risk on the training set (``empirical loss'') and the expected risk over the data distribution (``generalization error''). Existing approaches typically estimate the generalization error numerically, requiring gradient descent training and an ``early stopping''strategy. In this work, we introduce an analytic framework that estimates the optimal time of early stopping without the need for training. Several works in the literature also give such analytical estimations, but they are generally based on random matrix theory and often make assumptions on the distribution of the data or the eigenvalue distribution of the covariance matrix. In contrast, our work is based on Rademacher complexity (RC) without needing such probabilistic assumptions. For both theoretical and numerical reasons, it is more relevant to express RC with the L1- norm rather than with the L2-norm. We focus on the case of linear models and the problem of linear regression. Thanks to the ``linear probing''method, our results can, however, be successfully applied to nonlinear neural networks, as illustrated in the classification MNIST example.
D. Hoang, B. Berret, O. Bruneau et al.· 0 citations