Skip to content

Exact Degeneracy Under Balanced k-Shot Sampling:Consequences for Small-Sample Discriminant Analysis on LLM Embeddings

Sep 2026 · 0 citations · 13 references
Computer Science

TL;DR

Under balanced sampling, the within-class scatter operator of Kernelized Linear Principal Component Discriminant Analysis (KLPCDA) is not merely rank-deficient but exactly a scaled orthogonal projector.

Abstract

Balanced k-shot sampling draws exactly k labeled examples per class. We show that it induces an exact, provable degeneracy in a family of small-sample discriminant estimators. Under balanced sampling, the within-class scatter operator of Kernelized Linear Principal Component Discriminant Analysis (KLPCDA) is not merely rank-deficient but exactly a scaled orthogonal projector. We derive the consequences in closed form: two of KLPCDA's seven variants have every signal eigenvalue exactly equal, so their eigenvector selection criterion is provably indifferent rather than ill-conditioned, and a third has a provably void objective. This follows from the estimators'construction, not any dataset; we confirm it on frozen sentence embeddings and, separately, on residual-stream activations from a decoder-only generative model. An in-formula tie-break repairs the two repairable variants, with recovery gated by class count: the residual subspace constraint costs 5x more on few-class than many-class datasets (p=0.000001). We then evaluate the repaired framework on few-shot text classification on frozen LLM embeddings (n much smaller than d, up to 4096), across four datasets, three embedding sizes, and three trained baselines (SetFit, LoRA, in-context learning). A properly cross-validated logistic-regression probe still beats every KLPCDA variant on three of four datasets, at every embedding size; guidance carried from pixel, vibration-signal, and gene-expression data does not directly generalize to this feature space. Three independent geometric separability metrics fail to explain why one high-dimensional decoder-based embedding model underperforms smaller bidirectional encoders, ruling out anisotropy; the gap is substantially an estimation-efficiency effect, not a permanent ceiling, closing by more than 80% when the support set grows from k<=10 to k=30-50 (p=0.00195, both many-class datasets).

View source

Similar papers

Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias

We study the generalization of spectral gradient descent (SpecGD) in overparameterized matrix classification with corrupted labels. Each input combines a shared low-rank signal with a rank-one sample-specific perturbation, referred to as a shortcut, that enables memorization but does not generalize. We contrast collaps...

Guillaume Braun, Ichiro Hashimoto, Masaaki Imaizumi · 0 citations
Preprint Aug 2026

SPECTRA: Subspace-Preserving Embedding Calibration, Transport, and Replay for Fully Few-Shot Class-Incremental Audio Classification

SPECTRA is a framework built on a frozen encoder which adds three components, a lightweight trainable adapter that calibrates the generic embeddings to the task, an exemplar-free anti-forgetting scheme, and a transductive optimal-transport refinement of prototypes at test time.

Giries Abu Ayoub, L. Mualem, Simon Korman · 0 citations

Support Vector Generation: Kernelizing Zero-Shot Classifiers from Pre-Trained Language Models

Support Vector Generation is introduced, a kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero-and few-shot learning and suggests that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.

Shohei Ohsawa · 0 citations
#artificial intelligence Preprint Sep 2026

Follow the Geometry, Not the Model: Cold Start Semi-Supervised Learning

VAST (Veracity-Aware Semi-Supervised Training) outperforms the strongest graph-based SSL baselines at every operating point across three datasets, with statistically significant gains in 7 of 9 comparisons, while producing a deployable inductive classifier rather than requiring transductive graph inference.

Itai David, D. Weinshall · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.