Skip to content

Task-Induced Riemannian Metrics for Vision Transformer Feature Spaces

Sep 2026 · 0 citations · 52 references
Computer Science

TL;DR

The Spectral Pullback Network (SPN) is developed, which learns a low-rank version of the metric from randomized power iteration, and is distill into a $310$K-parameter importance head that predicts token importance directly from the features.

Abstract

Methods operating on Vision Transformer (ViT) feature spaces typically rely on Euclidean distance or cosine similarity. This assumes that every direction is equally meaningful, but there is no reason to believe the true task geometry has this property. The task-sensitive geometry of the feature space is given by the pullback metric $g(F) = J(F)^\top J(F)$, where $J$ is the Jacobian of the decoder's output fed to a task-specific distance, with respect to the features. Storing the full $g$ is infeasible at modern scales, and for dense outputs such as depth maps even forming $J$ is impractical. We show that whether a low-rank approximation of this metric can be learned depends on the model-decoder pair, and we characterize this with a matrix-free diagnostic $\kappa_{cap}(r)$ computable with a low number of Jacobian-vector products. For tractable pairs, we develop the Spectral Pullback Network (SPN), which learns a low-rank version of the metric from randomized power iteration, and we distill it into a $310$K-parameter importance head that predicts token importance directly from the features. When the Jacobian spectrum is too spread out for a low-rank approximation, passing the decoder's input features through a VAE bottleneck can restore tractability. Across DPT, DINOv2, CLIP, and VGGT backbones, $\kappa_{cap}(r)$ predicts which learned-metric architectures are viable. The importance head reaches Spearman $\rho = 0.998$ on DINOv2 CLS, and our geometric token pruning reduces the additional depth error of ToMe-based token selection by $25\%$ on DPT depth at prune ratio $0.5$, without fine-tuning the ViT. Project page: https://cyberiada.github.io/TaskInducedViTs/

View source

Similar papers

#machine learning Preprint Oct 2026

A Riemannian Geometry for Low-rank Adaptation

Low-rank adaptation (LoRA) is widely used as a parameter-efficient fine-tuning technique for pre-trained deep neural networks, which approximates the weight update via full fine-tuning by a low-rank matrix $BA^\top$. This parameterization leads to the equivalence relation $(B, A) \sim (BG^{-1}, AG^\top)$ for any invert...

Shoichiro Takeda, Shin'ya Yamaguchi, Satoshi Suzuki et al. · 0 citations
Preprint Sep 2026

Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention

Four propositions prove steepest descent in the embedded metric, gradient-scale independence, well-conditioning, well-conditioning, and exact $\mathrm{O}(d)$-equivariance; two five-seed grokking studies confirm the constrained arm does not grok better than the baseline and the weight-decay exemption has no grokking con...

Rubén Darío Guerrero · 0 citations
Preprint Aug 2026

Equivariant Covariance Tensors: Guaranteed SPD Uncertainty for Tensor-Valued Geometric Learning

A framework for E(3)-equivariant UQ is introduced, modeling the full predictive distribution where both mean and covariance preserve rotational symmetry, and a Log-Euclidean Equivariant Scoring Objective (LE-ESO) is formulated, a robust surrogate loss based on the Multivariate Laplace distribution providing robustness...

Rui-Han Liu, Yun-Ting Ji, Jian-Bo Yu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

JLD: Perceptual Distance Through A Jacobian Lens

Image compression, restoration, and generation all require a way to measure how different two images look to a person. Pixel error ignores how people see, while the most accurate perceptual distances are typically fitted to human judgments, tying them to a fixed data and resolution. For example, when image resolution i...

Shreshth Saini, Balu Adsumilli, Alan C. Bovik · 0 citations
#artificial intelligence Preprint Sep 2026

RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models

Riemannian Geometry-Sensitive Quantization (RGSQ), which formulates quantization as a reconstruction problem under a unified Fisher-Riemannian metric, enabling standard unimodal PTQ methods to evaluate multimodal quantization error under their original assumptions.

Zhi-Ping Wu, Dong-Dong Ren, Yang Zhou et al. · 0 citations
#machine learning Review Aug 2026

Rotational Equivariance in Machine Learning: A Comprehensive Tutorial

This tutorial provides a comprehensive introduction to rotational equivariance, starting from the physical and geometric intuition behind coordinate independence and building up the necessary machinery from geometric deep learning, group theory, and representation theory.

Peter Lippmann, Fred A. Hamprecht · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

Microsoft Research Blog Aug 11, 2026

Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

Radiology AI is evolving beyond report generation. CARE-X explores a unified approach that combines flexible reasoning, calibrated predictions, and measurement-based tools for chest X-ray interpretation. The post Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.