Skip to content

Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent

Sep 2026 · 0 citations · 30 references
Mathematics Computer Science

TL;DR

This paper characterise the optimal split under data-dependent evaluation geometries induced by the fine-tuning problem as a compute-split problem under a two-stage pretrain--fine-tune procedure with fixed total optimisation budget, using regularised least squares trained by gradient descent as a tractable setting.

Abstract

Pretraining followed by fine-tuning introduces a compute-allocation problem: under a fixed training budget, compute spent improving the upstream objective reduces the compute available for downstream adaptation. Despite its practical importance, this trade-off is not yet well understood theoretically, even in simple models. In this paper, we cast this allocation as a compute-split problem under a two-stage pretrain--fine-tune procedure with fixed total optimisation budget, using regularised least squares trained by gradient descent as a tractable setting. We characterise the optimal split under data-dependent evaluation geometries induced by the fine-tuning problem. Our results show that the allocation depends on how pretraining directions affect fine-tuning predictions and how fine-tuning shifts are seen through downstream data geometry. In particular, the relevant quantities are determined by prediction-relevant spectral components of the pretraining and fine-tuning empirical covariances. Technically, the analysis relies on a basis-invariant, eigenspace-level spectral decomposition, together with perturbative control of the non-commuting pretraining and fine-tuning dynamics.

View source

Similar papers

#artificial intelligence Preprint Aug 2026

LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment

This paper introduces a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead, and employs a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients.

Hao-Nan He, Xin Fan · 0 citations
#machine learning Preprint Oct 2026

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

A new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients, and its stability during training allows for infrequent, cheap recalibration.

Alexandra Volkova, Matin Ansaripour, Erik Schultheis et al. · 0 citations
Aug 2026

SPIRA: Sparse Information-Geometric Rank Adaptation for Parameter-Efficient Fine-Tuning of Large Pretrained Models.

SPIRA is a PEFT framework that separates support discovery from parameter-efficient adaptation and remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.

Zhong-Yi Wen, Zhikai Zhai, Guo-Min Sun et al. · 0 citations
Preprint Aug 2026

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

A simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution, and introduces a lightweight diagnostic for identifying settings where TailSFT is most likely to help.

Sadhika Malladi, Samy Jelassi, Dylan J. Foster et al. · 3 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.