Skip to content

Local Support Learning

Oct 2026 · 0 citations · 46 references
Computer Science

TL;DR

This work proposes Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data, and presents a gate based on a Gaussian Mixture Model, whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases.

Abstract

We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.

View source

Similar papers

#machine learning Preprint Oct 2026

Task Vector Descent: Learning from Non-IID Batches

A central challenge in continual learning is to acquire new knowledge without forgetting what the model has already learned. This challenge appears in language model training when training data comes from various domain-, user-, or task-specific distributions that are encountered unevenly over time. In such settings, s...

Anton Baumann, Jonas Hübotter, Zeynep Akata et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Sparse Autoencoders Can Induce Local Geometry for Mitigating Catastrophic Forgetting

This work introduces Sparse Autoencoder Geometric Anchoring (SAGA), which converts a Sparse Autoencoder (SAE) trained at each task boundary into local, low-rank constraints on representation drift, and establishes its local relation to latent code matching, define its spectral truncation error, and derive a conditional...

Kamil Faber, K. Filus, Łukasz Pawela · 0 citations

Continual Learning via Self-Probe Gradients

The method, CPLUS, uses self-probe and past-sample gradients to scale down parameter updates that conflict with prior behavior, and becomes more effective as models grow: within the Qwen3 model family, it recovers an increasing share of the forgetting caused by standard fine-tuning.

Dongkyu Cho, R. Chunara, Sung-Min Cha · 1 citation
#machine learning Preprint Oct 2026

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

A Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces, which is substantially more token-efficient than forgetting-aw...

Zheng-Hao Zhao, Gao-Wen Liu, Zhi-Ling Lan et al. · 0 citations
#machine learning Preprint Oct 2026

UnAct: Gradient-Free Unlearning via Targeted Activation Intervention

Machine unlearning seeks to remove the influence of designated training data from a trained model without retraining from scratch. Retrain-free methods such as Selective Synaptic Dampening (SSD) and its label-free variant LFSSD avoid full retraining but still require backpropagation and parameter importance computed ov...

Saeed Abdul Muizz, Aayat Rafiq, Iqra Altaf Gillani et al. · 0 citations

Understanding LLM Parameter Update Sparsity through the Lens of Fisher

Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training...

Yu-Fan Zhang, Sagnik Mukherjee, Hao Peng · 1 citation

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.