Skip to content

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

Sep 2026 · 3 citations · 42 references
Computer Science

TL;DR

This work proposes Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score and provides an empirical path toward quantization-resilient unlearning.

Abstract

Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns. Machine unlearning offers a practical alternative to full retraining, but many existing methods apply broad or fixed parameter updates that can degrade utility and remain brittle under deployment changes such as post-training quantization, where forgotten knowledge may partially re-emerge. We propose Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score. This score identifies layers with high influence on the forget set and low sensitivity to the retain set, allowing FOM-UL to concentrate updates where they are most effective while leaving most of the model unchanged. This targeted update strategy improves the forgetting-utility trade-off and provides an empirical path toward quantization-resilient unlearning by reducing the chance that small, diffuse updates are erased by low-bit rounding. Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization compared with strong GA, NPO, KLD, SURE, ReLearn, and LUNAR-based baselines while preserving retain-set utility close to the vanilla model. Under 8-bit and 4-bit post-training quantization, FOM-UL maintains stronger memorization suppression and utility preservation than competing methods, and adversarial prompt evaluations show lower recovery of forgotten content. Overall, FOM-UL provides an efficient unlearning strategy that improves targeted forgetting, utility preservation, and deployment robustness without claiming formal guarantees of erasure.

View source

Similar papers

Test-Time Unlearning via Sparse Autoencoder

This work proposes ARIA (autoencoder-gated inference-time unlearning), a test-time unlearning method that leaves model weights intact and gates access to unwanted knowledge only when generation enters a forget-related state and introduces three post-unlearning adversarial attacks targeting weight-space and decoding-spa...

Ping-Zhi Li, Jinhao Duan, Vaishnav Tadiparthi et al. · 1 citation
#artificial intelligence Preprint Sep 2026

Confess What You Know: Forget-Set Misalignment with Model Knowledge in LLM Unlearning

CONfession-to-Forget-Set (CONFS), a data-blind framework that constructs model-aligned forget sets by eliciting and formalizing the model's memorized knowledge, approaches Gold-standard performance on several metrics and achieves a competitive forgetting-utility balance, while preserving utility better than other data-...

Miso Kim, Georu Lee, Seungwon Jeong et al. · 0 citations

On-the-go Forgetting without Explicit Unlearning via ERASE

This work introduces ERASE, Erasure via Reconstructive Adversarial Signal Editing, a framework for on-the-go forgetting that suppresses the observable influence of private data without modifying model weights, establishing a scalable, regulation-aligned pathway for continual, privacy-conscious learning.

Kushal Chakrabarti, Mayank Baranwal · 0 citations
Preprint Aug 2026

Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality

ADU is presented, a fine-grained, training-based framework that shifts unlearning from token erasure to contextual attention-pathway decoupling, and achieves the strongest aggregate performance among evaluated baselines on the TOFU and WMDP benchmarks.

Xun-Lei Chen, Qi-Rui Ye, Yuang Li et al. · 0 citations
Preprint Aug 2026

The More Popular, The Harder to Forget: Adaptive Popularity for LLM Unlearning

The AdaPop (Adaptive Popularity) method is proposed, which combines local token confidence with a per-fact popularity-dependent exponent derived from an external proxy, and automates the forget-retain balance via a dual-ascent controller that adjusts the retain penalty each epoch.

Anna Borisiuk, A. Savchenko, Alexander Panchenko et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction

A quantization-robust unlearning framework that makes forgetting robust to quantization while maintaining overall model utility is proposed, and a curvature-based criteria that pinpoints sensitive weights in the unlearned model that leads to both non-robust forgetting and reduced utility is revealed.

Jia-Lu Wang, Jia-Ning Deng, Shu-Qing Luo et al. · 1 citation

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.