Skip to content

Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction

Sep 2026 · 1 citation · 36 references
Computer Science

TL;DR

A quantization-robust unlearning framework that makes forgetting robust to quantization while maintaining overall model utility is proposed, and a curvature-based criteria that pinpoints sensitive weights in the unlearned model that leads to both non-robust forgetting and reduced utility is revealed.

Abstract

Unlearning ensures LLM compliance by removing the influence of private or copyrighted training data. However, since LLM models typically undergo post-training compression, like quantization, in practical deployment, it has been observed that the unlearning effect can be substantially weakened, with the forgetting behavior degrading more severely than that of model utility. This paper proposes a quantization-robust unlearning framework that makes forgetting robust to quantization while maintaining overall model utility. We analyze this gap through the lens of loss landscape. Specifically, our analysis reveals a curvature-based criteria that pinpoints sensitive weights in the unlearned model that leads to both non-robust forgetting and reduced utility. We therefore propose sensitivity-guided noisy regularization, which is applied on the sensitive parameters to steer the model convergence towards a smoother minima of uniformly low forget and retain losses. Balancing unlearning and utility, we further propose forget-critical optimization, which updates only forget-critical layers, preserving most of the network to retain useful knowledge. Extensive experiments on the MUSE and TOFU benchmarks across multiple LLM unlearning algorithms show that our approach achieves substantially more quantization-resilient forgetting while maintaining utility.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

This work proposes Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score and provides an empirical path toward quantization-resilient unlearning.

Ravi Ranjan, O. Kotevska, Agoritsa Polyzou · 3 citations

On-the-go Forgetting without Explicit Unlearning via ERASE

This work introduces ERASE, Erasure via Reconstructive Adversarial Signal Editing, a framework for on-the-go forgetting that suppresses the observable influence of private data without modifying model weights, establishing a scalable, regulation-aligned pathway for continual, privacy-conscious learning.

Kushal Chakrabarti, Mayank Baranwal · 0 citations
#machine learning Preprint Oct 2026

Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide...

Han Wang, Tianqi Shen, Zong-Lin Liu et al. · 0 citations
Preprint Aug 2026

SAUL: Sharpness-Aware Augmented-Lagrangian Unlearning

SAUL (Sharpness-Aware Augmented-Lagrangian Unlearning), which formulates unlearning as a constrained minimization problem following the principle of "forget enough, but no more than necessary," is proposed.

Jaewan Choi, Junyoung Yang, Sangdon Park · 0 citations

Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Models

Experiments on multiple LVLMs show that the Balanced Fitting method consistently outperforms prior PTQ approaches under both weight-only and weight-activation quantization, while lower reconstruction loss does not reliably translate into better downstream performance.

Min-Chan Kang, Kyeonghye Park, Seungyeon Sa et al. · 0 citations
#artificial intelligence Preprint Oct 2026

When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

Weight-only post-training quantization (PTQ) relies heavily on reconstruction loss minimization to preserve model quality at low precision. We show that the weights favored by minimizing this loss need not yield better model performance on new tasks. In fact, we find that lower reconstruction loss can even degrade mode...

Yan-Long Zhao, Xiao-Yuan Cheng, Hui-Hang Liu et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.