Skip to content

Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning

Aug 2026 · arXiv.org · Vol abs/2609.29711 · 1 citation · 42 references
Computer Science

TL;DR

SPARK is proposed, a retention-correction decomposition that first freezes the learned post-task distribution and then applies selective correction around this stable reference and achieves effective selective PII suppression while preserving strong continual-learning utility and knowledge retention across diverse settings.

Abstract

Privacy-preserving continual learning (PPCL) must reduce the reproduction of sensitive content while retaining useful knowledge across sequential tasks. Formal privacy guarantees characterize randomized mechanisms, whereas operational output control concerns whether a trained model selectively reduces the likelihood of sensitive content in its outputs. In this work, we investigate the latter together with continual-learning utility under realistic task evolution. Retention and privacy correction operate at different granularities: task acquisition requires broad preservation of current- and old-task behavior, whereas privacy correction targets sparse annotated positions. Joint optimization leaves the current-task preservation target continually changing. We propose SPARK, a retention-correction decomposition that first freezes the learned post-task distribution and then applies selective correction around this stable reference. Self-Distillation Replay learns the current task while distilling behavior from previous tasks, and Post-Task Privacy Correction reduces annotated-PII likelihood while anchoring current- and old-task non-PII behavior to the resulting checkpoint. Extensive evaluations demonstrate that SPARK achieves effective selective PII suppression while preserving strong continual-learning utility and knowledge retention across diverse settings. Code and data will be released upon publication.

View source

Similar papers

#artificial intelligence Conference Open access Oct 2026

Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning

By avoiding weight updates entirely, Nullify serves as an efficient, plug-and-play inference-time intervention framework for LLM unlearning, and matches or surpasses established baselines in forget quality while achieving near-lossless preservation of model utility.

Wei Zhai, Xiang Liu, Qiang Huang et al. · 0 citations
Preprint Oct 2026

Sleeping Secrets: How Fine-Tuning Reawakens Privacy Risks in Language Models

This work proposes ReGap, a data-free attack that recovers private associations using task structure, filters them by answer-token likelihood, and updates the target model via low-rank adaptation, which requires neither target answers nor auxiliary genuine private supervision.

Jian-Hong Li, Jia-Hao Chen, Yuwen Pu et al. · 0 citations
#machine learning Preprint Oct 2026

Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning

Modeling LLM reasoning as search over a directed acyclic graph, a unified theoretical analysis of both the dynamics of post-training---OPSD and supervised fine-tuning (SFT) in continual learning---and the impact of pre-training on subsequent performance shows that reliable continual reasoning depends on how post-traini...

Shinichi Uemura, Taiji Suzuki · 0 citations

iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs

Controling how much and when teacher information is introduced improves specialisation while preserving broader capability is shown, and results show that controlling how much and when teacher information is introduced improves specialisation while preserving broader capability.

Ahmed Khaled Khamis, Xiao-Tong Ji, Hassan Jaber et al. · 1 citation
#machine learning Preprint Oct 2026

OP-Print: Tracing Membership Imprints in On-Policy Self-Distillation

This work investigates both gray-box and black-box threat models and proposes a unified attack framework, named OP-Print, based on the finding that the supervision imprint in on-policy self-distillation is sparse and structured, concentrating on uncertainty-sensitive correction sites rather than across the entire rollo...

Qi Li, Xin-Chao Wang · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.