MU4Sum: Aspect Unlearning in LLM-based Document Summarization

Abstract

Aspect-based Summarization (AbSum) aims to generate document summaries describing different facets of the content. The summaries produced by a Large Language Model (LLM) fine-tuned for AbSum likely cover all aspects in the training set, even though some of them may be undesired. Machine Unlearning (MU) techniques are designed to remove the influence of specific training samples from trained models. Applying MU to disentangle fine-tuned LLMs from undesired aspects is particularly appealing, as it enables efficient model reuse. Despite their relevance and diffusion, MU techniques have not yet been adapted to AbSum. In this work, we bridge the gap by proposing and addressing a new task, i.e., MU for LLM-based Summarization (MU4Sum, in short). Since a document may include both aspects to forget and aspects to retain, we extend traditional MU methods to operate at the token level. Then, we empirically compare the token- and sample-wise approaches using 3 LLMs and 12 state-of-the-art MU techniques on 4 public datasets suited to MU4Sum. Our approaches perform significantly better than existing techniques on both the retain and forget test sets. Furthermore, we propose to integrate MU4Sum into an extractive-then-abstractive summarization pipeline, where the content discarded by an extractive summarizer is treated as the forget set. MU4Sum improves the performance of the traditional pipeline, highlighting its potential for broader applications.