Entangled Representations Amplify Collateral Damage in Unlearning
By repurposing Selective Gradient Masking, a suite of six 254M-parameter language models on English Wikipedia with graded levels of disentanglement between biology and non-biology knowledge are trained, it is found that more disentangled models consistently achieve better retain-forget trade-offs.