Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks
This work studies refusal directions through the training dynamics across refusal datasets and reveals that their brittleness is associated with repetitive refusal starts, which is linked to concentration of gradients and refusal features in a low-dimensional subspace.
Andrey Labunets
· 0 citations