Arun Josephraj Arokiaraj

Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally

A targeted adversarial perturbation can drive a vision-language model's (VLM's) teacher-forced training loss for a fixed target caption to near zero, yet the same model, allowed to generate freely, produces the original, correct description with no trace of the target. We call this dissociation the train/inference gap,...