Jun Zhu

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Adversarial Contrastive Fine-Tuning (ACFT) uses an Adversarial Hallucination Attribute Flipping procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine- tuning.