Large Audio Language Models (LALMs) are prone to hallucinating and over-relying on text priors when simultaneously presented with audio and text inputs. To mitigate these hallucinations, we propose utilizing the multimodal Direct Preference Optimization (mDPO) objective, which forces the model to ground its generation...
Bebe Cosgrove, A. Grace, Wei-Ran Wang· 0 citations
It is shown that uncertainty in audio-language models depends substantially more on available audio evidence than on question text, and top-1 confidence, entropy, and sampling-based measures are established.
This work improves upon earlier prompt engineering by showing that a simple binary yes/no constraint reduces the model's tendency to falsely confirm absent audio features, and evaluates a diverse library of targeted audio perturbations and adaptively selecting the optimal negative branch for each task and example.
A. Grace, Zhou-Yuan Huo, Wei-Ran Wang· arXiv.org· 2 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.