Large Audio Language Models (LALMs) are prone to hallucinating and over-relying on text priors when simultaneously presented with audio and text inputs. To mitigate these hallucinations, we propose utilizing the multimodal Direct Preference Optimization (mDPO) objective, which forces the model to ground its generation...
Bebe Cosgrove, A. Grace, Wei-Ran Wang· 0 citations
This work introduces a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model, offering a practical latent-variable lens for characterizing continuous multimodal interactions.
It is shown that uncertainty in audio-language models depends substantially more on available audio evidence than on question text, and top-1 confidence, entropy, and sampling-based measures are established.
In LLM-based speech translation, transcription-based chain-of-thought (CoT) suffers from a mismatch between reference transcripts used in supervised fine-tuning (SFT) and model-generated transcripts at inference. To address this, we propose joint recognition and translation fine-tuning via group relative policy optimiz...
This work improves upon earlier prompt engineering by showing that a simple binary yes/no constraint reduces the model's tendency to falsely confirm absent audio features, and evaluates a diverse library of targeted audio perturbations and adaptively selecting the optimal negative branch for each task and example.
A. Grace, Zhou-Yuan Huo, Wei-Ran Wang· arXiv.org· 2 citations· ⚡1
This work proposes a semi-supervised generative model that utilizes both labeled and unlabeled samples in a unified framework and maximizes the likelihood of unlabeled samples to learn a latent space shared with the IB on labeled data.
Yiyang Shen, Wei-Ran Wang· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.