Skip to content

Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Models

Sep 2026 · arXiv.org · Vol abs/2609.28778 · 0 citations · 27 references
Computer Science

TL;DR

Experiments with silenced and replacement audio suggest that RT-OPD strengthens the student's reliance on acoustic evidence, and the 3B model achieves 72.72% accuracy on MMAU, the highest among the compared 3B models and competitive with several 7B and 8B models.

Abstract

Audio-language models (ALMs) can exploit textual shortcuts to answer questions while overlooking acoustic evidence, weakening audio understanding. On-policy distillation (OPD) trains compact ALMs by supervising student-generated responses with teacher predictions, but does not explicitly distinguish acoustic support from linguistic predictability. We propose Reward-Tilted On-Policy Distillation (RT-OPD) to strengthen acoustic grounding. Given the same question and student-generated text, a frozen teacher predicts the next token with and without audio inputs. Their log-probability contrast defines a reward that reshapes the teacher distribution for reverse-KL distillation, emphasizing the additional evidence provided by audio. Across two compact students and three benchmarks, RT-OPD consistently outperforms Vanilla OPD. Experiments with silenced and replacement audio further suggest that RT-OPD strengthens the student's reliance on acoustic evidence. Our 3B model achieves 72.72% accuracy on MMAU, the highest among the compared 3B models and competitive with several 7B and 8B models. Code and model checkpoints are available at https://github.com/KaiyangLi1992/RT-OPD.

View source

Similar papers

Sep 2026

OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning

Omni-modal large language models deployed in real-world environments encounter external noise that can interfere with their perception and understanding of multimodal inputs. We study their robustness in audio-visual understanding, focusing on question answering under environmental noise and competing speech. The chall...

Xing-Ming Shui, Da-Peng Chen, Bo-Wei Liu et al. · 0 citations
Open access 2026

Enhancing Auditory Reasoning in Large Audio–Language Models Via Supervised Fine-Tuning and Reinforcement Learning With Verifiable Rewards

It is suggested that auditory reasoning can benefit from training strategies tailored to audio signals and objective answer verification, and they provide empirical evidence for applying RLVR to large audio–language models.

Jian Wang, Sheng-Yan Hao · 0 citations

TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models

TTRSD separates update direction from update position, determined by group-relative advantages, from update position, determined by visual sensitivity, without requiring ground-truth labels, external verifiers, or a separate teacher, demonstrating cross-dataset generalization while preserving inherent reasoning integri...

Shu-Ning Wang, Zhi-Heng Wu, Xun-Lan Zhou et al. · 2 citations
#natural language process... Preprint Sep 2026

Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillation

Joint-Output On-Policy Distillation (JO-OPD) is proposed, which distills the model's stronger S2T policy into joint generation using student-generated S2TS trajectories, which reduces OMG across two interleaved generation architectures.

Da-Xin Tan, De-Hua Tao, Chen-Xi Deng et al. · 0 citations
#machine learning Preprint Oct 2026

AdaLoop: Adaptive-Depth Latent Reasoning for Audio Language Models

Large audio language models answer questions about speech, sound, and music, yet their accuracy drops sharply on tasks that need fine-grained acoustic analysis. Judging which of two speakers has the higher pitch demands iterative signal-level reasoning that a content question does not. Current models spend the same com...

Seung-Woo Lee, Bo-Wen Qi · 0 citations
#natural language process... Open access Sep 2026

The Bairong System for MLC-SLM 2026: Dynamic Question-Aware Evidence Routing for Multilingual Conversational Speech Understanding

Long multilingual conversational spoken question answering requires systems to balance long-range transcript semantics with sparse acoustic and speaker-sensitive cues. We present the Bairong system for the MLC-SLM 2026 Challenge, where a diarization-ASR front-end produces speaker-attributed transcripts and a dynamic ev...

Shang-Kun Huang, Jun-Chao Hu, Hua Shen et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.