LoopSLM builds on looped Transformers for latent reasoning, reusing a decoder block to refine hidden states with acoustic grounding at every pass, and further narrows the perception-reasoning gap by separating learning to reason from learning to respond, enabling direct inference without CoT.
Abstract
Empathetic spoken dialogue requires models to use both what is said and how it is said to decide how to respond. Explicit CoT can improve paralinguistic perception and make acoustic cues more explicit in replies, yet does not ensure their effective use in response planning. We call this mismatch the perception-reasoning gap. In addition, CoT may not fully capture acoustic cues in words, and generating it adds inference latency. To address these limitations, we introduce LoopSLM, which builds on looped Transformers for latent reasoning, reusing a decoder block to refine hidden states with acoustic grounding at every pass. Its two-stage training further narrows the perception-reasoning gap by separating learning to reason from learning to respond, enabling direct inference without CoT. On EchoMind, LoopSLM improves paralinguistic understanding, reasoning, and reply quality over Qwen2.5-Omni-7B. Against the CoT-SFT baseline, LoopSLM gains over 20 points in reasoning accuracy while generating 64.5% fewer tokens at half the latency. It also outperforms Qwen3-Omni-Thinking on most empathetic reply metrics with 34x lower latency. Despite training only on dialogue data, LoopSLM improves accuracy on general audio benchmarks.
Empathetic spoken dialogue is a sophisticated cognitive process that requires not only recognizing emotions but also inferring a user's latent mental states to provide appropriate support. However, current SpeechLLMs often treat empathy as a direct input-to-response mapping, leading to"superficially warm"but emotionall...
Ding-Dong Wang, Shu-Jie Liu, Ya-Yue Deng et al.· 0 citations
AURAL is introduced, which models a distribution over multiple plausible reasoning continuations in latent space and jointly predicts chunks of future states to reduce sequential forward passes and reasoning latency and shows that harder questions elicit more latent reasoning steps.
Yu-Xiang Wang, Kun-Yu Feng, Yuan-Cheng Wang et al.· 4 citations
While Chain-of-Thought (CoT) reasoning has improved the capability of language models, directly applying it to Spoken Language Models (SLMs) may introduce long silent intervals under the serial"think-then-speak"paradigm, disrupting real-time spoken interaction. To address this issue, we propose an asynchronous think-al...
Junyi Ao, Kainan Peng, Mingbo Ma et al.· 0 citations
Speech large language models (SpeechLLMs) offer reduced latency and retain paralinguistic nuances that are typically lost in cascaded automatic speech recognition (ASR) and text-based LM architectures. However, they continue to lag behind text-only LLMs on complex reasoning tasks, while real-time spoken interaction imp...
Yi-Jen Shih, Pu-Yuan Peng, Abdel-rahman Mohamed et al.· 1 citation
This work first adapts the model to the domain using supervised fine-tuning on synthesized spoken question-answering data and shows that RL improves the accuracy on GSM8K beyond levels previously achieved for speech models only with supplementary reasoning traces, and establishes a new state-of-the-art for mathematical...
A scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples and develops an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the...
Yen-Ju Lu, Yu-Zhe Wang, Yao-Han Guan et al.· 1 citation