Skip to content

Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Models

Sep 2026 · arXiv.org · Vol abs/2609.27378 · 0 citations · 59 references
Computer Science

Abstract

End-to-end speech-to-speech dialogue models listen and speak simultaneously, so a continuously open acoustic channel is exposed to adversarial manipulation. We formalize imperceptible attacks on full-duplex agents as optimization over additive perturbations confined beneath the psychoacoustic masking threshold of the carrier speech, under three goals: targeted semantic hijacking, response suppression, and policy jailbreaking. Against an undefended Moshi-style agent, white-box attacks succeed in up to 91.7% of trials. We then introduce psychoacoustically aligned latent smoothing (PALS), which injects anisotropic Gaussian noise shaped by local codebook covariance at the residual-vector-quantized latent interface, with input noise shaped by the masking threshold constraining the attacker and trained by a Kullback--Leibler consistency objective. Deployed with no inference-time cost, PALS reduces hijack to 8.3%, mute to 11.2%, and jailbreak to 9.1% at clean quality within 2.3%. A Monte Carlo-smoothed variant certifies an ellipsoidal latent radius up to 0.616, a guaranteed floor that the empirical robustness far exceeds.

View source

Similar papers

Preprint Sep 2026

Silence-the-Mimic: Accelerating Imperceptible Perturbation Generation Against Voice Cloning

Deep neural network-based Voice Conversion (VC) and Text-to-Speech (TTS) models have rapidly advanced, enabling realistic voice cloning with minimal input data. Such capabilities raise serious concerns over unauthorized cloning of speaker identities and the associated privacy and security risks. Current imperceptible a...

Run-Qiu Xu · 0 citations
Sep 2026

Domain-Incremental Learning for Multi-Channel Replay Speech Detection

This work frames this as Domain-Incremental Learning over acoustic environments and presents the first continual learning benchmark for multi-channel replay speech detection, evaluating a state-of-the-art beamformer-based detector over all 24 environment orderings of the ReMASC corpus with five seeds.

Michael Neri · 0 citations
#artificial intelligence Preprint Sep 2026

DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis

The proposed DiTAR+, a dual-optimization framework for continuous-latent Autoregressive Diffusion Transformer models, introduces Dilated Context Sampling and Hierarchical Acoustic Masking to prevent shallow layers from attending to acoustic pre-context, explicitly decoupling semantic alignment from acoustic detail reco...

Zi-Yu Zhang, Tian-Lun Zuo, Han-Zhao Li et al. · 0 citations
Open access Sep 2026

TTS-Guard: Black-Box Ownership Verification of Text-to-Speech Models via Adaptive Adversarial Speaker-Pair Fingerprints

TTS-Guard is presented, a black-box ownership verification framework for TTS models built on adversarial speaker-pair fingerprints that selects key speaker pairs in a dual embedding space for architecture-agnostic stealth and optimises a perturbation through an adaptive curriculum of shadow models covering fine-tuning,...

Xu-Bin Yue, Zhenhua Xu, Zhebo Wang et al. · 0 citations
Preprint Aug 2026

Grounded Decoding for Autoregressive Speech Enhancement via Adaptive Code-Space Grounding and Local LLM Refinement

An evidence-grounded generative SE framework that uses a deterministic estimate as imperfect evidence as imperfect evidence is proposed and SNR-Conditioned CSG (SNR-CSG), which maps a calibrated residual-SNR estimate to an utterance-level strength and constructs an adaptive grounded anchor is introduced.

Hao Shi, Yuan Gao, Zhao-Heng Ni et al. · 0 citations
Preprint Sep 2026

HAMMER: Harmonic-Aware Parallel Context Modeling and Discriminator-Free Perceptual Optimization for Speech Enhancement

Recent speech enhancement systems combine self-attention and Mamba to capture global interactions and long-range dependencies. Yet these hybrids usually operate as sequence mixers and do not explicitly exploit harmonic periodicity, a strong cue for preserving voiced speech under noise. Perceptual optimization poses ano...

Shang-Fu Chen, Szu-Wei Fu, Sung-Feng Huang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.