Skip to content
Preprint

SphereVAE: Hyperspherical Latent Autoencoders for Robust Autoregressive Speech Representation Modeling

Sep 2026 · 0 citations · 40 references
Engineering

TL;DR

SphereVAE is proposed, which constrains the VAE latent space to the unit hypersphere and regularizes the latent distribution toward a uniform prior, so that information is encoded mainly by directional variation, providing a bounded geometric target for autoregressive prediction and reducing the risk of norm drift.

Abstract

With the rapid development of speech generation technology, discrete codec representations have been widely used because they provide a stable prediction paradigm. In expressive speech generation, however, the quantization bottleneck of discrete codecs results in information gaps in fine-grained prosody, timbre, pronunciation, and frame-to-frame continuity. Continuous representations (e.g., VAE latents), by eliminating this constraint, have emerged as a more effective alternative for autoregressive modeling. Yet when continuous representations are used as autoregressive prediction targets, prediction errors can accumulate along the generation chain, causing latent drift and degrading long-form stability. To mitigate this problem, we propose SphereVAE, which constrains the VAE latent space to the unit hypersphere. SphereVAE defines a Power Spherical posterior on the hypersphere and regularizes the latent distribution toward a uniform prior, so that information is encoded mainly by directional variation, providing a bounded geometric target for autoregressive prediction and reducing the risk of norm drift. SphereVAE underperforms the standard VAE on reconstruction metrics due to reduced latent freedom. However, when integrated into VoxCPM for zero-shot TTS and long-text generation, it yields lower content error rates with comparable speaker similarity, and shows more stable long-range speaker consistency. These results indicate that an appropriate latent geometric constraint can effectively mitigate autoregressive error accumulation and drift in speech generation.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis

The proposed DiTAR+, a dual-optimization framework for continuous-latent Autoregressive Diffusion Transformer models, introduces Dilated Context Sampling and Hierarchical Acoustic Masking to prevent shallow layers from attending to acoustic pre-context, explicitly decoupling semantic alignment from acoustic detail reco...

Zi-Yu Zhang, Tian-Lun Zuo, Han-Zhao Li et al. · 0 citations
Preprint Aug 2026

ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure

ReLMCodec is a low-bitrate single-codebook speech codec built upon a preserve--control--refine principle that moves the empirical single-stream predictability--reconstruction frontier in the evaluations, with gains that carry over to downstream text-to-speech (TTS) synthesis in both intelligibility and speaker similari...

Zixiang Wan, Xusheng Yang, Zhengmeng Wang et al. · 0 citations
Preprint Aug 2026

Trajectory Dynamics in Self-Supervised Learning Latent Space for Audio Deepfake Detection

On near-domain benchmarks, static and dynamic approaches perform comparably, and on harder cross-corpus benchmarks with diverse synthesis methods, trajectory dynamics provide substantial gains, confirming that temporal physiological constraints carry detection signal beyond utterance-level statistics.

T. Weber · 0 citations
Preprint Aug 2026

CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents

CuteTTS is presented, a compact continuous-autoregressive TTS system that reconciles high-fidelity generation with the latency demands of real-time interaction and introduces guidance-step distillation, which absorbs classifier-free guidance and multiple solver steps into a single interval-conditioned student.

Yu-Qian Zhang, Yao Shi, Kexin Huang et al. · 0 citations
Preprint Aug 2026

Rethinking Language Model-Based Generative Speech Enhancement in the Latent Space of a Neural Audio Codec

A unified framework covering six popular LM-based generative SE modeling paradigms based on discrete/continuous latent NAC features, and a fine-tuning strategy with auxiliary losses on reconstructed speech to improve both intrusive and non-intrusive metrics is presented.

Yihui Fu, Zhengyang Li, Tim Fingscheidt · 0 citations
#artificial intelligence Preprint Sep 2026

SCAPES: Semantically Conditioned Autoregressive Prior for Environmental Sounds

This paper proposes a segmentation strategy that decomposes audio into overlapping segments, enabling a Continuous Normalizing Flow (CNF) to model the evolution of latent trajectories using Flow Matching, enabling a Continuous Normalizing Flow to model the evolution of latent trajectories using Flow Matching.

Esteban Gutiérrez, L. Wyse, F. Font et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.