A multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions, and dual-stage Valence-Arousal-Dominance conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings.
Abstract
Psychological research on emotion dynamics has established that human affect is a continuous, evolving process: emotions rise, decay, and transition within seconds. Current emotional text-to-speech (TTS) systems, however, condition on a single discrete label or static embedding per utterance, fundamentally misaligning with the temporal nature of affect. While recent LLM-based TTS systems may implicitly vary prosody through text understanding, such variation is neither explicitly controllable nor precise enough for targeted intra-utterance transitions. We address three challenges: (1) a multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions; (2) dual-stage Valence-Arousal-Dominance (VAD) conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings; (3) direction-magnitude decoupled injection structurally separates emotion direction from injection magnitude, preventing content degradation. EmoTra-TTS adds only +0.43% parameters with no latency overhead, achieves 30%-87% relative improvement on emotion transition quality, corroborated by 64.4%-79.5% overall win rates in pairwise preference tests against four SOTA baselines and two commercial systems.
DSSM-CRF, an audio-only architecture that explicitly separates cross-speaker contextual influence and within-speaker emotion evolution, is proposed and matched controls demonstrate complementary gains from speaker-wise factorization and CRF modeling.
HybridEmo is introduced, a post-training framework that initializes both multi-emotion TTS tasks with SFT and then aligns the speech-token policy through Group Relative Policy Optimization using a sample-aware hybrid reward.
A unified post-training framework that equips pretrained text-to-speech models with natural-language control over segment-level emotion and duration is proposed, highlighting post-training as a practical approach to extending existing speech synthesis models.
SteerSpeech is presented, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen.
A. Benazir, Darius Petermann, Felix Xiaozhu Lin et al.· 0 citations
ACERT (Averaged Contextual Emotion Representation through Time), a module that integrates a flexible-length window of conversational context to better capture emotional evolution in spoken interactions, is introduced.
Arthur Peuvot, Romaric Besançon, Gaël de Chalendar et al.· arXiv.org· 0 citations
An LLM-based framework that performs discrete emotion recognition and Valence-Arousal-Dominance dimensional evaluation on IEMOCAP, incorporating acoustic cues as natural language descriptions following the SpeechCueLLM approach is proposed.
Yu-Tong Hu, Jin-Ho D. Choi· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.