Yun-Ji Chu

Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation

Conversational speech depends on dialogue context and the listener's immediately preceding behavior. We propose ReACT-TTS, a two-stage framework that uses a one-second pre-response listener facial sequence to plan the next utterance's emotion and prosody before speech realization. On a strict dyadic MELD protocol, Temp...