Voice-cloning artificial-intelligence speakers can also mimic human-specific vocal expression

Abstract

As AI voice assistants become increasingly prevalent, psychological research on human-AI voice interaction requires stimuli that control for both speaker identity and prosodic characteristics. We tested whether voice cloning systems trained on prosodically homogeneous recordings can reproduce target prosody when generating novel utterances. Ten Mandarin speakers produced 30 sentences (15 Geography, 15 Trivia statements) in confident, doubtful, and neutral intonation. For each speaker and prosody, we trained separate AI voice clones on either the Geography or the Trivia recordings; these clones then generated all 30 sentences, yielding 2,700 utterances (900 Human, 900 AI-Geography, 900 AI-Trivia) for analysis. Machine learning classification using 88 eGeMAPS acoustic features achieved high within-source accuracy (mean 0.85) and substantial cross-source generalization (mean 0.65), with spectral flux emerging as a top contributor across all three sources. Linear mixed-effects models confirmed that confident prosody exhibited higher spectral flux, longer estimated vocal tract length, and lower fundamental frequency (F0) than doubtful prosody across human and AI speakers; notably, spectral flux showed no sex effect, suggesting it indexes prosodic differences rather than speaker-level anatomical variation. Time-resolved F0 decoding discriminated confident from doubtful prosody most reliably in late utterance windows across all sources, although human speakers showed prosodic markers more broadly distributed across the utterance than AI speakers. We demonstrate that voice cloning preserves not only speaker identity but also prosodic style, while AI-generated speech still forms a distinguishable acoustic population relative to human speech. Voice cloning thus provides a viable method for preparing more controlled human-AI stimuli for psychological research.