Skip to content
#generative ai Open access

Generative AI emotion recognition from bodily gestures and vocal tone reveals modality-specific performance and positivity bias

Sep 2026 · Scientific Reports · Vol 16 · 0 citations · 65 references
Medicine

TL;DR

GenAI models exhibit a non-humanlike social-cognitive profile, excelling with positive emotions but struggling with the negative content, highlighting the risks of integrating GenAI into sensitive fields and underscoring the need for continued validation.

Abstract

Advances in generative artificial intelligence (GenAI) have prompted interest in its application in sensitive fields, including mental health. Yet the validity of its nonverbal social-cognitive abilities remains unclear. This study addresses two issues that probe GenAI’s social intelligence: its capacity to interpret nonverbal channels (bodily gestures and vocal tone) with accuracy comparable to humans, and whether accuracy of emotion recognition is different by emotional valence. Emotion recognition accuracy of three Gemini GenAI models (Pro 1.5, Pro 2 (Experimental) and Flash 2) were evaluated using the EU-Emotion Stimulus Set of bodily gestures and vocal tone depicting basic emotions. Model accuracy was compared to validated human benchmarks. Advanced GenAI models achieved near-human accuracy in recognizing emotions from vocal tone. However, their performance was significantly lower than that of humans in interpreting bodily gestures. GenAI models recognized positive emotions more accurately than negative ones. This bias was significant across all models for bodily gestures and in the advanced models for vocal tone. GenAI models exhibit a non-humanlike social-cognitive profile, excelling with positive emotions but struggling with the negative content. This finding carries profound clinical and theoretical relevance, highlighting the risks of integrating GenAI into sensitive fields and underscoring the need for continued validation.

Read PDF

Similar papers

Open access Aug 2026

Demographic and clinical influences on emotion prosody recognition

Introduction Emotion recognition from speech prosody is a complex cognitive ability that depends on the functional integrity of a distributed neuronal network and may be shaped by demographic, clinical and lifestyle factors. The Terrapino mobile application was developed as a tool to promote cognitive health, but its d...

Martin Šimčík, Ross Andel, Jan Pavlík et al. · 0 citations
Open access Aug 2026

Unraveling emotional signatures: comparing physiological methods and algorithm-based recognition of spontaneous emotional facial expressions

Recognizing others’ emotions is central to social interaction. Traditional biological psychology infers emotional responding via laboratory measures, whereas contemporary computer vision algorithms claim to identify emotions unobtrusively from facial video. However, the validity of such algorithms for classifying spont...

S. Scholz, Johanna Kissler · 0 citations
Open access Aug 2026

Testing the relationships between facial emotion recognition, levels of Autistic traits, and eye gaze patterns

Individuals with high levels of Autistic traits often experience challenges in interpreting others’ emotions, potentially stemming from atypical patterns of visual attention to facial cues. This study investigates whether levels of Autistic traits in a non-clinical sample are associated with overall emotion recognition...

Jayden Z. Li, Fan Yang, M. Chege et al. · 1 citation
#human-computer interacti... Preprint Sep 2026

A barrier or a booster? Familiarity effects on Mandarin emotion prosody recognition using AI-powered voice cloning

Emotion prosody perception requires simultaneous processing of acoustic cues and speaker identity. While listeners effortlessly decode natural speech, AI synthetic voices introduce cognitive complexities due to subtle acoustic atypicalities. It remains unclear how these synthetic features interact with a listener's pri...

Feng-Yi Xu, Gao-Yuan Zhang, Shan-Shan Xue et al. · 0 citations
Open access Sep 2026

Multimodal fusion for emotion classification based on GSR, BVP, and SKT

A novel multimodal and unimodal emotion classification system that leverages Galvanic Skin Response, Blood Volume Pulse, and Skin Temperature parameters is proposed and its performance is evaluated on the PHYMER dataset.

Adhitya Velip, H. Virani, Amita Dessai · 0 citations
Open access Sep 2026

Multimodal Emotion Recognition System Using Facial Expressions and Speech Analysis

Human communication involves emotion as an important issue. Because of emotion, people express and understand feelings, attitudes, and actions. However, effective automatic emotion recognition is still difficult to achieve when based on one single cue, as for instance facial expressions or voice may both present weak o...

S. G., N. S · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Putting a spotlight on AI, society, and humanity

Supported by the MIT Human Insight Collaborative, scholars are exploring how AI is impacting the human experience and reshaping higher education.

Microsoft Research Blog Nov 17, 2026

Can AI help build a healthier future?

What can AI enable in medicine, and how do we get there? Microsoft Research lead Doug Burger, Microsoft Principal Research Manager Ava Amini, Microsoft Chief Scientific Officer Eric Horvitz, and physician-scientist Eric Topol of Scripps Research explore these questions and more. The post Can AI help build a healthier future? appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.