Skip to content
Review Open access

Reading the unspoken: a systematic review of multimodal artificial intelligence for detecting psychological and affective states from nonverbal cues

Sep 2026 · JRTI (Jurnal Riset Tindakan Indonesia) · 0 citations · 50 references

TL;DR

This systematic literature review brings together the findings of multimodal artificial intelligence (AI) in detecting psychological and affective states from nonverbal cues and examines whether it is ready for mental-health and counseling applications.

Abstract

The psychological assessment is limited to only relying on the patients' self-reports, as they may not even understand or have the desire or capacity to express the distress they show with their body language such as facial movements, voice, gaze, posture, and physiology. This systematic literature review brings together the findings of multimodal artificial intelligence (AI), mainly the use of such a system in detecting psychological and affective states from nonverbal cues and examines whether it is ready for mental-health and counseling applications. Using PRISMA 2020, we identified the records from a Scopus search, then screened them by the predefined eligibility criteria and evaluated them with the tool for Mixed-Method Appraisal. The original list of 669 articles was reduced first by removing the duplicates to 665 screened articles, and among them, we selected 20 articles released in 2024, 2025 that fulfilled the criteria. The themes identified in the five modality Channels facial & micro expression, speech & acoustic, text & linguistic, physiologic, and body movement together with their amalgamation were compared against the target states which were mainly depression, general emotion, and concealed affect. Multimodal approach generally yielded better results than unimodal methods and recognized concealed distress and suicidality, but the research is limited in scope, data sets being very small, external validation is missing and ethical precautions are not even considered yet.

Read PDF

Similar papers

Review Sep 2026

How emotions sculpt bodily expression: a systematic review of emotional induction, measures, and findings.

Emotional expressions conveyed through body postures and movements play a central role in social communication and adaptive behaviour, yet they have received considerably less attention than facial expressions. This systematic review examines how emotional bodily expressions are induced, measured, and analyzed. Forty-n...

L. Rogez, Ali Oker, Stéphanie Caillies · 0 citations
Open access Oct 2026

A Reliability Sensitive Theory of Multimodal Emotion Recognition

A reliability-sensitive framework has been proposed, in which recognition is driven by uncertainty management rather than feature accumulation, while preserving the modality-specific nature of different types of evidence.

Yi-Hang Chen · 0 citations
Review Oct 2026

Conversational artificial intelligence and children’s social-emotional development: a scoping review

Conversational AI may offer useful, supervised scaffolding for engagement and communication, but the evidence is heterogeneous, largely short term, and insufficient to support AI as an autonomous emotional caregiver or a substitute friend.

A. Bhatt, S. Nimbalkar, A. Nimbalkar · 0 citations
Review Open access Sep 2026

Beyond what people say: a systematic review of artificial intelligence-based detection of psychological non-disclosure and hidden mental states in counseling and mental health care

Not revealing psychologically distressing thoughts, symptoms, and experiences, that is, psychological non-disclosure, contributes to the failure of detection and treatment in many cases of mental illness. Since diagnostic methods mostly depend on self-reporting, such states often remain unrecognized. This article is a...

I. Ifdil, Nur Adila Wafiqah Zulvi, M. F. Zatrahadi · 0 citations
Book Open access Oct 2026

Recognition of Affective States Using Multimodal and Adaptive AI for Individuals with Motor Disabilities

People with motor disabilities often display atypical emotional expressions that elude conventional recognition models, leading to communication difficulties, isolation, and restricted access to emotional support. Multimodal AI affective recognition systems offer promising avenues to overcome these barriers, but existi...

Maram Djebbi, Kirmene Marzouki, Fernando Marmolejo-Ramos et al. · 0 citations
Review Open access Sep 2026

Affective Computing Approaches in Child–Robot Interaction: A Systematic Review and Taxonomy

Affective computing has become increasingly relevant in child–robot interaction (CRI), particularly in social robotics, emotion recognition, engagement assessment, and autism-related interventions. This systematic review with a critical and integrative synthesis analyzes 105 included studies to examine how affect is se...

Sandra Cano, J. Vásconez, Kiara Villarroel et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.