A Prompt-Guided and Quality-Aware Robust Text–Audio Intent Recognition Framework for Elderly Care

Application of Conformer Architecture in Clinical Speech Input and Intelligent Medical Record Generation

This study proposes a multi-stage intelligent documentation framework that integrates a 12-layer Conformer architecture, BERT-BiLSTM-CRF semantic modeling, and BART-based structured text generation that illustrates the effectiveness of deep time-frequency feature extraction and contextual sequence modeling for complex noisy signals.

Semantic-aligned multimodal human activity recognition using visual and audio data

Human activity recognition (HAR) requires robust performance under heterogeneous sensing conditions for practical deployment. However, single-modality approaches are limited in capturing the rich contextual information inherent in complex human behaviors. This paper presents a semantic-aligned multimodal HAR framework that integrates visual and audio information without assuming instance-level synchronization. To address dataset heterogeneity, samples from the HMDB51 video dataset and the ESC-50 audio dataset are aligned by mapping fine-grained classes into a shared high-level activity label space. For each modality, ResNet-18-based models are trained independently using frame-based visual inputs and 64-bin Mel-spectrogram-based audio representations. During inference, the output logits of the two models are combined through score-level weighted linear fusion. Experimental results show that the proposed multimodal approach consistently outperforms unimodal baselines in terms of accuracy and Macro-F1 score, with particularly notable improvements in activity groups where environmental context plays a significant role. These findings indicate that semantic-aligned score-level fusion can improve recognition robustness even under mismatched dataset conditions.

Edge AI-Based Object Detection via Voice Recognition with an LLM-Based Emotional Assistant for Elderly Care Robots

This paper presents a fully integrated, real-time assistive system that combines voice-based object recognition with a generative conversational interface, specifically designed to enhance elderly care through edge AI deployment. The proposed framework enables intuitive human–robot interaction in domestic environments by fusing natural language understanding, optimized visual detection, and local generative response. Voice commands are processed through a speech-to-text pipeline using the Google Web Speech API, with keyword extraction triggering object detection via a quantized YOLOv8n model accelerated through TensorRT with FP16 inference on an NVIDIA Jetson Nano. In parallel, a locally deployed generative AI assistant, executed entirely on-device, provides empathetic dialogue to support social engagement and emotional well-being. The proposed system adopts a hybrid edge architecture in which object detection, robot control, and LLM-based dialogue generation are executed on-device, while speech-to-text transcription relies on a cloud-based service. This generative interface is implemented as an LLM-based Emotional Assistant. The system achieves 13 FPS with an inference latency of 70 ms for object detection, 94.3% speech recognition accuracy, and an F1-score of 0.69 at a 0.5 confidence threshold. All AI components are executed on-board, preserving privacy for on- device processing while maintaining real-time responsiveness. Experimental validation confirms the effectiveness of deploying multimodal AI, including generative models, on resource-constrained hardware. This work lays the foundation for autonomous, voice-guided care robots that not only assist in locating objects but also engage users socially, promoting greater autonomy and quality of life for older adults.

Advancing emotion recognition: a novel adaptive audio–visual fusion approach

Adaptive Reliability-Guided Multimodal Learning Framework for Robust Emotion and Sentiment Classification Under Noisy and Missing Modalities

Intelligent human–computer interaction, healthcare monitoring, social media analytics, and conversational AI rely on emotion recognition and sentiment analysis. However, multimodal learning systems frequently presume equal dependability between text, audio, and visual modalities, rendering them susceptible to environmental noise, missing information, and modality-specific errors. An Adaptive Reliability-Guided Multimodal Artificial Intelligence (ARGMAI) system for robust multimodal emotion and sentiment classification addresses these problems. In a single end-to-end architecture, reliability-aware preprocessing, adaptive cross-modal attention, dynamic feature fusion, contrastive representation learning, noise suppression, and cross-modal consistency optimisation are combined. Missing modality reconstruction allows the framework to preserve discriminative representations when one or more modalities are lacking. In model is tested using the public CMU-MOSEI and IEMOCAP datasets. Experimental results show that ARGMAI achieves 90.36% accuracy, 90.04% precision, 89.78% recall, 89.91% F1-score, and 96.08% AUC on the CMU-MOSEI dataset and 88.14% accuracy, 87.88% precision, 87.53% recall, 87.70% F1-score, and 88.41% weighted accuracy on the IEMOCAP The approach outperforms existing state-of-the-art multimodal learning algorithms under noisy and missing modality situations and provides consistent convergence and enhanced generalisation for real-world affective computing applications.

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

Cognitive impairment (CI) is a growing public health concern. Early and accurate diagnosis is critical for enabling timely intervention and improving patient outcomes. Speech-based CI detection has emerged as a promising non-invasive approach, as speech signals encode both linguistic and acoustic markers associated with cognitive decline. Recent advances in large language models (LLMs) further strengthen the potential of speech-based assessment by enabling more expressive representation learning and improved generalization across diverse speakers, recording devices, and clinical environments. Moreover, multimodal learning by jointly modeling linguistic and acoustic features allows for a more comprehensive characterization of cognitive and behavioral changes related to CI, leading to more reliable detection. In this work, we propose a multimodal CI detection framework based on open-source LLMs that integrates speech audio and corresponding transcripts while preserving patient privacy. Acoustic embeddings are extracted directly from speech signals, while textual embeddings are generated from automatically transcribed speech. These modality-specific embeddings are then concatenated to create a combined feature vector and used for downstream classification, without requiring access to raw or sensitive patient data. The proposed approach is evaluated on the ADReSS20 and ADReSSo21 benchmark datasets. Experimental results show that the proposed multimodal framework achieves an CI classification accuracy of 92.4% and consistently outperforms single-modality baselines. Our work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization. This advance highlights the power of an LLM-based multimodal framework that fuses linguistic and acoustic data to enable robust, scalable, and non-invasive screening.