Jul 2026· Symposium on Pattern Recognition and Applications· Vol 14264, pp. 142640C - 142640C-6· 0 citations· 14 references
Engineering
TL;DR
A framework for sentence-level language decoding based on transfer learning and multimodal contrastive alignment is proposed, training an EEG encoder to align neural signals with pretrained text embeddings that are subsequently inverted into text using a pretrained embedding-to-text model.
Abstract
Decoding natural language from non-invasive electroencephalography (EEG) is a key step toward practical brain–computer interfaces for individuals with speech impairments. While prior work on the Large Spanish Speech EEG Dataset has focused on sentence classification, semantic reconstruction remains largely unexplored. We propose a framework for sentence-level language decoding based on transfer learning and multimodal contrastive alignment, training an EEG encoder to align neural signals with pretrained text embeddings that are subsequently inverted into text using a pretrained embedding-to-text model. Under 10-fold subject-wise cross-validation, our approach achieves a BERTScore of 0.320 ± 0.021, BLEU-4 of 0.249 ± 0.026, and 21.7 ± 0.031% sentence classification accuracy over 30 classes. Ablation results indicate that reconstruction performance is primarily driven by contrastive alignment. These findings demonstrate the feasibility of semantically meaningful sentence reconstruction from non-invasive EEG.
The integration of artificial intelligence (AI) and brain-computer interfaces (BCIs) technologies shows great potential in assisting patients with speech impairments and improving cognitive-linguistic decline. Electroencephalogram (EEG) based BCIs, characterized by non-invasiveness, low cost, and high temporal resolution, hold significant application value in speech decoding and cognitive rehabilitation. Currently, most mainstream public EEG datasets rely on Western languages. As a tonal language, Chinese Mandarin differs significantly from Western languages in speech production mechanisms, making existing data insufficient to support future BCI research for Mandarin-speaking patients. To address this gap, we establish a systematic Mandarin EEG dataset and conduct effective speech decoding and related analyses. We design four distinct experimental conditions, namely overt, overt-noisy, intend, and imagine, to simulate different types of speech disorders in clinical scenarios. Using typical Mandarin tonal-vowels and common vocabularies as stimuli, we construct an EEG dataset collected from a healthy adult. We evaluate the speech decoding performance using short-time Fourier transform combined with support vector machine (STFT-SVM) and EEG-Conformer models. Furthermore, we design a multi-task architecture based on the EEG-Conformer to perform a unified decoding task for the two stimulus types and a classification task across the four dataset conditions. To interpret the model, we combine Shapley value computation and decision trees to calculate the importance of different electrodes during classification. Experimental results show that the models achieve effective decoding on our dataset. The EEG-Conformer model performs significantly above chance level across all data, reaching an accuracy of 69.83% in normal speaking conditions and up to 61.46% in conditions simulating speech disorders. In the multi-task setting, the classification accuracy across different conditions exceeds 97%. By utilizing the important electrodes identified through interpretability methods as new feature inputs, the classification performance further improves even with a reduction of over 50% in the channels. These results demonstrate the potential of neural signal decoding technologies in communication assistance, reveal the decodability of Chinese Mandarin EEG datasets, and provide feasible recommendations for the future design of Chinese BCI applications.
Haoming Wang, Gaoyuan Zhang, Xurong Xie et al.· Frontiers in Psychology· 0 citations
Non-invasive decoding of inner speech faces a fundamental data problem: a corpus pairing brain activity with a person's spontaneous inner monologue cannot be collected, and the available proxy paradigms (cued repetitive and retrospectively reported generative inner speech) are slow to acquire, poorly time-locked, and subject compliance is unverifiable. We therefore treat silent reading as a scalable proxy task and ask how much lexical and semantic information a contrastive decoder can extract from it. We report an open-vocabulary analysis of approximately 240,000 word presentations recorded from a single densely-sampled participant across 393 runs (ca. 49 h) of 19-channel dry-electrode EEG. Words from continuous narrative text were presented in rapid serial visual presentation, with typography randomised on every trial to partially decorrelate word identity from low-level visual form. A convolutional EEG encoder, optionally followed by a causal transformer, was trained with a CLIP-style contrastive objective to align short EEG windows with hidden-state embeddings of the presented word taken from a large language model. Decoding, evaluated as word-grouped top-10 retrieval against permutation baselines, was reliably above chance, extended to mid-frequency and rare words, and scaled log-linearly with training-data volume with no sign of saturation. Removing occipital and posterior-temporal electrodes reduced the word-level gain by roughly one third but left context tracking unchanged. Control analyses separate word-level decoding from narrative context tracking and from a non-neural positional prior introduced by the transformer's positional embedding. These results establish that open-vocabulary word-level information is recoverable from EEG during silent reading, and that decoding is data-limited rather than saturated.
I. Marquardt, A. Alchanat, Priyanka Jain· 0 citations
EEG foundation models pretrained on thousands of hours have shown large gains over task-specific networks for motor imagery, seizure detection, sleep staging, and emotion recognition, but their transfer to speech decoding-arguably the most demanding non-invasive BCI application-remains untested. We present the first systematic benchmark of EEG foundation models against strong convolutional baselines for speech decoding, using two corpora: UGR-MINDVOICE (overt and covert Iberian Spanish) and BCI Competition 2020 Track 3 (imagined speech). We compare two foundation models (LaBraM, EEGMamba) against three established baselines (EEGNet, ShallowFBCSPNet, EEGConformer) under a unified preprocessing and fine-tuning protocol. Large-scale EEG pretraining yields no consistent advantage over a 16K-parameter CNN on speech tasks, indicating that current general-purpose EEG pretraining does not yet transfer to speech production and motivating speech-specific foundation models.
O. M. Khanday, Mohamed Baha Ben Ticha, Sanae Belfrouh et al.· 0 citations
Decoding speech information directly from scalp electroencephalography (EEG) into text provides a potential non-invasive neural communication pathway for individuals with severe speech and motor impairments. Compared with invasive approaches such as electrocorticography, EEG is safer and more widely deployable, yet substantially more challenging to decode.This challenge is exacerbated for Chinese sentence decoding, which must handle a high-dimensional output space with thousands of characters, severe inter-subject variability, and low signal-to-noise ratios for text alignment.Existing methods commit to a single supervisory axis---either text semantics or audio acoustic features---yet neither can simultaneously satisfy the demands of sentence-level discriminability and fine-grained temporal resolution required for large-vocabulary Chinese decoding. We introduce EEGAlign, a novel parameter-efficient framework that jointly aligns EEG with two axes---text alignment with BGE-M3 text embeddings and audio alignment with wav2vec~2.0 speech features via contrastive learning followed by CTC character-sequence decoding. On ChineseEEG-2 data, EEGAlign yields state-of-the-art closed-set sentence classification performance, reaching up to 82.37% Top-1 accuracy on Reading Aloud EEG and 41.43% on Passive Listening EEG out of 101 candidates. Ablation studies show that the two alignment axes are highly complementary: combining them yields consistently better performance than either alone. To the best of our knowledge, this is the first study on decoding large-vocabulary Chinese sentences from non-invasive EEG during overt speech production, and achieving strong classification performance with relatively large closed-set candidate-sentence setting.
Tian Zheng, Xurong Xie, Xinxin Zhu et al.· 0 citations
Decoding perceived speech from non-invasive brain recordings has garnered significant attention in recent years due to its wide range of potential applications. However, existing methods face considerable challenges in cross-subject decoding, primarily due to limited generalizability and the absence of explicit mechanisms for extracting subject-consistent information. These limitations result in high training costs and suboptimal decoding performance. To address these challenges, we propose an innovative Cross-Subject Perceived Speech Decoding (CPSD) framework, which comprises two training stages: source model pre-training and personal specialization. In the source model pre-training stage, contrastive learning is employed to capture shared representations across multiple source subjects. Subsequently, personal specialization initializes the model for the target subject by extracting consistent components from the source model and fine-tuning it using target subject data. Additionally, we introduce the Positional Encoding-based Spatial Attention (PESA) module, which remaps MEG/EEG data into a standardized reference space, thereby enhancing cross-subject consistency and facilitating model training. We evaluate the proposed CPSD framework on three perceived speech neural datasets encompassing different modalities and languages. The results demonstrate that our framework outperforms baseline methods by more than 6.8%, 15.4%, and 15.8% in Top-10 accuracy on the Armeni 2022, PKUEEG 2025, and Broderick 2018 datasets, respectively. Further analyses confirm the effectiveness, efficiency, and robustness of the proposed approach.
Aoke Zhang, Bo Wang, Xihong Wu et al.· 0 citations
Electroencephalography (EEG) decoding models often generalize poorly across datasets and subjects due to domain shifts in acquisition protocols and individual neurophysiology. We propose EEG-PRIME, a two-stage EEG foundation model for cross-dataset multi-task decoding. EEG-PRIME combines masked pretraining with prototype-aligned instruction tuning to enable instruction-aware and subject-invariant decoding across diverse BCI paradigms. During pretraining, an EEG encoder learns transferable representations through masked reconstruction with frequency-cutoff spectral augmentation. During instruction tuning, EEG-PRIME incorporates task-semantic, dataset-specific, and subject-invariant conditioning. The resulting conditioning signal modulates the Q-Former through Layer-wise Query Modulation, while frozen text embeddings of class labels serve as prototypes for cosine-similarity-based prediction across heterogeneous label spaces. Experiments on sixteen datasets covering motor imagery, emotion recognition, ADHD detection, covert speech, and mental workload show consistent improvements over state-of-the-art baselines and prior EEG foundation models under cross-subject settings. On two additional held-out datasets, EEG-PRIME achieves balanced accuracy comparable to within-session calibration models without target-domain optimization, calibration, or linear probing, demonstrating promising zero-shot transfer capability.
Shuailei Zhang, Muyun Jiang, Wei Zhang et al.· 0 citations