Multi-Objective Performance-Cost Optimization of Multimodal EEG-Eye Tracking Systems for Emotion Recognition

Abstract

Multimodal emotion recognition systems based on electroencephalography (EEG) and eye tracking (ET) provide complementary information about neural and visual responses; however, practical deployment requires balancing classification performance and computational cost. This study investigates this trade-off by systematically evaluating 60 feature representation configurations generated from different EEG channel regions, frequency bands, feature types, and signal modalities. Support Vector Machine (SVM), Random Forest (RF), and Artificial Neural Network (ANN) classifiers were evaluated under subject-independent leave-one-subject-out (LOSO) cross-validation, resulting in 180 realizable classifier-feature configuration systems. Median Macro-F1 was used as the primary performance objective, while median classifier inference time was used as the computational-cost objective. A multi-stage selection strategy integrating Δ-based near-optimal filtering and Pareto dominance analysis was applied to identify performance-efficient systems. The highest median Macro-F1 (0.6474) was achieved by an ANN using temporal delta-band PSD features combined with ET information, with a median classifier inference time of 0.0057 s. This system remained the final selected system across Δ values of 0.01, 0.02, and 0.03. An ET-only ANN baseline achieved a median Macro-F1 of 0.6265, indicating a modest improvement when temporal delta-band EEG information was added. These findings demonstrate that feature representation, modality composition, classifier choice, and computational cost should be considered jointly when designing subject-independent multimodal emotion recognition systems.