M3CA: Multi-modal and Multi-view Fusion based on Multi-layer Cross Attention for Isolated Sign Language Recognition
Abstract
Sign language recognition serves as essential interaction way between hearing-impaired and non-signing communities. However, sign language is inherently a multi-modal interactive process, requiring the complex interplay of spatial morphology and dynamic kinematics. While multi-modal approaches show promise, effectively modeling the interaction between heterogeneous visual modalities, especially in mitigating view-specific occlusions, remains a fundamental challenge. This study proposes a hierarchical attention interaction framework that systematically addresses cross-modal feature fusion through cascaded processing modules. Rather than mere concatenation, the architecture establishes robust spatiotemporal representations via multi-layer Cross Attention interaction that explicitly facilitates the bidirectional alignment between global visual semantics and localized motion dynamic kinematics. Comparative evaluations against state-of-the-art methods confirm that modeling this semantic-motion interaction is crucial for capturing discriminative features and resolving occlusion ambiguities. Extensive experiments on multiple datasets validate the framework’s robustness across varied interactive configurations, providing empirical insights into optimal feature synergy strategies.