STFNet: spatio-temporal fusion network with hierarchical semantic alignment for continuous sign language recognition

Abstract

Current continuous sign language recognition (CSLR) methods often struggle to capture fine-grained spatio-temporal dynamics. Although widely adopted, 3D convolutions and their variants process spatial appearance and temporal motion jointly. This joint processing introduces representational ambiguity, blurring subtle motion cues at sign boundaries and causing the loss of detailed spatial information. Moreover, the standard CTC loss provides supervision exclusively at the final output. This causes severe gradient attenuation in shallow layers, forcing the model to rely on a sparse set of discriminative frames. Consequently, the model overfits to a single dominant alignment path. To address these issues, we propose STFNet, an RGB-only framework comprising a Spatio-Temporal Fusion (STF) module and a Hierarchical Semantic Alignment (HSA) module. STF adopts a factorized fusion strategy. It first extracts multiscale temporal features through lightweight depthwise convolutions while preserving the original spatial features in a separate branch. The two pathways are then fused via a learnable structured mechanism that enforces channel-wise pairing between appearance and motion cues. This enables adaptive local fusion, preserving fine-grained spatial details while injecting temporal motion cues in a learnable manner to produce highly discriminative joint representations. HSA injects auxiliary CTC supervision at multiple intermediate stages, acting as a subnetwork regularizer. It directly backpropagates gradients to earlier layers to counteract gradient attenuation, encourages the exploration of diverse alignment paths, and mitigates the CTC peak-collapse problem. All auxiliary losses share the same classifier, which ensures semantic consistency across scales. Moreover, HSA incurs no additional inference cost, as it is active only during training. Extensive ablation studies and visualizations validate the effectiveness of our approach. Results on PHOENIX14, PHOENIX14-T, and CSL-Daily demonstrate that STFNet achieves state-of-the-art performance using only RGB frames. Code is available at https://github.com/zhanglong908/STFNet.