Bridging Isolated and Continuous Sign Language Recognition Using Multiview Wavelet-Based Modeling and Post-Processing

Abstract

Sign Language Recognition (SLR) has gained increasing attention, while Continuous Sign Language Recognition (CSLR) remains substantially more challenging than Isolated SLR (ISLR) due to the difficulty of accurately detecting sign boundaries within continuous signing streams. This challenge is intensified by co-articulation effects, temporal ambiguity, and the coexistence of slow hand trajectories with rapid and fine-grained finger articulations. To address these issues, this paper presents a diffusion-enhanced, skeleton-based multi-view Wavelet Transformer framework for boundary-aware sign recognition in continuous videos. The proposed model is trained using isolated sign samples and is subsequently extended to continuous sequences through an uncertainty-aware boundary modeling strategy. Hand motion is represented using 3D skeletal keypoints projected onto multiple orthogonal planes, namely the XY, XZ, and YZ views, enabling complementary perspectives that capture both global motion patterns and subtle articulatory variations. Wavelet decomposition is employed to jointly encode temporal and frequency-domain characteristics, facilitating the detection of important motion cues such as pauses, rapid finger movements, and inter-sign transitions. To explicitly model uncertainty in transitional regions, a Boundary-Aware Temporal Diffusion module is introduced to stochastically refine wavelet-domain motion features and to estimate frame-level uncertainty that is particularly informative near sign boundaries. The refined representations are subsequently processed by a Transformer encoder to capture long-range temporal dependencies. A diffusion-augmented post-processing mechanism then integrates classification confidence, wavelet energy, and diffusion-based uncertainty to achieve precise boundary localization within continuous signing streams. Extensive experiments on four benchmark datasets against multiple methods demonstrate competitive performance, confirming robust diffusion-enhanced transfer between isolated and continuous recognition.