SMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spotting

Gloss-Free Representation Learning for Cross-Dataset Sign Spotting

Sign-language research for resource-constrained languages is often limited by the cost of dense linguistic labels such as glosses, temporal boundaries, and sign order. Broadcast news offers a practical alternative by pairing continuous signing with spoken-language transcripts, but this supervision is weak since text and signing are loosely aligned. Morphologically rich languages such as Turkish add further difficulty, as the same lexical meaning can appear in many inflected forms while some derived forms should remain distinct. We study whether weak transcript-based supervision can pretrain a reusable sign encoder in this setting, where poor text normalization can fragment pseudo-gloss targets and weaken representation learning. Unlike prior pseudo-gloss pipelines designed mainly to improve translation, we test whether the pretrained encoder transfers as a reusable representation for cross-dataset sign spotting. We pretrain on TSL-News, a new Turkish broadcast corpus, using pseudo-gloss labels derived from transcripts rather than manual annotation, comparing rule-based morphological lemmatization with constrained LLM-assisted normalization over a fixed vocabulary. We evaluate the learned representations via cross-dataset sign spotting on a new TSL Spotting Benchmark built from the TSL Dictionary corpus. The LLM-assisted encoder raises top-5 temporal localization mean IoU from 0.235 to 0.465, with 56.2% of examples reaching an IoU of at least 0.50; a frequency analysis suggests this gain is not mainly driven by memorizing frequent pseudo-gloss labels. In a downstream translation check, the same pretraining improves BLEU-4 from 9.60 to 11.04 and ROUGE from 23.48 to 27.43. These results show that loosely aligned broadcast data can provide effective weak supervision for learning sign representations that capture both lexical content and temporal structure.

TransSLR: A Lightweight Transformer for Sign Language Recognition

Automated Sign Language Recognition for under-represented languages remains a largely unsolved problem. Central African Sign Language (CASL) exemplifies this gap: the only available bench-mark, CASL-W60, has a best reported accuracy of 69.93%, and we show that the common heuristic of fine-tuning high-resource models fails to close it. This failure stems from two compounding factors: the limited scale of available CASL data and the significant lexical and visual domain gap between CASL and large-scale corpora such as WLASL, which renders pre-trained representations largely uninformative. To address this, we propose TransSLR, a lightweight Temporal Transformer Encoder trained from scratch on 64-frame normalized pose sequences, with average pooling and a classification head. By operating on geometric keypoint representations rather than raw RGB, TransSLR achieves signer-independent generalization without relying on visual appearance. On the CASL-W60 benchmark, TransSLR establishes a new state-of-the-art accuracy of 80.39%, surpassing the prior best by +10.46%. Beyond accuracy, our encoder-only design significantly reduces computational overhead, making deployment feasible in resource-constrained environments. We conduct extensive experiments on the CASL-W60 benchmark, comparing against RGB-based and multimodal baselines, and demonstrate that TransSLR achieves state-of-the-art performance.

STFNet: spatio-temporal fusion network with hierarchical semantic alignment for continuous sign language recognition

Current continuous sign language recognition (CSLR) methods often struggle to capture fine-grained spatio-temporal dynamics. Although widely adopted, 3D convolutions and their variants process spatial appearance and temporal motion jointly. This joint processing introduces representational ambiguity, blurring subtle motion cues at sign boundaries and causing the loss of detailed spatial information. Moreover, the standard CTC loss provides supervision exclusively at the final output. This causes severe gradient attenuation in shallow layers, forcing the model to rely on a sparse set of discriminative frames. Consequently, the model overfits to a single dominant alignment path. To address these issues, we propose STFNet, an RGB-only framework comprising a Spatio-Temporal Fusion (STF) module and a Hierarchical Semantic Alignment (HSA) module. STF adopts a factorized fusion strategy. It first extracts multiscale temporal features through lightweight depthwise convolutions while preserving the original spatial features in a separate branch. The two pathways are then fused via a learnable structured mechanism that enforces channel-wise pairing between appearance and motion cues. This enables adaptive local fusion, preserving fine-grained spatial details while injecting temporal motion cues in a learnable manner to produce highly discriminative joint representations. HSA injects auxiliary CTC supervision at multiple intermediate stages, acting as a subnetwork regularizer. It directly backpropagates gradients to earlier layers to counteract gradient attenuation, encourages the exploration of diverse alignment paths, and mitigates the CTC peak-collapse problem. All auxiliary losses share the same classifier, which ensures semantic consistency across scales. Moreover, HSA incurs no additional inference cost, as it is active only during training. Extensive ablation studies and visualizations validate the effectiveness of our approach. Results on PHOENIX14, PHOENIX14-T, and CSL-Daily demonstrate that STFNet achieves state-of-the-art performance using only RGB frames. Code is available at https://github.com/zhanglong908/STFNet.

Learning Effective Sign Features without Text for Gloss-free Sign Language Translation

VIPO-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation

Gloss-free Sign Language Translation (SLT) translates sign language videos into spoken-language sentences without gloss annotations, avoiding costly labeling but requiring fine-grained modeling of hands, body, and facial cues. Existing methods often use single-modality or weakly fused features, limiting performance. We propose ViPo-MLLM, a framework that integrates spatio-temporal RGB and human pose features. Dedicated encoders model intra-modal dynamics and cross-modal attention captures long-range dependencies. The fused representation is conditioned with a structured prompt and processed by an LLM trained with contrastive and language modeling objectives. The proposed model was evaluated on the PHOENIX14T and CSL-Daily datasets and achieved new state-of-the-art results on both datasets. Moreover, the ViPo-MLLM model attained competitive performance compared to gloss-based recognition approaches, confirming the effectiveness of the proposed pose cues and cross-modal attention mechanisms.

VTaMo: Video-Text Alignment Model for Sign Language Translation

Sign language translation (SLT) converts continuous sign videos into spoken language text. Gloss-free approaches leverage pre-trained visual encoders and language models but rely on implicit cross-modal alignment from translation supervision alone. We present VTaMo, a framework that introduces explicit multi-granularity alignment at three levels: (1) local alignment via entropy-regularized optimal transport with a learnable null token for fine-grained frame-to-token correspondences; (2) global alignment via a learnable orthogonal transformation that calibrates embedding space geometry through Earth Mover's Distance; and (3) position-aligned contrastive learning for discriminative token-level representations. Experiments on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL demonstrate consistent state-of-the-art performance, with ablations confirming the complementary contributions of each component. Code is available at https://github.com/junyi2005/vtamo.