Toward In-the-Wild Gloss-Free Sign Language Translation via Domain-Adaptive Cross-Modal Alignment
Abstract
Enabling gloss-free sign language translation (SLT) in unconstrained real-world environments is a critical step toward practical deployment. Existing progress has been driven primarily by benchmarks collected in visually homogeneous laboratory settings, where simplified scenes substantially reduce the difficulty of cross-modal alignment. As a result, a pronounced domain gap emerges when models are evaluated on newly released in-the-wild benchmarks or deployed in real-world environments. Under such a domain shift, genuine sign semantics are easily confounded by nuisance visual factors, leading models to rely on non-semantic scene cues rather than the fine-grained spatio-temporal dynamics of sign articulations. To address this challenge, we propose Domain Adaptive Contrastive Learning (DACL), a domain-adaptive training framework that promotes robust cross-modal semantic grounding in visually unconstrained environments. Specifically, DACL transfers semantic alignment priors from controlled source domains to in-the-wild target domains, while introducing signer-free pure background samples as an explicit regularizer to suppress nuisance-driven correlations and encourage the disentanglement of scene context from sign semantics. Without introducing architecture-specific modifications, DACL achieves a BLEU-4 score of 21.47 (+19.03) on the in-the-wild CE-CSL and 18.60 (+3.68) on OpenASL, substantially outperforming counterparts trained directly on the target domain.