To the authors' knowledge this is the first description-based, open-vocabulary sign lookup from continuous signing without gloss supervision, and the first for JSL.
Abstract
Isolated Sign Language Recognition (ISLR) is conventionally cast as closed-set classification over gloss labels, which cannot generalize to signs unseen in training and ties every deployment to a gloss-annotated lexicon. We instead recognize signs extracted from continuous signing by (1) captioning a sign-level clip into a free-form procedural description of the articulation with an open-weight vision-language model, and (2) retrieving the closest entry from a vocabulary of target descriptions with a multilingual sentence encoder: a reverse sign language dictionary that needs no gloss supervision and admits an open vocabulary. On 1,300 sign-level segments from a Japanese Sign Language (JSL) dialogue corpus annotated with procedural descriptions (against a 2% top-10 chance floor over the 503-entry target vocabulary), fine-tuning the captioner substantially improves seen-class retrieval: language and vision tower fine-tuning raises top-10 retrieval on seen classes from 4.5% (untrained) to 49%, becoming statistically indistinguishable from a standard supervised closed-set classifier (I3D) on two of the three test sets where a closed-set classifier can be evaluated at all. More importantly, unseen-class retrieval also improves significantly over the untrained pipeline (11.5% ->21.0% top-10, p=0.0094), a regime in which the closed-set classifier cannot participate. A matcher-side empirical upper-bound analysis shows the sentence encoder already recovers close to 100% of paraphrased gold descriptions, locating a gap in captioning quality that we aim to address in future work. To our knowledge this is the first description-based, open-vocabulary sign lookup from continuous signing without gloss supervision, and the first for JSL.
SignSeek sets a new state-of-the-art performance in cross-corpus retrieval on ASL-Citizen, WLASL, and NMFs-CSL without any downstream fine-tuning, surpassing methods explicitly trained on BSL and outperforming prior skeleton-based methods.
Sobhan Asasi, Ozge Mercanoglu Sincan, Richard Bowden· 0 citations
The results show that loosely aligned broadcast data can provide effective weak supervision for learning sign representations that capture both lexical content and temporal structure, and raises top-5 temporal localization mean IoU from 0.235 to 0.465.
Oğuz Akif Tüfekcioğlu, Ezgi Ekin, Mustafa Kaan Çevik et al.· 0 citations
A prototype-structured sign embedding space is learned from continuous video annotated with signs, where each learnable prototype corresponds to a sign class, enabling the matching between dictionary exemplars and continuous sign instances.
Ryan Wong, Youngjoon Jang, Liliane Momeni et al.· 0 citations
Vietnamese Sign Language (VSL) remains under-resourced for automatic sign language recognition and translation research. Existing VSL datasets primarily focus on isolated gloss-level recordings, while a large-scale dataset containing continuous sentence-level signing sequences is still unavailable. This limitation rest...
Hien Phuong Nhat Nguyen, Hoai Nhan Nguyen, Thi Diem Tran· International Conference on...· 0 citations
The first experiments on isolated sign language recognition (ISLR) for Icelandic Sign Language (ITM) are presented, and transfer from better-resourced sign languages is promising for very low-resource ones.
F. Ingimundarson, Gudny Bjork Thorvaldsdottir, Mathias Müller et al.· Proceedings of the Second Wo...· 1 citation
Identifying an unfamiliar sign is difficult when a learner remembers its movement but does not know its meaning or formal feature codes. SignTrace addresses this longstanding reverse-lookup problem through natural-language access to a Chinese sign-language dictionary. The system integrates LLM-based dictionary enrichme...
Zeng-Ji Tu, Xing-Ye Zhu, Ning Wang et al.· arXiv.org· 0 citations
The new ChartNet training dataset could improve the accuracy of vision-language models that help analyze business trends or interpret scientific figures.