Skip to content

A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval

Sep 2026 · 0 citations · 13 references
Computer Science

TL;DR

To the authors' knowledge this is the first description-based, open-vocabulary sign lookup from continuous signing without gloss supervision, and the first for JSL.

Abstract

Isolated Sign Language Recognition (ISLR) is conventionally cast as closed-set classification over gloss labels, which cannot generalize to signs unseen in training and ties every deployment to a gloss-annotated lexicon. We instead recognize signs extracted from continuous signing by (1) captioning a sign-level clip into a free-form procedural description of the articulation with an open-weight vision-language model, and (2) retrieving the closest entry from a vocabulary of target descriptions with a multilingual sentence encoder: a reverse sign language dictionary that needs no gloss supervision and admits an open vocabulary. On 1,300 sign-level segments from a Japanese Sign Language (JSL) dialogue corpus annotated with procedural descriptions (against a 2% top-10 chance floor over the 503-entry target vocabulary), fine-tuning the captioner substantially improves seen-class retrieval: language and vision tower fine-tuning raises top-10 retrieval on seen classes from 4.5% (untrained) to 49%, becoming statistically indistinguishable from a standard supervised closed-set classifier (I3D) on two of the three test sets where a closed-set classifier can be evaluated at all. More importantly, unseen-class retrieval also improves significantly over the untrained pipeline (11.5% ->21.0% top-10, p=0.0094), a regime in which the closed-set classifier cannot participate. A matcher-side empirical upper-bound analysis shows the sentence encoder already recovers close to 100% of paraphrased gold descriptions, locating a gap in captioning quality that we aim to address in future work. To our knowledge this is the first description-based, open-vocabulary sign lookup from continuous signing without gloss supervision, and the first for JSL.

View source

Similar papers

Preprint Sep 2026

SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval

SignSeek sets a new state-of-the-art performance in cross-corpus retrieval on ASL-Citizen, WLASL, and NMFs-CSL without any downstream fine-tuning, surpassing methods explicitly trained on BSL and outperforming prior skeleton-based methods.

Sobhan Asasi, Ozge Mercanoglu Sincan, Richard Bowden · 0 citations
Preprint Aug 2026

Gloss-Free Representation Learning for Cross-Dataset Sign Spotting

The results show that loosely aligned broadcast data can provide effective weak supervision for learning sign representations that capture both lexical content and temporal structure, and raises top-5 temporal localization mean IoU from 0.235 to 0.465.

Oğuz Akif Tüfekcioğlu, Ezgi Ekin, Mustafa Kaan Çevik et al. · 0 citations
Preprint Sep 2026

SignMatch: Matching Dictionary Signs to Continuous Sign Language Video

A prototype-structured sign embedding space is learned from continuous video annotated with signs, where each learnable prototype corresponds to a sign class, enabling the matching between dictionary exemplars and continuous sign instances.

Ryan Wong, Youngjoon Jang, Liliane Momeni et al. · 0 citations
Conference Aug 2026

VSL-HV8: A Vietnamese Sign Language Dataset for Sign Recognition and Translation Research

Vietnamese Sign Language (VSL) remains under-resourced for automatic sign language recognition and translation research. Existing VSL datasets primarily focus on isolated gloss-level recordings, while a large-scale dataset containing continuous sentence-level signing sequences is still unavailable. This limitation rest...

Hien Phuong Nhat Nguyen, Hoai Nhan Nguyen, Thi Diem Tran · 0 citations
#natural language process... Open access Sep 2026

Isolated Sign Language Recognition for Icelandic Sign Language: Experiments in a Low-resource Setting

The first experiments on isolated sign language recognition (ISLR) for Icelandic Sign Language (ITM) are presented, and transfer from better-resourced sign languages is promising for very low-resource ones.

F. Ingimundarson, Gudny Bjork Thorvaldsdottir, Mathias Müller et al. · 1 citation

SignTrace: Describe a Sign, Find the Word

Identifying an unfamiliar sign is difficult when a learner remembers its movement but does not know its meaning or formal feature codes. SignTrace addresses this longstanding reverse-lookup problem through natural-language access to a Chinese sign-language dictionary. The system integrates LLM-based dictionary enrichme...

Zeng-Ji Tu, Xing-Ye Zhu, Ning Wang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Jun 3, 2026

MIT researchers teach AI models to interpret charts

The new ChartNet training dataset could improve the accuracy of vision-language models that help analyze business trends or interpret scientific figures.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.