SignGPT is presented, a unified, pose-based framework for gloss-free SLT and SLG that integrates part-aware hierarchical representations of body, hand, and facial motion into a shared language model and employs asymmetric multi-token prediction and progressive training for bidirectional modeling.
Abstract
Large language models (LLMs) provide limited support for sign language interaction. Unifying sign language translation (SLT) and generation (SLG) to enable sign language as both input and output can reduce switching between separate models during sign-text interaction. We present SignGPT, a unified, pose-based framework for gloss-free SLT and SLG. SignGPT integrates part-aware hierarchical representations of body, hand, and facial motion into a shared language model and employs asymmetric multi-token prediction and progressive training for bidirectional modeling. We evaluate SignGPT on How2Sign (ASL) and Phoenix-2014T (DGS) through benchmark comparisons, qualitative analyses, and component ablations. An exploratory study with 12 Deaf ASL signers assesses an LLM-mediated sign-to-sign response pipeline, highlighting the potential of unified modeling to support sign language conversation (SLC).
To enable bidirectional mapping between text and sign, SignFLIP adopts a symmetric architecture together with a stage-wise training strategy built on large-scale data, which shows competitive performance compared with task-specific models on both translation and generation tasks, as well as strong transferability to si...
Zhaoyi An, Si-Han Tan, Youngbae Hwang et al.· Conference on Empirical Meth...· 0 citations
Sign Language Translation (SLT) is an essential tool for overcoming communication barriers faced by the Deaf and Hard-of-Hearing (DHH) community, enabling equal access to vital information and interactions. This need is especially critical in large-scale, multilingual settings such as Hajj and Umrah, where millions of...
Sufyan Danish, S. Khan, Rama Abdullah Alghamdi et al.· International Journal of Mac...· 0 citations
Sign Language Translation (SLT) is crucial for communication between Deaf or Hard-of-Hearing communities and society. Advances in deep learning have enabled promising sign-to-text systems, yet progress remains constrained by limited data, complex spatiotemporal dynamics, and non-manual cues. This survey reviews deep le...
Contemporary decoder-only large language models (LLMs) have demonstrated strong capabilities across a wide range of domains. However, existing pretraining paradigms for gloss-free sign language translation (SLT) are largely designed around conventional encoder-decoder pretrained language models, which limits their dire...
Zhi Rao, Yucheng Zhou, Qian-Ran Sun et al.· 0 citations
Sign language machine translation has progressed substantially over the past decade, evolving from isolated sign recognition to end-to-end translation systems. Advances in pose estimation, transformer architectures, and large-scale dataset collection have driven progress, yet challenges remain. Datasets are limited com...
Ozge Mercanoglu Sincan, A. Pelykh, Edward Fish et al.· Annual Review of Linguistics· 0 citations
Sign language production (SLP) systems using avatars have been proposed to quickly provide accessible information in sign language (SL) to Deaf and hard-of-hearing people during emergencies. However, generating accurate SL for breaking news remains difficult. This is due to two key challenges: the difficulty of transla...
Tsubasa Uchida, Soichiro Shima, Kohei Hakozaki et al.· Proceedings of the 2026 28th...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.