Skip to content

SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation

Sep 2026 · arXiv.org · Vol abs/2609.21709 · 0 citations · 60 references
Computer Science

TL;DR

SignGPT is presented, a unified, pose-based framework for gloss-free SLT and SLG that integrates part-aware hierarchical representations of body, hand, and facial motion into a shared language model and employs asymmetric multi-token prediction and progressive training for bidirectional modeling.

Abstract

Large language models (LLMs) provide limited support for sign language interaction. Unifying sign language translation (SLT) and generation (SLG) to enable sign language as both input and output can reduce switching between separate models during sign-text interaction. We present SignGPT, a unified, pose-based framework for gloss-free SLT and SLG. SignGPT integrates part-aware hierarchical representations of body, hand, and facial motion into a shared language model and employs asymmetric multi-token prediction and progressive training for bidirectional modeling. We evaluate SignGPT on How2Sign (ASL) and Phoenix-2014T (DGS) through benchmark comparisons, qualitative analyses, and component ablations. An exploratory study with 12 Deaf ASL signers assesses an LLM-mediated sign-to-sign response pipeline, highlighting the potential of unified modeling to support sign language conversation (SLC).

View source

Similar papers

#computer vision Open access Sep 2026

SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale

To enable bidirectional mapping between text and sign, SignFLIP adopts a symmetric architecture together with a stage-wise training strategy built on large-scale data, which shows competitive performance compared with task-specific models on both translation and generation tasks, as well as strong transferability to si...

Zhaoyi An, Si-Han Tan, Youngbae Hwang et al. · 0 citations
#large language models Review Open access Sep 2026

Leveraging large and vision language models for gloss-free sign language translation in deaf communication: a survey

Sign Language Translation (SLT) is an essential tool for overcoming communication barriers faced by the Deaf and Hard-of-Hearing (DHH) community, enabling equal access to vital information and interactions. This need is especially critical in large-scale, multilingual settings such as Hajj and Umrah, where millions of...

Sufyan Danish, S. Khan, Rama Abdullah Alghamdi et al. · 0 citations
Review Open access Sep 2026

Deep Learning-Based Sign Language Translation: A Survey and Taxonomy

Sign Language Translation (SLT) is crucial for communication between Deaf or Hard-of-Hearing communities and society. Advances in deep learning have enabled promising sign-to-text systems, yet progress remains constrained by limited data, complex spatiotemporal dynamics, and non-manual cues. This survey reviews deep le...

De-Sen Yuan, Rui-Li Wang · 0 citations
#computer vision Preprint Oct 2026

SignRAG: Unified Retrieval-Augmented Gloss-Free Sign Language Translation

Contemporary decoder-only large language models (LLMs) have demonstrated strong capabilities across a wide range of domains. However, existing pretraining paradigms for gloss-free sign language translation (SLT) are largely designed around conventional encoder-decoder pretrained language models, which limits their dire...

Zhi Rao, Yucheng Zhou, Qian-Ran Sun et al. · 0 citations
#large language models Review Oct 2026

Machine Translation for Sign Languages

Sign language machine translation has progressed substantially over the past decade, evolving from isolated sign recognition to end-to-end translation systems. Advances in pose estimation, transformer architectures, and large-scale dataset collection have driven progress, yet challenges remain. Datasets are limited com...

Ozge Mercanoglu Sincan, A. Pelykh, Edward Fish et al. · 0 citations
Book Open access Oct 2026

Evaluation of an Automatic Sign Language Production System for Breaking News

Sign language production (SLP) systems using avatars have been proposed to quickly provide accessible information in sign language (SL) to Deaf and hard-of-hearing people during emergencies. However, generating accurate SL for breaking news remains difficult. This is due to two key challenges: the difficulty of transla...

Tsubasa Uchida, Soichiro Shima, Kohei Hakozaki et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.