This work introduces three tasks/objectives for reverse localization of text as an instruction-tuning mechanism, where the model is guided to extract textual content based on spatial localization cues, thereby enhancing its spatial grounding ability.
Scene text spotting requires high-precision alignment between textual recognition and spatial localization. While visual-token grounding has emerged as a promising formulation for Multimodal Large Language Models (MLLMs), the previous multi-patch paradigm often introduces redundant noise and localization ambiguity, particularly for dense or small text instances. To address this, we propose Single-Patch Text Spotting (SPaTS), a vision-centric framework that routes each text instance through a single anchor visual token and then recovers geometry via full-image refinement. To accurately identify this anchor without oracle labels, we introduce Single-Patch Selective Optimization (SPaSO), a reinforcement learning framework that optimizes discrete visual-token selection using patch-level rewards. To further improve representation robustness and localization precision, we introduce Directional Embedding Alignment (DEA) to suppress unstable norm bias by decoupling feature magnitude and direction, and Patch-Enhanced Decoding (PED) to fuse the routed anchor with language semantics and cross-attend over the full-image feature map for geometry-aware boundary regression beyond coordinate-space surrogates. Extensive experiments demonstrate that SPaTS consistently and significantly outperforms both frontier closed-source MLLMs and OCR MLLMs. Code is available at https://github.com/eeNickTang/SPaTS.
Rui Tang, Wentao Yang, Peirong Zhang et al.· 0 citations
Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding.
Changhao Xiang, Shangyu Xing, Zhen Wu et al.· 0 citations
It is shown that paragraph supervision enables effective use of long token sequences, whereas caption-only training degrades beyond 60 tokens, and paragraph supervision consistently benefits long-description benchmarks and hard negatives prove detrimental in text-only fine-tuning.
Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz et al.· 0 citations
Scene text recognition (STR) in natural images remains highly challenging due to the large variations in character appearance across diverse real-world conditions, such as changes in font, color, layout, and background complexity—which hinder model generalization and remain insufficiently explored. To address this issue, we propose a visual prompt-guided differential learning (VPDL) framework designed to improve the generalization capability of STR models without requiring scene-specific fine-tuning. Inspired by the human ability to reference prior visual knowledge when recognizing text, VPDL introduces a set of character-level visual prompts that guide the model in perceiving appearance variations among characters. Built upon these prompts, we develop a local-to-global differential learning strategy that enhances patch-level representations and aligns global features with character cues while preserving scene-specific information. Additionally, to mitigate exposure bias in autoregressive decoding, we replace conventional label inputs with context-aware textual prompts, encouraging the decoder to better utilize textual cues embedded in image features. Extensive experiments on widely used benchmarks and real-world datasets demonstrate the effectiveness of VPDL.
Yali Niu, Jiahao An, Shihao Zou· Journal of Advanced Computat...· 0 citations
MonkeyOCRv2, a visual-text pretrained model for document AI, is presented, and a pretraining strategy that jointly learns image-to-text generation and pixel-level document reconstruction is proposed: the former aligns visual representations with textual content, while the latter preserves character strokes and layout details.
Yuliang Liu, Zhang Li, Ziyang Zhang et al.· 1 citation
