Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 2921-2932· 0 citations· 7 references
TL;DR
JZ-Tab is proposed, the first framework dedicated to the automated recognition of Jianzi Pu, which functions as an end-to-end structured visual information extraction system for mixed-script documents and introduces music-structured generation, injecting sequential regularities derived from symbolic music logic into the learning process.
Abstract
Structurally extracting information from mixed-script documents that interleave standard text with open, compositional symbol systems is challenging for both optical character recognition (OCR) and vision–language models (VLMs). This difficulty is epitomized by Jianzi Pu—the ancient Guqin tablature. Unlike closed-set scripts, Jianzi glyphs are formed via infinite compositional rules and are densely integrated with Hanzi text, demanding a model that can simultaneously perform script discrimination, layout parsing, and structural transcription. We propose JZ-Tab, the first framework dedicated to the automated recognition of Jianzi Pu, which functions as an end-to-end structured visual information extraction system for mixed-script documents. Unlike traditional pipelines, JZ-Tab generates layout-aware markup directly from full-page images, bypassing the need for pre-segmentation. Specifically, to overcome the total absence of large-scale annotated datasets, we develop a novel, scalable synthetic-to-real pipeline that constructs layout-consistent pages from canonicalized glyph inventories. Furthermore, to capture the unique action-oriented semantics of the tablature, we introduce music-structured generation, injecting sequential regularities derived from symbolic music logic into the learning process. Finally, we train a VLM for direct page-to-markup generation. Evaluated zero-shot on authentic historical manuscripts, JZ-Tab improves F1 by +40.10 over the strongest generic VLM baselines, highlighting its potential for large-scale automated digitization of historical Guqin manuscripts and open, compositional symbol systems.
By bridging the gap between visually complex documents and machine-readable knowledge, Docling provides a foundation for reliable document understanding in next-generation AI systems.
P. Staar· Proceedings of the 2026 ACM...· 0 citations
SmolDocling, a compact 256M-parameter vision-language model (VLM), is fine-tune to perform end-to-end key-value extraction directly from document images, jointly solving identification, localization, and association in a single pass without OCR preprocessing.
A. Gurbuz, A. Nassar, Christoph Auer et al.· 0 citations
Understanding newspaper images remains a challenging task due to their complex, nested hierarchical structures and dense, heterogeneous layouts. In this paper, we explore two complementary approaches for newspaper structure understanding. First, we present a modular bottom-up pipeline that combines state-of-the-art open-source models: YOLO for layout detection, LayoutReader for reading order prediction, and a custom algorithm for article segmentation. This approach leverages existing robust components while maintaining flexibility and interpretability. Second, we introduce Tiramisu (Tiered Transformers for Hierarchical Structure Understanding), a novel end-to-end transformer-based architecture that explicitly models document hierarchy through an iterative tiered process. Tiramisu performs section and article separation, block localization, semantic categorization, and reading order prediction using highly parallelized attention mechanisms. Finally, we release Finlam La Libert\'e, a new dataset designed specifically for evaluating hierarchical information retrieval in historical newspapers. Experimental results demonstrate the effectiveness of both approaches in reconstructing complex newspaper hierarchies, with comparative analysis highlighting their respective strengths for scalable document digitization. The Tiramisu training code, including the synthetic newspaper generator, is available at https://git.litislab.fr/tiramisu/tiramisu-newspaper-articles-extractor.
William Mocaër, Solène Tarride, Thomas Constum et al.· 0 citations
Open-vocabulary semantic segmentation (OVSS) leverages textual semantics to segment objects beyond predefined categories. While the self-supervised model DINOv3 provides strong structured visual representations, its lack of native textual alignment hinders its direct application to OVSS. To bridge this gap, we propose DINOde, an ODE-based framework that continuously aligns CLIP text embeddings with the DINO visual manifold. Our approach employs two complementary components: (i) Semantic Text Flow (STF), which evolves text embeddings toward the DINO manifold through a continuous ODE trajectory, and (ii) Global Context Flow (GCF), which progressively refines the holistic image representation carried by DINO's CLS token. To preserve the hyperspherical geometry of the feature space during this evolution, we further introduce Velocity Tangent Projection, which constrains the learned velocity field to the tangent space. By modeling alignment as a continuous trajectory, DINOde avoids the manifold entanglement inherent in discrete MLP projections and yields more robust cross-modal alignment. Extensive experiments demonstrate that DINOde consistently outperforms existing methods and achieves state-of-the-art performance across multiple OVSS benchmarks. The code is available at https://github.com/yoon307/DINOde.
Sung-Hoon Yoon, Hoyong Kwon, Chang-Hwan Oh et al.· 0 citations
Extracting structured information from visually rich documents remains a complex task due to variations in layout, text alignment, and reading order. Traditional methods based on IOB tagging or graph decoding often struggle with irregular text sequences and the computational burden of large relational graphs. This paper introduces a novel anchor-based approach that redefines entity representation and association for structured information extraction. The proposed model, named Hwte, integrates visual and linguistic features through a multi-modal transformer architecture that jointly detects entities and their relationships. A new pre-training objective, Masked Detection Modelling (MDM), is introduced to enhance the model’s ability to predict both textual and spatial information simultaneously. Experimental evaluations on benchmark datasets demonstrate that the proposed method achieves superior accuracy and robustness compared to existing solutions, highlighting its effectiveness for real-world document understanding tasks.
Anbu Lakshmi S, P. R. Raksha, Mohamadi Ghouisya Kousar et al.· International Research Journ...· 0 citations
An open, documented workflow for layout segmentation and Automatic Text Recognition of mid-twentieth-century typescripts from the Prefecture of Naples for TEI encoding and named-entity extraction is presented.
