A Survey on Transformer-Based Long-Range Dependency Modeling in Intelligent Document Understanding

Abstract

Transformers have revolutionized Natural Language Processing (NLP) by capturing complex patterns in data, but they still face challenges in modeling long-range dependency, especially in tasks like Intelligent Document Understanding (IDU), where maintaining coherence across lengthy and varied documents is crucial. This survey reviews Transformer-based models for addressing long-range dependency challenges in IDU. It introduces a taxonomy of key approaches and discusses representative models within memory-focused, attention-focused, and augmentation-focused strategies. Then it explores the types of input modalities and datasets used in IDU, as well as practical applications in generation, text classification, information extraction, multimodal document understanding and sequence modeling. This survey also discusses practical deployment challenges that arise when applying these models to long and complex documents in real-world document processing pipelines. Finally, we outline future research directions, including enhancing memory efficiency, optimizing attention mechanisms, improving long-sequence processing, and advancing multimodal integration. We also emphasize the importance of developing models that can effectively handle complex multimodal documents, incorporate domain-specific knowledge (e.g., legal, medical) and generalize across domains for broader applicability. Overall, this paper highlights advancements in Transformer-based models for long-range dependency handling in IDU and identifies key research directions for building more scalable and structurally aware document understanding systems.