Multi-Scale Transformer-Based Lexicon-Guided Handwritten Text Recognition Using Adaptive Feature Fusion

Abstract

Offline handwritten text recognition (HTR) remains one of the most challenging problems in document image analysis because of cursive writing, character overlap, unconstrained writing styles, long inter-character dependencies and background degradation. Classical CNN-BLSTM pipelines and single-scale attention models extract features at a fixed resolution and therefore struggle to jointly represent thin strokes of small characters and the global shape of large or connected characters. In this paper we propose a Multi-Scale Transformer-Based Lexicon-Guided HTR framework built around an Adaptive Feature Fusion (AFF) mechanism. A multi-scale convolutional backbone extracts shallow, middle and deep feature maps using parallel 3×3, 5×5 and 7×7 receptive fields. The AFF module learns content-dependent soft weights and channel attention to fuse these heterogeneous features into a single scale-balanced representation, replacing the fixed feature map used by previous lexicon-and-attention systems. A Transformer encoder with multi-head self-attention then models long-range context far more effectively than bidirectional recurrent layers. The network is trained end-to-end with the Connectionist Temporal Classification (CTC) objective and decoded with a lexicon-guided beam search that combines an n-gram prior with edit-distance dictionary matching. On the IAM line-level benchmark the proposed model attains a character error rate (CER) of 3.08% and a word error rate (WER) of 7.24%, improving over our lexicon-and-attention baseline (4.15% CER / 9.72% WER) by 25.8% and 25.5% relative, respectively. On RIMES we obtain 2.71% CER / 7.92% WER, and on the George Washington collection 5.82% CER / 13.05% WER, again outperforming the baseline.