Agentic AI Based Multiscale Differentiable Vision Transformer on LLM For Language Independent Scene Text Recognition

Abstract

Accurate text recognition from images is pivotal for several tasks in real-life scenarios and scene understanding like, autonomous driving, assistive mechanism for visually impaired, image-based translation and so on. Nevertheless, text recognition in natural scenes is challenged owing to the presence of noise, uneven text font distribution and deterioration of image quality under complicated acquisition circumstances. These features along with poor image quality, different text formats and complicated image backgrounds critically influence the text recognition accuracy, and these issues still requires immediate action in the field. To address these issues, an AI agent-based language dependent scene text recognition (STR) method named the Multi-scale Dilate Differentiable Binary Vision Transformer (MDDB-ViT) supported on Large Language Model (LLM) is proposed. Initially, Radial Basis Thin Plate Spline Image Transformation-based pre-processing is employed to align original input image into a common coordinate system for recognition. Based on an encoder–decoder model, the proposed MDDB-ViT method introduces a Multi-scale Dilate 2D Positional module in the encoder that by integrating agent with vision transformer boost multi-scale feature representation. In the decoder, a Differentiable Binary Position Visual Reinforced Large Language Model is designed to enable effective semantic correlation between encoder and decoder. The efficiency of the proposed MDDB-ViT method is validated on ICDAR 2019 MLT dataset. Experimental results demonstrate that MDDB-ViT method outperforms state-of-the-art methods across numerous performance metrics, therefore achieving a good balance between scene text recognition accuracy, training time and peak signal to noise ratio (PSNR).