A Hybrid Convolutional Neural Network-Vision Transformer Framework for Gujarati Scene Text Recognition in Assistive Applications
TL;DR
Images capture, pre-processing, adaptive text segmentation, sequence modelling with BiLSTM with attention, feature extraction using a hybrid Convolutional Neural Network Vision Transformer (CNNViT) and optimisation directed by Bayesian Optimisation Reinforcement Learning (BORL) are all included in the suggested framework.
Abstract
For visual impairments it is very difficult to read and comprehend textual information included in instructional materials, notice boards, announcements and visual representations of general information. Text-to-speech (TTS) and optical character recognition (OCR) systems are helping to make data more accessible, but it is challenging to reliably recognise Gujarati text in practical settings due to the complexity of real-world conditions like varying light levels, text background strength, font styles and character arrangement. Thus, two wearable device access projects urgently require a high-level support framework with technological skills that can distinguish Gujarati speech and provide information. Image capture, pre-processing, adaptive text segmentation, sequence modelling with BiLSTM with attention, feature extraction using a hybrid Convolutional Neural Network Vision Transformer (CNNViT) and optimisation directed by Bayesian Optimisation Reinforcement Learning (BORL) are all included in the suggested framework. While CNNs recover local Gujarati character-level representations, the Vision Transformer records global contextual associations. The second function entails identifying Gujarati text in speech and translating it into a spoken language that users with visual impairments can understand. Because speech translation is asymmetric, the use of BiLSTM and attention mechanisms improves sequential recognition performance.