BlindSight:Real-Time Image Captioning and Speech Synthesis for Assistive Navigation of the Visually Impaired

Abstract

BlindSight is a smart device created specifically to empower people with vision impairment to develop situational awareness by converting vision information into speech. It captures images through a camera module that can be activated through voice instructions. Images are briefly stored in the memory of the device before being sent to the image processing module where techniques of denoising and enhancement are applied to images to increase the visibility of objects. Images are further forwarded to the image captioning unit where the image processing techniques transform them into natural languages, thus giving the semantic description of the environment to the user. Finally, the image captions are converted to speech, allowing users to orient themselves within the environment. The framework proves its efficiency on benchmark datasets such as Flickr8k and MS COCO. In terms of captioning results, the Florence-2-Large vision-language model provided the highest scores in comparison to CNN-LSTM, ViT-GPT2, BLIP, and ViT Caption with BLEU (0.2980), METEOR (0.4910), ROUGE-L (0.5610), and CIDEr (1.0200). The hazard detection algorithm performed well in comparison to YOLOv5 using YOLOv8 algorithm. Depth Anything V2 proved efficient with depth estimation results of 16.33 FPS.