Skip to content
Open access

Leveraging Assistive Image Captioning on VizWiz through Multi-Caption Supervision

Oct 2026 · Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi) · Vol 10, pp. 1361-1370 · 0 citations

TL;DR

It is suggested that, even without architectural changes, simply leveraging the available caption diversity during training can provide a practical boost for VizWiz-style captioning tasks.

Abstract

Most image captioning datasets provide several human captions for the same image, yet model training is frequently carried out in a way that effectively treats each image as having only one caption. Our study revisits this mismatch by examining the impact of training with multiple captions per image on caption generation performance. For this purpose, we utilized the VizWiz-Captions dataset, which reflects a real assistive scenario with images taken by blind or visually impaired users with five reference captions per image. We evaluated two conventional CNN-based encoder–decoder models, namely CNN–LSTM and CNN–GRU. In both models, the visual encoder was fixed using InceptionV3 image features, while the recurrent decoder was varied between LSTM and GRU. We performed experiments by using training data by varying the number of captions used per image from one to five (k = 1–5), while keeping the overall pipeline unchanged. The models were evaluated using standard multi-reference metrics (BLEU-1 to BLEU-4, CIDEr, and SPICE). Across both baselines, performance generally improves when more captions are included in training, although the gains are not perfectly monotonic for every metric. The strongest results are observed at k = 5. For CNN–LSTM, the best setting achieved BLEU-4 = 0.175, CIDEr = 0.206, and SPICE = 0.120. On the other hand, CNN–GRU achieved BLEU-4 = 0.192, CIDEr = 0.226, and SPICE = 0.119. These findings suggest that, even without architectural changes, simply leveraging the available caption diversity during training can provide a practical boost for VizWiz-style captioning tasks.

Read PDF

Similar papers

Open access Oct 2026

HARNN: experiment analysis of semi-supervised learning on image captioning using Hybrid AutoEncoder-RNN

Image captioning lies at the intersection of computer vision and natural language processing, aiming to generate coherent and contextually meaningful textual descriptions for images. This capability supports applications such as assistive technologies for the visually impaired, enhanced image retrieval, and automated c...

Aishwarya D. Shetty, J. Shetty · 0 citations
Conference Open access 2026

Automated image captioning with a ResNet-LSTM encoder-decoder for distributed surveillance scenes

Automatic image captioning is a fundamental artificial intelligence task at the intersection of computer vision and natural language processing. This paper presents a generative encoder-decoder model for automatic caption generation from images of military exercises. The model employs an ImageNet-pre-trained 152-layer...

Anđa Denić, Snežana Zurovac, Ivana Petković · 0 citations
Open access Oct 2026

Voice-enhanced image caption generation with vision transformers and Google Cloud TTS

Today, when it comes to the digital world, engaging image captions are crucial to such industries as travel, where the strong visuals play a crucial role in attracting the audience. Nonetheless, manual captioning is both time-consuming and usually lacks the subtlety of a scene. The paper presents an AI-based model of...

Srujana Inturi, Vanitha Guda, M. V. K. Reddy · 0 citations
2026

Model-Intrinsic Attention as Explanation for Radiology Image Captioning

CSMorgan’s participation in the two ImageCLEFmedical 2026 subtasks, caption prediction and explainability, is described, addressing both with a single family of vision–language models, and the divergence between automated ranking and expert clinical judgment that participation in both subtasks makes visible is examined...

Mahmudul Hoque, Raisa Nusrat Chowdhury, Ojonugwa Oluwafemi Ejiga Peter et al. · 1 citation
Open access Sep 2026

ConvNeXt-Driven Image Captioning in Hindi with Adaptive Attention and Transformer Decoder

This work presents a unified deep learning framework for automatic Hindi image caption generation in low-resource settings. The framework combines a ConvNeXt encoder, an SE-inspired adaptive attention module, and a Transformer decoder to improve visual representation learning and generate contextually relevant Hindi de...

Anjali Sharma, Mayank Aggarwal · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.