Image captioning using a transformer with topic–word semantic modeling and multimodal feature fusion

Abstract

Despite the recent advances in Transformer-based image captioning models, reliance on implicit semantic representations and the lack of integration of topic-level and word-level semantic information with appearance and geometric features remain challenging. To address these limitations, we propose a semantic modeling framework consisting of two components: Topic Embedding and Estimation Predictor (TEEP) and Topic-Related Representative Words (TRRW). TEEP explicitly predicts image-level semantic topics, whereas TRRW generates the top-10 representative words corresponding to each predicted topic. The predicted topics and their corresponding representative words are jointly fused to establish a unified semantic representation. This representation provides a more informative and accurate context, thereby improving caption generation accuracy. Moreover, we propose an extended Transformer architecture that jointly fuses semantic representations generated by TEEP and TRRW with appearance and geometric features. By concurrently modeling semantic context, visual attributes, and spatial relationships, the proposed approach produces more descriptive and semantically consistent captions. Additionally, we propose incorporating the exponential moving average (EMA) into cross-entropy training. This strategy enhances training stability, thereby improving performance across evaluation metrics. Extensive experiments on the MS-COCO dataset demonstrate that our model shows competitive performance compared to several approaches across standard evaluation metrics.