Benchmarking Pre-Trained Vision-Language Models for Bidirectional Image-Text Retrieval on MS-COCO: BERT+ResNet, CLIP, and BLIP

Abstract

Bidirectional image-text retrieval evaluates whether a model can align visual and textual representations for both text-to-image and image-to-text search. This paper presents a controlled benchmark on the Microsoft Common Objects in Context (MS-COCO) Karpathy split, using the same 5,000-image test gallery, 25,000 captions, similarity-matrix layout, and Recall@K/Recall Sum (RSUM) evaluation for four settings: a frozen Bidirectional Encoder Representations from Transformers (BERT)+ Residual Network (ResNet) dual encoder, Contrastive Language-Image Pre-training (CLIP) Vision Transformer Base (ViT-B)/32 zero-shot, CLIP ViT-B/32 fine-tuned on COCO, and Bootstrapping Language-Image Pre-training (BLIP) image-text contrastive only (ITC-only). The results show a clear gap between independently combined unimodal encoders and pre-trained vision-language models. BERT+ResNet obtains RSUM 148.740, while CLIP zero-shot reaches 361.868. Fine-tuning CLIP improves RSUM to 423.496, while the COCO-trained BLIP checkpoint evaluated with ITC-only scoring achieves the strongest result in this experiment, with RSUM 489.588. Image-to-text retrieval scores higher under COCO's asymmetric protocol, while longer captions are associated with higher text-to-image recall. Qualitative errors center on attributes, counting, spatial relations, and plausible false negatives. These results characterize the evaluated checkpoints and training conditions on this benchmark; they do not establish a universal model hierarchy.