Retrieval-Stage Interactions in Cross-Modal Text-to-Image Retrieval Pipelines

Abstract

Cross-modal retrieval systems enable searching content in one modality using queries from another. Recently, retrieving images using natural language queries -- text-to-image retrieval -- has received a lot of attention due to advances in deep learning vision models and interest in multi-modal processing in general. While several modeling approaches exist, ranging from simple keyword matching to joint embedding models and neural rerankers, there is limited understanding of how candidate generation, reranking, and fusion interact across retrieval pipelines. In this paper, we present an analysis of retrieval-stage interactions in text-to-image retrieval pipelines, spanning a comprehensive set of categories drawn from recent advances in information retrieval: single index pipelines, multi-stage retrieve and rerank pipelines, and ensemble pipelines. We compare the effectiveness of these pipelines on the MS-COCO and Flickr30K benchmarks revealing several retrieval-stage phenomena governing reranking effectiveness, candidate-set expansion, and ensemble behavior.