Global-to-Local Visual Conditioning for Image Captioning with a Frozen Vision–Language Model
TL;DR
This study investigates an input-level visual conditioning approach for image captioning using a frozen ResNet-50 image encoder and a frozen Phi-3 Mini language model, providing an empirical analysis of global and local visual conditioning for parameter-efficient caption generation with frozen pretrained components.
Abstract
Image captioning with pretrained vision and language models often requires substantial model adaptation, while lightweight settings restrict the number of components that can be updated. This study investigates an input-level visual conditioning approach for image captioning using a frozen ResNet-50 image encoder and a frozen Phi-3 Mini language model. A global image representation is first projected into ten continuous visual tokens. A gated local residual module is then introduced to incorporate spatial features from the final ResNet feature map without modifying either pretrained backbone. The framework was evaluated on the MS COCO Karpathy split under a fixed training and decoding protocol. The global visual-conditioning baseline achieved a CIDEr score of 0.8382 on the test split. Adding the gated local residual module increased CIDEr to 0.8693, while adding only 3.95 million stage-specific trainable parameters. Under a near-matched trainable-parameter budget, a rank-10 QKV-only LoRA control achieved a CIDEr score of 0.8439, whereas direct and gated local residual conditioning achieved 0.8611 and 0.8693, respectively. These results are limited to the evaluated frozen ResNet-50–Phi-3 configuration and do not imply a general advantage over LoRA variants. As an additional sequence-level optimization experiment, self-critical sequence training increased CIDEr marginally from 0.8693 to 0.8704 without changing the inference architecture. Caption quality was further examined using a structured LLM-based evaluation over all 5000 images in the Karpathy test split, assessing semantic accuracy, information coverage, fluency, and reference-based unsupported content. A zero-shot evaluation on Flickr30k showed a marked performance decrease, indicating limited cross-dataset generalization. Overall, the experiments provide an empirical analysis of global and local visual conditioning for parameter-efficient caption generation with frozen pretrained components.