Skip to content

ConvCue: Complementary Visual Inductive Biases for Vision-Language Models

Sep 2026 · arXiv.org · Vol abs/2609.34196 · 0 citations · 37 references
Computer Science

TL;DR

Results show that pretrained convolutional representations, when integrated through learned adaptation and fusion, can improve the visual understanding of existing VLMs without replacing their original visual encoders.

Abstract

Modern vision-language models (VLMs) achieve strong performance across a broad range of multimodal tasks, yet still struggle with visual questions that require fine-grained discrimination and spatial understanding. These limitations motivate investigating whether supplementary visual representations can improve existing VLMs without replacing their native visual encoders. Pretrained convolutional networks offer a candidate feature source, motivated by their local connectivity and spatial weight sharing. We introduce CONVCUE, which augments the native visual representations of a pretrained VLM with final-stage features from a parallel, frozen pretrained CNN. A learnable adapter maps convolutional features to the native visual feature dimension, while gated cross-attention allows the original visual tokens to retrieve information from the CNN features. The enhanced tokens are passed through the original visual-to-language projector, and the model is adapted through a two-stage training procedure. We evaluate CONVCUE on Qwen3-VL-2B, Qwen3-VL-4B, and LLaVA-OneVision-7B across 13 multimodal benchmarks covering visual question answering, document and chart understanding, and multimodal reasoning. CONVCUE improves average benchmark performance over both the original models and matched two-stage fine-tuning controls on all three backbones. On Qwen3-VL-4B, it improves over the original model on all 13 benchmarks and raises the average score from 75.00 to 78.82 relative to the matched fine-tuning control. These results show that pretrained convolutional representations, when integrated through learned adaptation and fusion, can improve the visual understanding of existing VLMs without replacing their original visual encoders.

View source

Similar papers

Preprint Aug 2026

VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision

VIVAS is proposed, a framework built upon the unified token space paradigm, which introduces a dense-structural-semantic vision tokenizer, which expands the textual vocabulary into a unified vision-language vocabulary by incorporating a visual vocabulary.

Zhe-Han Kan, Yu-Bo Zhu, Xing-Hua Jiang et al. · 1 citation
Conference Open access Sep 2026

Inductive Visual Logic for Few-Shot Out-Of-Distribution Adaptation in VLMs

Generative vision-language models (VLMs) such as Qwen-VL and LLaVA achieve strong zero-shot performance on tasks overlapping with their pretraining distribution, yet fail on specialized domains where the required discriminative features were never learned, a regime we term distant out-of-distribution (OOD). Standard ad...

Hung-Jen Chen, Yuek F. Ho, Ting-Yao Huang et al. · 0 citations
Preprint Oct 2026

AdaLoop: Thinking with the Vision Encoder

AdaLoop is presented, a framework for question-conditioned visual recurrence in VLMs that adapts visual computation to the question and reuses the encoder's transformations across multiple iterations, with the pretrained vision weights frozen.

Xinran Nicole Han, Yi-Yang Lu, Katelyn Gan et al. · 0 citations
Review

Toward Multimodal Understanding in Low-Level Vision: The Rise of Vision-Language Models

This paper presents the first comprehensive survey of VLM applications in low-level vision, proposing a unified taxonomy that organizes existing literature into two complementary paradigms, and identifies critical open challenges, including semantic-pixel misalignment and computational efficiency.

Feng-Yang Xiao, May May Liqeliqe Mar. Liqe Mar., Mariusz Dec et al. · 0 citations

Detection-Guided Attention Steering for Vision Language Models

A detection-guided dynamic attention steering system that leverages the locality insight from CNNs to efficiently steer a VLM’s attention toward more relevant sections of an image and demonstrates the effectiveness of combining different model architectures to harness their respective strengths for advancing VLM capabi...

Alan William Zhang, Rui Pan, Mike Wong et al. · 0 citations
#artificial intelligence Preprint Oct 2026

VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs

Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Down...

Yuan Feng, Qi-Ze Yang, Rui-Zhe Chen et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.