Skip to content

NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning

Sep 2026 · 0 citations · 35 references
Computer Science

TL;DR

NeuronEye, a plug-in framework that constructs a sparse, concept-level neuron vocabulary from intermediate VLM representations and selectively activates query-relevant visual concepts during inference, suggests that sparse neuron vocabularies can serve not only as post-hoc interpretability tools but also as active interfaces for concept-level visual reasoning.

Abstract

Current vision-language models (VLMs) encode visual information in dense hidden states where object identity, spatial layout, and local attributes are implicitly entangled rather than explicitly disentangled, limiting their ability to isolate and modulate the specific visual evidence required by a given language query. Inspired by sparse population coding and top-down modulation in biological vision, we introduce NeuronEye, a plug-in framework that constructs a sparse, concept-level neuron vocabulary from intermediate VLM representations and selectively activates query-relevant visual concepts during inference. NeuronEye decomposes vision-token states into an overcomplete sparse basis organized by concept-level clusters, uses the language query to activate relevant clusters and localize the patches where selected concepts are expressed, and injects the focused evidence back into vision tokens. A complementary suppression mechanism attenuates dominant perceptual directions to preserve weaker but relevant cues. All operations run in a single forward pass over a frozen VLM backbone. On Qwen2.5-VL-7B, NeuronEye raises CV-Bench overall accuracy by +3.1 with gains of +9.5 on Distance, and improves BLINK Multi-view by +8.3, with similar trends on LLaVA-1.6-7B. These results suggest that sparse neuron vocabularies can serve not only as post-hoc interpretability tools but also as active interfaces for concept-level visual reasoning.

View source

Similar papers

Preprint Aug 2026

State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Models

Experiments on document parsing benchmarks show that SCVER improves robustness under reduced input resolution and achieves a better accuracy-efficiency trade-off, demonstrating the effectiveness of on-demand visual evidence retrieval for fine-grained perception.

Ming-Xu Chai, Chen-Yu Liu, Zi-Yu Shen et al. · 0 citations
#small language model Preprint Aug 2026

Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information

This work introduces Visual Retrieval Heads (VRHs), a small subset of attention heads that are causally responsible for grounding text descriptions to image regions, and shows that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads.

Chanho Park, Daehyeon Choi, Jihyun Lee et al. · 1 citation · ⚡1
#machine learning Preprint Sep 2026

OmniMoE-VL: A Sparse Vision-Language Model with Coupled Visual-Depth Routing

OmniMoE-VL, a sparse VLM with a coupled visual-depth routed projector, enables question-dependent visual access while preserving the native visual-token sequence, and complements token-level expert routing in the vision and language stacks.

Long Qian, Bing-Ke Zhu, Jia-Qi Wei et al. · 0 citations
Preprint Aug 2026

GaussVLA: Geometry-Aware Spatial Reasoning for Vision-Language-Action Model

GaussVLA is proposed, a Mamba-based VLA that incorporates two custom modules: Gaussian Spatial Tokenizer (GST) to lift frozen semantic and depth features into compact 3D Gaussian tokens, and Depth-Aware Chain-of-Thought (DA-CoT) that performs structured, non-autoregressive geometric reasoning under language and flow-ti...

MD SELIM SAROWAR, Md Tanvir Islam, Sungho Kim et al. · 1 citation
#natural language process... Preprint Oct 2026

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured relations required for such reasoning. We introduce EICS (Evaluation-Invariant Concept Stability), a metric for testing whether representations consi...

Shubh Garg, Sha-Ni Chen · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.