Skip to content

From Text Decisions to Pixels: An Study of Jev-Style Visual Choice Model

Sep 2026 · arXiv.org · Vol abs/2609.29283 · 7 citations · 17 references
Computer Science

Abstract

Visual software often needs a decision over supplied alternatives rather than a generated explanation. We present PixelJev, a native-image decision interface that maps an image, a task instruction, and a runtime candidate set to a structured choice and candidate-conditioned probabilities using small open multimodal models. Its initial realization unifies recognition and multiplechoice visual question answering through an existing language-model readout, with separately evaluated options for frozen inference, language-side adaptation, and held-out calibration. Across seven benchmark evaluations, 64-shot source adaptation raises Pets accuracy from 60.13% to 92.40% across optimization seeds and transfers to natural resampling, new texture labels, and A-OKVQA without target fitting, while frozen inference already supports both VQA tasks. A matched prompt-only follow-up on Pets and ScienceQA attributes the large Pets gain to adaptation and identifies a narrower output validity benefit of candidate readout in adapted VQA. Specialist DINOv2 probes remain stronger on source recognition, frozen 4B is stronger than adapted 2B on DTD and ScienceQA, and accuracy gains do not ensure calibrated target probabilities. These findings establish a working starting point for general-purpose visual decision models and identify the remaining requirements: schema robustness, cross-family transfer, and reliable use of visual evidence.

View source

Similar papers

Visual Jev: Accurate and Efficient Decisions from Shared Visual Context

Many vision applications ask several independent, forced-choice questions about the same image. Visual Jev encodes the image and public context once, executes isolated question suffixes as a batch, and reads candidate probabilities from the backbone's language-model head. Across four benchmarks, answer-supervised post-...

Guan-Xu Yu, Yu-Hang Yao · 15 citations · ⚡1
Review Aug 2026

Which Source Wins? Task-Dependent Reliance in Vision-Language Models

Modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, models, and evaluation settings, which shows that modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, and evaluation settings.

Rodela Ghosh, Aviral Gupta, Guang-Jing Wang · 0 citations
#artificial intelligence Book Open access Sep 2026

When Helpful Text Hurts: Option-Redirecting Bias in Vision–Language Models

In tri-modal visual question answering (VQA), auxiliary text is commonly used to complement visual and textual inputs, yet its reliability is often uncontrolled. While prior work studies modality conflicts in general, it remains unclear how different types of unreliable auxiliary text affect answer selection under fixe...

Tam Le Thi Thanh, Van Tran Hoang, Hong-Hanh Nguyen-Le et al. · 0 citations
Preprint Oct 2026

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missin...

Vasco Ramos, Sandra Godinho Silva, João Magalhães et al. · 0 citations
Review Aug 2026

Benchmarking Frontier Text-to-Image Models on Image-Description Prompts

Failure analysis shows that the leading systems primarily lose points through object miscounting and geometric artifacts, whereas the trailing systems more frequently produce garbled text, and Ideogram 3.0 also frequently omits requested elements.

Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmed Rashad · 1 citation
#artificial intelligence Preprint Sep 2026

Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA

Every document QA system begins with a choice that is rarely studied on its own: whether to feed the model page images, extracted text, or both. We isolate this choice, holding the prompt, judge, and scoring pipeline fixed, across four commercial model endpoints, two corpora, and two context regimes (gold evidence page...

Nikhil Reddy Pottanigari, Sepideh Kharaghani, Saverio Vadacchino et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.