Jul 2026· Annual International ACM SIGIR Conference on Research and Development in Information Retrieval· 0 citations· 55 references
Computer Science
TL;DR
This work presents RegionSLM, a region-aware SLM designed to explicitly connect the question to its supporting regions, and curates ReDoc, a region-supervised corpus with 105k documents and 350k question-answer pairs obtained via a question-guided two-step filtering procedure.
Abstract
Real-world document question-answering that relies on screenshots, such as bills and forms, requires evidence that is often spatially localised and visually cluttered. However, most Screenshot Language Models (SLMs) encode the entire page holistically and rely on implicit attention to ''find'' relevant content, which limits both accuracy and efficiency. We present RegionSLM, a region-aware SLM designed to explicitly connect the question to its supporting regions. RegionSLM has two key components: (1) a patch-relevance router that learns a query–region relevance distribution, enabling the model to produce a box-free relevance prior at inference; and (2) Relevance-Guided Region Pooling (RGRP), a query-conditional attention–pooling module that aggregates dense features into a small set of region tokens, which preserves grounding signals while reducing computational overhead. To support training and evaluation, we further curate ReDoc, a region-supervised corpus with 105k documents and 350k question-answer pairs, obtained via a question-guided two-step filtering procedure. Extensive experiments on 12 datasets demonstrate that explicitly learning query–region relevance and pooling it into compact region tokens is an effective and practical recipe for document retrieval and understanding.
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Visual Question Answering (VQA), yet they continue to struggle with questions requiring precise spatial reasoning and fine-grained visual understanding. These limitations often manifest as object, attribute, and spatial hallucinations, where models generate confident but visually unsupported responses due to insufficient region-level and fine-grained visual grounding. To address this challenge, we propose ReVA, a region-aware VQA model that employs a frozen CLIP ViT-L/14 Vision Transformer (ViT) and a Qwen2.5-7B-Instruct large language model (LLM) connected through a dual bridge that aligns both whole-image and region-level representations with the LLM's embedding space. The image bridge maps final transformer block features into image tokens. The region bridge maps cropped features from enriched intermediate features across ViT blocks so early texture and later object cues are more evident, into K region tokens for every bounding box. ReVA uses a detector stack that supplies automatic zero-shot bounding boxes that are both question-agnostic and question-dependent, using RAM++ (Recognize Anything Model), spaCy, and Grounding DINO. The image tokens and region tokens are concatenated as an LLM prompt prefix to jointly encode scene-level context and fine-grained regional evidence when answering questions. Evaluated on VQAv2, MMBench, POPE, and SEED-Bench, ReVA achieves 82.85% mean F1 on POPE, compared with 81.14% for an image-token baseline without region tokens. These results demonstrate that explicit region-aware visual representations reduce object hallucination and improve the factual grounding of MLLMs.
Multimodal LLMs can see a document, but they often can't read it reliably. Small text, tables, visual cues, and topological elements still trip them up under direct visual inference, even when the page is already sitting in the model's context. Most document-VQA systems treat perception as fixed: they encode the page once, ask the question, and answer from whatever the model happened to extract in that single fast pass. We think document VQA needs slower, more deliberate perception: rather than answering from one fixed encoding, the model should spend a bit of extra compute at inference time working out what to look at next, and only then answer. We build this into \textbf{Q-Guide}, a small agent that reads a question, works out what evidence it is still missing, and calls targeted tool(s) to recover it---reading text where text is needed, zooming in where detail is needed, or grounding a region where position matters. On DocVQA2026 and Manga109, Q-Guide outperforms both direct prompting and recent multi-agent document systems ($65.0\%$ vs.\ $40.0\%$ on DocVQA2026, $32.4\%$ vs.\ $24.4\%$ on Manga109), and the improvement holds across three Claude backbones (Opus 4.6, Sonnet 4.6, and Opus 4.5). We find that accuracy scales with the perception budget---most of the gain appears within two to three deliberate rounds---and that the gain comes from directing perception to the right place, not from complex control logic: adding planners, routers, or multiple collaborating agents does not help.
Long-video question answering (QA) forces multimodal large language models (MLLMs) to work within a tight frame budget, so the choice of frames largely decides whether a question can be answered at all. The standard recipe scores every frame against the question with a pretrained image–text matching (ITM) model and keeps the top scorers. A fundamental mismatch underlies this recipe: ITM models are trained on short, concrete visual descriptions, while QA questions are interrogative and often involve abstract terms. Scored against the question alone, the ITM yields a near-random signal whenever the question is not a direct image–text match, such as one asking for the temporal order of scenes. In our LongVideoBench diagnostic analysis, the score collapses even on benchmark-provided answer-relevant frames, with more than half falling into a near-zero region —not because the encoder is faulty, but because it behaves exactly as it was trained to. We argue that this format mismatch should be absorbed at the two ends of the pipeline while the encoder itself stays frozen. On the input side, a type-conditioned routed pipeline reformulates each question into a single ITM-aligned description by selectively applying grounding, decomposition, and constrained synthesis. The ITM therefore receives exactly one description per frame, preserving the per-frame matching cost of a standard single-query baseline. On the output side, because the score distribution remains polarized and answer frames are scattered in time, we replace top- $K$ selection with a parameter-free Rosin threshold followed by a temporal maximal-marginal-relevance (MMR) step that uses frame positions alone. Across three benchmarks (LongVideoBench, Video-MME, MLVU) and and four backbones (Qwen2-VL, Qwen2.5-VL, LLaVA-OneVision, LLaVA-Video), the resulting training-free pipeline, RECAST, consistently outperforms recent frame-selection baselines without modifying the ITM encoder.
S. Han, Thang Vu, Junyeong Kim· IEEE Access· 0 citations
This work introduces Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed, and requires no OCR or region metadata at inference.
Yongxin Wang, Ruizhe Zhou, Yueling Tang et al.· 0 citations
Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity. Reference-based metrics (e.g., CIDEr and SPICE) and LLM-as-scorer protocols struggle to verify dense factual claims, while existing QA-based alternatives generally offer lower probe density, narrower domain coverage, or no explicit alignment between individual questions and segmented image regions. We introduce CapProbe, a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking. Each image is decomposed into coarse semantic regions covering both foreground and background elements; for every retained region, we generate multiple-choice questions spanning 10 semantic categories, forming a dense checklist of probed visual facts. Guided by a two-tier taxonomy of 37 L1 domains and 219 L2 sub-domains, CapProbe comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image. A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones, while density-based metrics penalize verbose yet uninformative captions. The protocol is cost-effective: by converting unconstrained scalar scoring into structured MCQ reading, it reduces open-ended scoring bias while remaining judge-conditioned and yields relatively stable model rankings under a fixed reader. Experiments on 13 VLMs show large Coverage gaps across models, a clear competency-efficiency trade-off, and failure modes that sparse or overlap-based evaluation often misses. The benchmark data, annotations, and evaluation code will be released soon.
Mouxiao Huang, Qi Yan, Borui Jiang et al.· 0 citations
High-resolution pixels and crop or zoom tools give multimodal large language models the ability to inspect an image, but they do not provide a reliable task-conditioned policy for deciding where to inspect. Q-CueGraph makes this decision explicit. It maps a question and an image representation to budgeted, coordinate-level observations for a frozen reader. Text-rich images use a reusable OCR/layout graph; natural-image search instantiates query-conditioned visual nodes behind the same selection, composition, and budgeting interface. Optional utility refinement learns which candidate crops the frozen reader can use from training-answer correctness, without region-box supervision. With a frozen Qwen2.5-VL-7B reader, Q-CueGraph reaches 0.833 accuracy on V*Bench versus 0.696 for full-image inference from a 19% image-area budget, and reaches 92% of full-image ANLS on InfographicVQA from about half the image area. Across six benchmarks, explicit observation is most valuable when evidence is localizable, the question discriminates its location, and resolution limits full-image reading.