Skip to content

RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection

Sep 2026 · ACM International Conference on Multimedia (ACM MM) · 0 citations
Computer Science

TL;DR

RankGround is a two-stage framework that achieves accurate GUI grounding with a single VLM call per query and achieves 1.4 times faster inference and improves localization accuracy by 5.5% on average over the second-best method across all backbones and screen scales, establishing a new state of the art in both efficiency and precision for GUI grounding.

Abstract

Graphical User Interface (GUI) grounding is a fundamental perception task for multimodal agents, enabling them to interpret natural language instructions and interact with digital interfaces. Existing methods face a fundamental trade-off between accuracy and efficiency: direct full-image inference often fails to capture small or visually similar UI elements, while multi-crop strategies improve localization at the cost of multiple expensive Vision-Language Model (VLM) calls per query. To address this challenge, we propose RankGround, a two-stage framework that achieves accurate GUI grounding with a single VLM call per query. Central to our approach is GroundRanker, a lightweight multimodal reranker that identifies the most promising crop from a dense candidate set. Because no off-the-shelf ranking dataset is available, we construct ranking supervision data from existing grounding datasets. A strict containment criterion and boundary-aware positive augmentation improve alignment and spatial coverage in cluttered layouts. GroundRanker is then trained with a two-stage curriculum: a pointwise objective first learns coarse containment, and a listwise objective refines subtle semantic and spatial distinctions among visually similar crops. Experimental results show that RankGround consistently outperforms strong baselines while reducing computational cost. It achieves 1.4 times faster inference and improves localization accuracy by 5.5% on average over the second-best method across all backbones and screen scales, establishing a new state of the art in both efficiency and precision for GUI grounding.

View source

Similar papers

Sep 2026

Enhancing target identification and query discrimination for visual grounding

FSG-AID is presented, which integrates fine-grained semantic guidance with an attribute-aware iterative decoder and jointly exploits visual and language features to mine attribute semantics, initialize the target query, and iteratively refine the target representation.

Xiya Bu, Yu Liu, Jizhe Yu et al. · 0 citations
#artificial intelligence Preprint Aug 2026

Cost-efficient Active Learning for Referring Image Segmentation and Grounding

This work formulates active learning for VG under the realistic setting where only raw images are available without accompanying text, and introduces Referred Region Ambiguity, a new acquisition function that measures whether the model's confidence collapses onto a single region or disperses across multiple candidates.

Junbeom Hong, Seonghoon Yu, Hyungsik Jung et al. · 0 citations
Preprint Sep 2026

Beyond Similarity: Foundation Models as an Efficient Backbone for Training-Free Composed Video Retrieval

Results show that adaptively orchestrating foundation-model capabilities can combine scalable retrieval with fine-grained reasoning without task-specific training, and are shown to be state-of-the-art among training-free approaches.

Dmitry Demidov, Muhammad Zaigham Zaheer, Omkar Thawakar et al. · 0 citations
Preprint Sep 2026

QCPruner: Query-Conditioned Population Coverage for Visual Token Pruning

QCPruner fuses two cross-modal cues into utility and applies it to both visual targets and candidate representatives within visual-affinity-based coverage, which achieves the highest average relative performance among evaluated complete-system pruning methods at every reported token budget.

Sheng-Li He, Yong-Chao Liang, Rou-Meng He et al. · 2 citations
Sep 2026

DeCo: Efficient Decouple-to-Couple Learning for Multi-Task Visual Grounding

Multi-task visual grounding requires models to jointly understand linguistic semantics and perform accurate visual localization and segmentation. Despite the success of multimodal large language models, effectively adapting them to multiple grounding objectives remains challenging. Existing methods commonly enforce tas...

Xiaoqiang Lu, Li-Cheng Jiao, Long Sun et al. · 1 citation
Preprint Sep 2026

Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding

Recent zero-shot 3D visual grounding methods leverage vision-language models (VLMs) to localize objects in 3D scenes from natural language queries. However, these methods typically rely on heuristic rules to select which camera views are provided to the VLM, often prioritizing object visibility rather than grounding re...

Tsung-Chih Chiang, Hsuan-Kung Yang, Jou-Min Liu et al. · 0 citations

Related blog posts

Microsoft Research Blog Jul 8, 2026

Flint: A visualization language for the AI era

Short chart specifications are easy to write, but often produce uninspiring results. Flint is an open-source visualization language that offers a middle path, letting AI agents create expressive charts from compact, human-editable specifications. The post Flint: A visualization language for the AI era appeared first on Microsoft Research.

MIT News · Artificial Intelligence Jun 3, 2026

MIT researchers teach AI models to interpret charts

The new ChartNet training dataset could improve the accuracy of vision-language models that help analyze business trends or interpret scientific figures.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.