Skip to content
Open access

Open-Vocabulary Object 6D Pose Estimation via Modulated Textual Semantics

Sep 2026 · IJCAI/ECAI · pp. 1667-1675 · 0 citations · 37 references
Computer Science

TL;DR

This paper introduces a framework that modulates static textual embeddings into adaptable semantic guidance for RGB-D open-vocabulary 6D pose estimation, and designs a Semantic-Anchored Cross-view Fusion module that aggregates image features from the reference and query views using modulated semantics as an invariant anchor.

Abstract

Estimating the 6D pose of novel objects without CAD models or video sequences remains a challenging problem. Recent works explore text-driven approaches to address this challenge in an open-vocabulary manner. However, these methods typically treat text embeddings as static priors, which lack the flexibility to adapt to specific visual scenes, thereby limiting reliable cross-view correspondences. In this paper, we propose a framework that modulates static textual embeddings into adaptable semantic guidance for RGB-D open-vocabulary 6D pose estimation. Specifically, we introduce a Text Feature Modulation (TFM) module that uses learnable queries to transform fixed text embeddings into adaptive semantic representations. These representations are explicitly calibrated by a Foreground-Grounded Semantic Alignment (FGSA) strategy, which suppresses background distractions via contrastive learning. Furthermore, we design a Semantic-Anchored Cross-view Fusion (SACF) module that aggregates image features from the reference and query views using modulated semantics as an invariant anchor. Finally, we predict object segmentation and establish semantic-aware correspondences, leveraging modulated textual semantics to filter mismatches for robust 6D pose estimation. Extensive experiments on standard benchmarks demonstrate that our method achieves superior performance.

Read PDF

Similar papers

Open access Oct 2026

Learning Visual and Motion Aware 3D Model Representations for Semantic Retrieval.

Virtual Reality (VR) content creators today face lengthy trial-and-error cycles when searching for 3D models, which relies heavily on manually annotated metadata. However, this approach struggles to capture the rich visual and dynamic details inherent to 3D models. Prior work has attempted to address this by rendering...

Sruti Srinidhi, Anthony Rowe · 0 citations
Sep 2026

LEGAU: Learning Semantic Gaussian Priors for Scalable Category-level Pose Estimation

Category-level 6D pose estimation from a single RGB-D observation is inherently under-constrained, since partial visible geometry must be interpreted together with a canonical object structure before a stable pose can be determined. We present LEGAU, a unified framework that jointly predicts NOCS correspondence, object...

Hong-Li Xu, Zhao-Wei Lu, Jun-Wen Huang et al. · 0 citations
Book Open access Nov 2026

Compose, Route, Complete: Text-Routed Vision Prototypes for Multimodal Point Cloud Completion

Point cloud completion aims to recover complete 3D shapes from partial observations. Recent methods introduce single-view images as auxiliary guidance, yet visual features are bound to the camera viewpoint and carry little information about occluded regions. Text generated by a Vision-Language Model can complement imag...

Wang Luo, Di Wu, Heng-Yuan Na et al. · 0 citations
Book Open access Nov 2026

Similarity-Driven Retrieve-and-Refine Category-Level 6D Object Pose Estimation via Geometry-Language Retrieval using Bidirectional Feature Fusion

Category-level 6D object pose and size estimation remains challenging due to severe intra-class geometric variation, which makes direct absolute prediction unreliable for unseen instances. In this paper, we reformulate the task as a similarity-driven retrieve-and-refine problem, where pose estimation is grounded in geo...

Jun-Yi Wang, Zhi-Kun Hu, Xiang Lei et al. · 0 citations
Open access Oct 2026

AOV-GS: Active Open-Vocabulary Semantic Mapping with 3D Gaussian Splatting

Robots deployed in human-centered environments need maps that support both geometric navigation and open-vocabulary semantic queries beyond a fixed label set. A key challenge is enriching such maps with semantic information without relying on manually annotated segmentation masks or closed-set semantic categories. Exis...

Rodion Shkokov, T. Zemskova, Dmitry A. Yudin · 0 citations
Preprint Nov 2026

Lie Routing Vision Language Models for View-Robust 3D Understanding

Multimodal Large Language Models (MLLMs) have made substantial progress in 3D scene understanding from multi-view RGB observations. Yet a fixed object and query can elicit inconsistent target-related responses across camera viewpoints, a phenomenon we characterize as viewpoint-induced semantic drift. We introduce LieVL...

Jing-Jun Yi, Chen Hu, Qi Bi et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.