Sep 2026· IJCAI/ECAI· pp. 1667-1675· 0 citations· 37 references
Computer Science
TL;DR
This paper introduces a framework that modulates static textual embeddings into adaptable semantic guidance for RGB-D open-vocabulary 6D pose estimation, and designs a Semantic-Anchored Cross-view Fusion module that aggregates image features from the reference and query views using modulated semantics as an invariant anchor.
Abstract
Estimating the 6D pose of novel objects without CAD models or video sequences remains a challenging problem. Recent works explore text-driven approaches to address this challenge in an open-vocabulary manner. However, these methods typically treat text embeddings as static priors, which lack the flexibility to adapt to specific visual scenes, thereby limiting reliable cross-view correspondences. In this paper, we propose a framework that modulates static textual embeddings into adaptable semantic guidance for RGB-D open-vocabulary 6D pose estimation. Specifically, we introduce a Text Feature Modulation (TFM) module that uses learnable queries to transform fixed text embeddings into adaptive semantic representations. These representations are explicitly calibrated by a Foreground-Grounded Semantic Alignment (FGSA) strategy, which suppresses background distractions via contrastive learning. Furthermore, we design a Semantic-Anchored Cross-view Fusion (SACF) module that aggregates image features from the reference and query views using modulated semantics as an invariant anchor. Finally, we predict object segmentation and establish semantic-aware correspondences, leveraging modulated textual semantics to filter mismatches for robust 6D pose estimation. Extensive experiments on standard benchmarks demonstrate that our method achieves superior performance.
Virtual Reality (VR) content creators today face lengthy trial-and-error cycles when searching for 3D models, which relies heavily on manually annotated metadata. However, this approach struggles to capture the rich visual and dynamic details inherent to 3D models. Prior work has attempted to address this by rendering...
Sruti Srinidhi, Anthony Rowe· IEEE Transactions on Visuali...· 0 citations
Category-level 6D pose estimation from a single RGB-D observation is inherently under-constrained, since partial visible geometry must be interpreted together with a canonical object structure before a stable pose can be determined. We present LEGAU, a unified framework that jointly predicts NOCS correspondence, object...
Point cloud completion aims to recover complete 3D shapes from partial observations. Recent methods introduce single-view images as auxiliary guidance, yet visual features are bound to the camera viewpoint and carry little information about occluded regions. Text generated by a Vision-Language Model can complement imag...
Wang Luo, Di Wu, Heng-Yuan Na et al.· Proceedings of the 34th ACM...· 0 citations
Category-level 6D object pose and size estimation remains challenging due to severe intra-class geometric variation, which makes direct absolute prediction unreliable for unseen instances. In this paper, we reformulate the task as a similarity-driven retrieve-and-refine problem, where pose estimation is grounded in geo...
Jun-Yi Wang, Zhi-Kun Hu, Xiang Lei et al.· Proceedings of the 34th ACM...· 0 citations
Robots deployed in human-centered environments need maps that support both geometric navigation and open-vocabulary semantic queries beyond a fixed label set. A key challenge is enriching such maps with semantic information without relying on manually annotated segmentation masks or closed-set semantic categories. Exis...
Rodion Shkokov, T. Zemskova, Dmitry A. Yudin· International Joint Conferen...· 0 citations
Multimodal Large Language Models (MLLMs) have made substantial progress in 3D scene understanding from multi-view RGB observations. Yet a fixed object and query can elicit inconsistent target-related responses across camera viewpoints, a phenomenon we characterize as viewpoint-induced semantic drift. We introduce LieVL...
Jing-Jun Yi, Chen Hu, Qi Bi et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.