Co-Attention Guided Multimodal Fusion for Robust 6-D Pose Estimation in Occluded Scenarios

Abstract

Achieving accurate and efficient object pose estimation is a key goal in computer vision. Most existing methods rely on controlled environments, limiting their effectiveness in complex, dynamic, and unstructured real-world scenarios, especially for novel objects, severe occlusion, or sensor noise. Recent studies show that integrating multimodal information can improve adaptability and accuracy, with RGB-D fusion gaining attention due to its low cost and accessibility. However, effectively combining geometric and semantic cues from RGB and depth particularly at local regions remains challenging. To address this, we propose a Co-attention Refinement Network using a neighborhood-weighted voting mechanism to construct collaborative attention for efficient RGB-depth fusion. A non-iterative global-local context aggregation strategy further mitigates fusion errors from modality differences and local inconsistencies. The key components include Co-attention Alignment and Fusion (CCAF) and Enhanced Global-Local Context Aggregation (EGCA), which enable adaptive feature alignment and fine-grained context modeling. Extensive experiments on LINEMOD, Occlusion-LineMOD, and YCB-Video demonstrate the effectiveness and generalization of our approach. Note to Practitioners—This paper addresses the challenge of accurate and robust 6D object pose estimation in real-world environments, where factors such as poor lighting, occlusion, background clutter, and sensor noise commonly affect perception systems. Existing approaches relying on either RGB or depth information alone often fail when faced with textureless or reflective objects. To overcome these limitations, we propose a multimodal fusion framework that exploits the complementary strengths of RGB and depth modalities through a novel neighbor-based cross attention mechanism. By adaptively integrating information across neighboring regions, our method enhances the reliability of feature fusion and reduces the negative impact of noisy or misleading background data. Additionally, we introduce a confidence estimation module to assess the quality of the predicted poses, which is useful for practical robotic applications that require high reliability. This system is particularly relevant for tasks such as robotic grasping, assembly, and object manipulation in industrial and service settings. Practitioners interested in deploying multimodal perception systems can benefit from the proposed lightweight and generalizable design that supports improved performance in cluttered, dynamic, or previously unseen environments.