Rethinking Multi-Modal Point Cloud Completion: Query-Aware Gating Attention and Gramian Volume Alignment

Abstract

Multi-modal point cloud completion aims to recover complete 3D geometric structures from partial observations by integrating auxiliary data. Although image-guided techniques are well-established, the potential of natural language as a source of high-level semantic cues remains under-explored. Therefore, effectively introducing natural language and synergizing it with the image modality to assist 3D structure recovery is key to breaking current performance bottlenecks. To address this, we propose Query-Aware Gating Gramian Alignment (QGGA), a framework for multi-modal completion. QGGA employs a novel query-dependent bidirectional attention mechanism to dynamically filter information flows between modalities and adaptively regulate fusion intensity. To facilitate alignment among the three heterogeneous modalities, we utilize the volume of the parallelotope spanned by multi-modal local features as a consistency metric, constraining geometric alignment by minimizing this volume. Finally, the fused and aligned features are fed into a coarse-to-fine decoder to first predict a coarse point cloud and then progressively recover the complete shape through stepwise upsampling and refinement. Furthermore, we construct ShapeNet-ViPC-Desc, a benchmark dataset enriched with detailed text descriptions. Experimental results demonstrate that QGGA achieves new state-of-the-art (SOTA) performance on this benchmark. The source code is freely accessible at https://github.com/whysq/QGGA.