ShapeNav: 3D Geometric Perception and Reasoning for Multi-Floor Vision-and-Language Navigation
Abstract
Enabling navigation agents to follow human instructions and reach target locations in complex environments is a fundamental requirement for embodied intelligence. Despite recent advances in Vision-and-Language Navigation (VLN) that leverage Vision-Language Models (VLMs) for environmental and semantic understanding, two critical limitations persist in multi-floor VLN tasks: (1) token-level representations lack explicit 3D geometric perception, and (2) 2D pre-trained VLMs fail to extract effective visual features from 3D inputs. To address these challenges, we propose ShapeNav, a shape-aware navigation agent that integrates geometric perception while directly utilizing VLMs. First, we construct a Floor-to-Floor (F2F) dataset based on the Matterport simulator for multi-floor VLN tasks and generate corresponding shape-instruction pairs. Second, we introduce a geometric patch extraction strategy to explicitly align spatial patches from input images and pointmaps, achieving cross-domain feature alignment. Specifically, the Masked Optimal Transport (MOT) module aligns shape-aware features with the image modality space, which are then fed into a ViT to derive geometry-enhanced visual features. Third, these features are aligned with shape instructions through Q-Former for geometry–language alignment, followed by a graph-based action selector to determine the next step. ShapeNav achieves SOTA performance on the F2F dataset (SR ↑ 12% and SPL ↑ 9%), demonstrating the effectiveness of incorporating 3D geometric perception into navigation tasks.