Skip to content
Book Open access

ShapeNav: 3D Geometric Perception and Reasoning for Multi-Floor Vision-and-Language Navigation

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 45 references

Abstract

Enabling navigation agents to follow human instructions and reach target locations in complex environments is a fundamental requirement for embodied intelligence. Despite recent advances in Vision-and-Language Navigation (VLN) that leverage Vision-Language Models (VLMs) for environmental and semantic understanding, two critical limitations persist in multi-floor VLN tasks: (1) token-level representations lack explicit 3D geometric perception, and (2) 2D pre-trained VLMs fail to extract effective visual features from 3D inputs. To address these challenges, we propose ShapeNav, a shape-aware navigation agent that integrates geometric perception while directly utilizing VLMs. First, we construct a Floor-to-Floor (F2F) dataset based on the Matterport simulator for multi-floor VLN tasks and generate corresponding shape-instruction pairs. Second, we introduce a geometric patch extraction strategy to explicitly align spatial patches from input images and pointmaps, achieving cross-domain feature alignment. Specifically, the Masked Optimal Transport (MOT) module aligns shape-aware features with the image modality space, which are then fed into a ViT to derive geometry-enhanced visual features. Third, these features are aligned with shape instructions through Q-Former for geometry–language alignment, followed by a graph-based action selector to determine the next step. ShapeNav achieves SOTA performance on the F2F dataset (SR ↑ 12% and SPL ↑ 9%), demonstrating the effectiveness of incorporating 3D geometric perception into navigation tasks.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.