RO-VLMap: Real-Time Occupancy-Aware Visual Language Mapping for Robust Robot Navigation

Abstract

Vision-and-Language Navigation (VLN) tasks require an agent to interpret natural language instructions and visual observations to navigate complex environments. Existing methods mostly construct topological or semantic maps and rely on the Large Language Model (LLM) for navigation decision-making, however, they still suffer from limited adaptability to dynamic environments and robustness to complex instructions, with task performance being limited by LLM performance. To overcome these limitations, we introduce the Real-time Occupancy-aware Visual Language Map (RO-VLMap). This framework quantifies the instantaneous risks posed by moving entities in the environment by fusing real-time occupancy sensing with visual language 3D reconstruction. Specifically, RO-VLMap, when combined with our proposed Navigation Adaptive Module (NAM), leverages the complementary advantages of the Knowledge Graph (KG) and LLM to parse open-vocabulary instructions into precise navigation goals. By planning over a unified occupancy-aware map, the agent proactively generates safe paths that avoid dynamic obstacles. Experimental results show that RO-VLMap significantly improves navigation success rates and efficiency. Furthermore, it demonstrates strong robustness in unseen scenarios, providing a practical solution for autonomous navigation of embodied agents in complex real-world environments.