ReasonWalker: Reasoning Iterative Vision-and-Language Navigation With Implicit Instructions.
TL;DR
This article proposes ReasonWalker, a novel navigation model designed to enable reasoning-based navigation using implicit instructions over time, and proposes a new hierarchical learning paradigm, where the model first learns navigation actions and then acquires scene associations for implicit instruction reasoning.
Abstract
Existing vision-and-language navigation (VLN) agents typically cannot infer users' implicit intentions. They are unable to leverage past experiences in persistent environments. In this article, we propose ReasonWalker, a novel navigation model designed to enable reasoning-based navigation using implicit instructions over time. To ensure persistent and efficient operation, ReasonWalker constructs and stores explicit scene maps, allowing it to learn scene associations for improved renavigation in subsequent episodes. To facilitate comprehension and reasoning over implicit instructions, ReasonWalker leverages a large language model (LLM) to jointly process user instructions, agent observations, and scene maps, generating semantic navigation tokens that guide action prediction. To train ReasonWalker, we propose a new hierarchical learning paradigm, where the model first learns navigation actions and then acquires scene associations for implicit instruction reasoning. Additionally, we provide a new implicit instruction benchmark to support training and evaluation of reasoning-based navigation tasks. Extensive experiments demonstrate the effectiveness and superiority of the proposed ReasonWalker. The project page with video presentations and code is at: https://wangxudongsia.github.io/ReasonWalker-Web/.