RTNav is proposed, a simple but effective architecture that treats inference latency, asynchronous environment stepping, and bounded compute as explicit design considerations and improves the success rate and the Success weighted by Completion Time under real-time execution.
Abstract
Navigation in unknown environments to find unforeseen objects has become increasingly feasible with capable vision and language foundation models. However, these models also introduce non-negligible inference latency, which becomes an important concern when agents must operate continuously in the real world. Most state-of-the-art methods are still developed in synchronous simulators, where the environment waits for the agent to act and inference time is effectively free. As a result, agents are often designed around the sequential execution of perception, reasoning, and action, with little regard for time constraints. Under real-time execution, where wall-clock time counts towards the task budget, the inefficiencies of these architectures become clear. We show that recent zero-shot object navigation methods suffer consistent performance degradation under such realistic timing conditions. Motivated by this observation, we propose RTNav, a simple but effective architecture that treats inference latency, asynchronous environment stepping, and bounded compute as explicit design considerations. Evaluated on real-time variants of HM3D-v1, HM3D-v2, and HM3D-OVON, RTNav improves the success rate by up to 11% and the Success weighted by Completion Time by up to 5.1 points over prior work.
This paper presents a hierarchical fast-slow agent that turns what the robot has already seen into the object of deliberation in zero-shot object-goal navigation, and reaches the highest success rate among the zero-shot methods compared here.
Zhaochen Lan, Zhi Yang, Yuxiang Fu et al.· 0 citations
Vision-and-Language Navigation (VLN) enables embodied agents to follow natural-language instructions. However, route-level instructions commonly encode spatial priors, such as orientation, distance, and layout, that are not explicitly available from onboard sensing at deployment in open, GPS-denied environments. Benchmark performance under such interfaces therefore jointly reflects visual navigation ability and the use of route structure explicitly supplied by the task description. As a complementary formulation, we propose Vision-Only Long-Horizon Navigation (VoLN), which shifts route-relevant information from externally supplied instructions and global guidance to locally observable in-scene cues. In VoLN, goal views specify the destination, while route-relevant information is available only through locally observable in-scene cues that the agent must detect, interpret, and select online. We instantiate VoLN for aerial navigation through VoLN-UAV, a 7,210-episode benchmark that combines long-horizon goal-directed flight, continuous 3D motion, large viewpoint changes, and context-dependent beacon selection. We further provide VoLN-MLLM as an initial reference baseline. It aligns self-supervised visual features with a structured semantic space and predicts short-horizon waypoint segments from observation history, goal views, retrieved visual--semantic tokens, and proprioception. On the five-environment Test-Unseen split, it obtains success rates of 7.4%, 4.5%, and 1.8% on Easy, Normal, and Hard episodes, respectively. These results provide an initial evaluation of VoLN and reveal substantial remaining challenges in long-horizon evidence integration, cross-view goal matching, and closed-loop stability. Project page: https://admire-ljb.github.io/VoLN-UAV/
Jiabin Lou, Haopeng Wang, Yuanshuai Wang et al.· 0 citations
We present FlowPilot, a compact world-action model for real-time onboard UAV navigation from depth. Unlike map-then-optimize pipelines that require local reconstruction or end-to-end policies that lack explicit scene prediction, FlowPilot jointly denoises future depth observations and executable trajectories with flow matching. A dual-stream mixture-of-transformers couples video and action experts through shared attention, allowing future-scene prediction and trajectory generation to inform each other. At deployment, the model runs action-centrically and outputs only a trajectory. To ensure trackability, actions are parameterized as degree-7 Bernstein polynomials: the current state constrains the initial control points, and the network predicts five free control points, yielding C^2-continuous references with closed-form velocity, acceleration and jerk. FlowPilot is trained on a three-level depth pyramid spanning high-throughput simulation, photorealistic simulation, and real onboard data. In closed-loop simulation, it outperforms learning- and optimization-based baselines under increasing clutter and commanded speeds up to 8m/s. On a physical quadrotor, the full perception-to-action pipeline runs in under 18ms on a Jetson Orin NX and reaches 5.5m/s in cluttered indoor and forest environments using only onboard sensing and computation.
Runqing Wang, Ding Yu, Pengyuan Min et al.· 0 citations
Vision-Language-Action (VLA) models have achieved impressive performance on diverse embodied tasks. However, deploying VLA models on low-power onboard devices, such as the Jetson Orin, remains challenging due to their high computational complexity, which leads to substantial inference latency and low control frequency. Asynchronous inference can partially mask this latency by parallelizing action execution and subsequent inference, but it introduces two critical issues: perception-execution misalignment and long reaction time. In this paper, we propose Jetson-PI, a method for efficient VLA deployment on onboard devices via Foresight-Aligned Asynchronous Correction. To address misalignment, we train a lightweight future correction module that predicts future environment representation conditioned on committed actions, enabling the action expert to directly predict actions from the future time step. To reduce reaction time, we introduce confidence-based scheduling optimization that adaptively balances VLM and action expert invocations, complemented by system-level accelerations including CUDA graph reuse, GPU-resident intermediate buffering, and flow unrolling. Extensive experiments demonstrate that Jetson-PI achieves 8.66x and 5.41x improvements in control frequency compared with naive PyTorch and vla.cpp on NVIDIA Jetson Orin, while outperforming VLASH by 14.8\% in average success rate on the LIBERO benchmark. The code of our asynchronous algorithm is available on https://github.com/PKU-SEC-Lab/Jetson-PI, and our efficient llama.cpp-based inference engine is available on https://github.com/PKU-SEC-Lab/Jetson-PI-Edge.
Zebin Yang, Qi Wang, Yunhe Wang et al.· 1 citation
Existing visual navigation models often aim to develop foundation models that can generalize robot navigation across diverse platforms. However, many of these models are prone to collisions when deployed in complex indoor environments, particularly in structured layouts and narrow passages. To address this problem, we propose a depth image- and point-goal-conditioned navigation system, DevGRU. The proposed system employs an action predictor (AP) that generates collision-aware future trajectories, enabling effective avoidance of immediate obstacles. In conjunction with a collision predictor, the AP further compensates for errors accumulated in the goal pose estimation and proactively mitigates future deviations. To evaluate our method, we conducted experiments across nine different scenes and three state-of-the-art approaches - ViNT, NoMaD, and NavDP - as well as four additional variants of ViNT and NoMaD. In terms of navigation performance, DevGRU significantly outperforms ViNT and NoMaD by a large margin. In addition, the proposed model has a relatively small number of trainable parameters, resulting in the fastest inference time among the baselines, particularly outperforming NavDP by 7x in model size and 17x in inference time.
Kyung Min Han, Eunsom Kim, Young J. Kim· 0 citations
Safe map-free multi-robot navigation requires robots to make real-time decisions from partial and noisy local observations in dynamic, unstructured environments. Existing approaches often depend on prior maps, computationally intensive perception pipelines, or carefully tuned interaction models, which limit their robustness on resource-constrained platforms. This paper proposes LSTP-Nav, a lightweight, decentralized navigation framework built on LSTP-Net that maps stacked 2D LiDAR observations, goal information, and velocity feedback directly to action. LSTP-Net combines GRU-based temporal modeling with attention to extract interaction-critical spatiotemporal cues at low computational cost. We further introduce an HS reward to provide smooth, heading-aware safety feedback, and develop PhysReplay-SimLab to improve training effectiveness through local replay of near-failure interactions. Extensive simulation and real-world experiments show that LSTP-Nav outperforms representative baselines, achieving 98.6%–100.0% success in single-robot settings and 97.8%–99.0% success in 10-robot settings with 5–35 obstacles. The policy also remains robust to LiDAR degradation, nonconvex layouts, random seeds, and large-scale density tests, while achieving over 40 Hz onboard inference on a Raspberry Pi 3 Model B with zero-shot sim-to-real transfer. Overall, LSTP-Nav provides a practical, deployable solution for real-time, map-free multi-agent navigation on computationally constrained mobile robots. Note to Practitioners—This study addresses a pressing practical need in large-scale site inspection and outdoor search-and-rescue: enabling mobile robots with limited onboard computation to navigate safely and efficiently in complex, previously unseen environments without relying on pre-built maps. In real deployments, robots must make decisions from partial and noisy sensor observations while interacting with moving objects (e.g., pedestrians or teammates), and conventional map-based pipelines or computationally heavy planners can become unreliable or infeasible. To tackle these constraints, we propose a lightweight spatiotemporal navigation policy that directly converts raw LiDAR observations into motion commands, combining temporal memory with attention to focus computation on interaction-critical cues. In addition, a heading-stability objective and a physics-based replay-assisted training workflow improve behavior smoothness and reduce failure cases such as oscillation, collisions, and getting trapped in cluttered areas. Extensive simulation and real-world tests show that the proposed approach improves navigation success and safety over representative baselines while maintaining real-time performance on CPU-only platforms, making it suitable for practical deployment on resource-constrained robots in dynamic, unstructured environments.
Xingrong Diao, Zhi-Qiang Sun, Jianwei Peng et al.· IEEE Transactions on Automat...· 0 citations