Task-Oriented Navigation with Semantic Perception for Monocular Camera-Based Mobile Robots

Integrated FSM and Visual Servoing for Autonomous Object Search in Indoor Mobile Robot

Autonomous mobile robots require reliable coordination among navigation, perception, tracking, and precision approach modules to complete indoor object-search missions. Existing systems often remain fragmented, treating navigation, detection, tracking, and docking as separate tasks rather than as an end-to-end pipeline. This study has two objectives: to validate a fault-tolerant coordination architecture for autonomous search-and-approach behavior and to compare two search strategies under controlled indoor target-position scenarios. The primary contribution is a methodological integration framework based on a Finite State Machine (FSM) that coordinates ROS 2 Navigation2 global navigation, YOLO11n object detection, centroid tracking, and Image-Based Visual Servoing (IBVS), while managing transitions among navigation, visual servoing, recovery, and mission-completion states. A quantitative Gazebo simulation experiment used 40 controlled trials to compare Random Exploration and Waypoint-Based Search. The integrated system achieved a 100% mission success rate without command conflicts, indicating effective FSM-based coordination between global navigation and local visual control. Waypoint-Based Search was more efficient when the target was aligned with predefined nodes, achieving a mean detection time of 50.55 s compared with 164.80 s for Random Exploration. Conversely, Random Exploration performed better when the target was away from predefined paths, reducing mean detection time to 87.00 s compared with 183.64 s. Fault-tolerant behavior was demonstrated in simulation through successful mission completion despite repeated LiDAR-triggered obstacle-recovery events during visual approach. These findings show that search efficiency depends on alignment between exploration design and spatial structure, not universal strategy superiority.

Experimental validation of vision-based hybrid navigation for differential-drive robots

Reliable mobile-robot deployment requires mapping, localization, global planning, and real-time obstacle avoidance to operate consistently under sensor noise and physical constraints. This study presents a vision-based hybrid navigation framework for a differential-drive mobile robot in a warehouse-like environment. The main contribution lies in the system-level integration and experimental validation of established techniques rather than the development of a new standalone navigation algorithm. A ceiling-mounted camera converts top-view images into an occupancy-grid map and world coordinates, while a convolutional neural network (CNN) recognizes goal markers. Odometry–augmented reality University of Cordoba (ArUco) fusion is used to correct accumulated localization drift. Particle swarm optimization (PSO) generates smooth global paths offline, whereas the dynamic window approach (DWA) performs real-time local motion control. Experiments in a 3.3 m × 2.4 m workspace achieved average obstacle-localization errors of 0.959 cm and 0.696 cm along the x- and y-axes, respectively, and a goal-recognition accuracy of 99%. The DWA controller required 12.61±2.42 ms per cycle, while rapidly-exploring random tree (RRT) and PSO required 4–7 s and 477–692 s, respectively, for global path generation. Simulation and physical experiments confirmed collision-free navigation and successful quick response (QR)-code-based goods inspection, demonstrating the feasibility of the proposed framework for small, structured indoor environments.

Depth-Augmented Pose Tracking for Autonomous Robots: A Filter-based Multi-Sensor Fusion Approach

Robotic platforms operating in GPS-denied environments require robust ego-motion estimation systems that fuse complementary sensor modalities under onboard computational constraints. This paper proposes a navigation framework estimating six-degree-of-freedom (6 DoF) robot pose in unstructured scenes using a monocular camera stream, inertial measurement unit (IMU) data, and sparse depth cues within the multi-state constraint Kalman filter (MSCKF) architecture. The key innovation integrates 3D landmark measurements into visual feature tracks, reducing positional uncertainty and drift accumulation compared to vision-only approaches. The method is efficient enough for resource-constrained systems such as micro aerial vehicles and small ground robots. The measurement fusion strategy is analytically derived and evaluated on aerial robot trajectory datasets. Results show improved tracking accuracy and stability in challenging indoor and outdoor scenarios without GPS, enabling prolonged autonomous missions in complex 3D environments with real-time pose feedback and low computational burden.

Time-to-Collision Based Dynamic Obstacle Avoidance Using Pretrained Vision Models for Robots in Unstructured Environments

Dynamic obstacle avoidance in unstructured outdoor environments remains a critical challenge for autonomous mobile robots, particularly when large-scale robot-specific training data and simulation-based policies are impractical. We present a data-efficient, interpretable method for vision-based dynamic obstacle avoidance that operates entirely on real-world data, avoiding the sim-to-real transfer problem inherent in simulation-trained policies. Our approach leverages UniDepth, a large pretrained monocular depth estimation model, to produce dense depth maps from RGB video without requiring stereo cameras or LiDAR at inference time. Dynamic obstacle avoidance is achieved by extending the SuperPoint and SuperGlue feature correspondence pipeline to track keypoints across long frame sequences, projecting their 2D pixel-space positions into 3D using camera intrinsics and predicted depth, running bundle adjustment initialized from these 3D keypoints, and computing per-keypoint time-to-collision (TTC). A 2D motion primitive in the ground plane is then selected to move the robot away from the closest point of approach of the minimum-TTC keypoint. Evaluated on real-world data from the M3ED dataset, our pipeline achieves a precision of 0.49 and a recall of 0.38 in identifying frames with a ground truth TTC below 1 second, and correctly generates the evasive motion direction in 84\% of true positive detections. Crucially, it detects at least one frame with TTC less than 1 second for 20 out of 22 unique physical obstacles present in our test sequences. Unlike end-to-end learned methods that demand thousands of hours of robot-specific training data, our approach eliminates model training entirely, requiring only 74 seconds of data for hyperparameter tuning. This demonstrates exceptional data efficiency while preserving interpretable and generalizable behavior across diverse obstacle types.

Crowd navigation in a multi-room environment: a model predictive control framework for mobile robots

Mobile robots operating in human-populated environments must navigate complex, multi-room spaces while ensuring safety, i.e., generating collision-free motion. In this study, we present a sensor-based model predictive control (MPC) scheme designed for safe crowd navigation in such non-convex environments. The proposed framework decomposes the free space into a set of overlapping convex regions to construct a topological graph, enabling a high-level planner to compute optimal sequences of traversable areas. To effectively perceive the crowd, the system employs a robust perception pipeline that fuses 2D LiDAR data with semantic information from an RGB-D camera, utilizing Kalman filters (KFs) to estimate and predict human motion. These predictions are integrated into an MPC controller which generates robot commands by enforcing safety through discrete-time control barrier function (DT-CBF), ensuring that the robot avoids collisions while remaining within navigable regions. The approach is validated through high-fidelity simulations and real-world experiments using the TIAGo mobile manipulator. The results demonstrate that integrating vision-based semantic data with geometric constraints significantly improves collision avoidance and success rates in cluttered, multi-room scenarios.

Multi-Phase Vision-Based Navigation and Inspection for Legged Robots with Online Goal Refinement and Vision-Only Halting