Safe Multi-Robot Coordination via VLM–LLM Reasoning and Reachability Analysis

Abstract

Safe coordination in heterogeneous machine-to-machine (M2M) robotic systems is difficult when robots differ in sensing capability, environmental awareness, and motion execution roles. This study presents a centralized safety aware M2M framework for cooperative goal-directed navigation in a heterogeneous mobile robot system composed of a vision-capable robot and a cameraless robotic vehicle. The objective is to guide the robots toward a detected goal region, such as a traversable target area or open door direction, while avoiding static and dynamic obstacles and preventing unsafe inter robot interactions. The central cooperation principle is shared perception: the vision-capable robot provides semantic environmental awareness through a centralized server, allowing the camera-less robot to act using this shared scene representation together with its own odometry, IMU, and state feedback. Both robots communicate with the server through an MQTT broker and continuously publish robot-state data, while the vision capable robot additionally transmits visual observations. A vision language model interprets the scene, and the extracted semantic information is converted into conservative geometric constraints, including obstacle regions, traversable areas, goal regions, safe corridors, and motion boundaries. A large language model supports high level task allocation reasoning by proposing robot specific navigation decisions, while deterministic controllers remain responsible for low-level execution. Before any command is issued, each candidate action is verified by a zonotope based reachability engine that checks obstacle avoidance, safe corridor containment, and inter-robot collision constraints. Only commands satisfying these reachability-based safety conditions are approved and transmitted to the corresponding robot. Online validation in clear-path and dynamic-obstacle scenarios demonstrates that the framework can approve safe motion, trigger conservative replanning or holding behavior, and preserve a strict separation between semantic reasoning and executable control. The proposed framework unifies shared semantic perception, broker-based M2M communication, cooperative task allocation, and formal reachability verification to support safer coordination of heterogeneous mobile robots with asymmetric sensing capabilities.