Multi-Modal Radio-Vision SLAM for ISAC: Prototype Design, Experimental Validation, and Beam-Management Applications

Abstract

Radio-based simultaneous localization and mapping (radio-SLAM) offers a promising solution for user equipment (UE) localization and radio feature map construction without external positioning infrastructure, with its feasibility already demonstrated through integrated sensing and communication (ISAC) prototypes. However, the performance of radio-SLAM is constrained by the sensing capability of wireless devices and irregular UE motion, necessitating the fusion of multi-modal sensors to harness complementary advantages in multi-modal SLAM. Critically, existing multi-modal SLAM methods are underdeveloped and do not leverage SLAM outputs for communication enhancement, nor have they been demonstrated in prototypes. To address these challenges, we propose a multi-modal SLAM framework integrated with SLAM-aided beam management, fusing radio measurements, inertial measurement unit (IMU) data, and stereo camera images to enhance localization robustness and mapping accuracy in both line-of-sight (LoS) and non-line-of-sight (NLoS) scenarios under varying lighting conditions. The resulting localization and mapping information facilitates beam tracking and blockage prediction. We further develop an ISAC prototype system capable of collecting synchronized multi-modal data in dynamic scenarios. Experiments demonstrate that the proposed SLAM achieves decimeter-level UE localization and radio map construction under both bright and low-light conditions. The beam management module enables precise beam alignment with an average beam direction angular error of 0.0403 rad and blockage prediction with an average detection rate of 84.33% across LoS and NLoS paths. The collected dataset and implementation are publicly released to support reproducible research in multi-modal ISAC.