Real-time acoustic-visual source localization and tracking in urban environments

Abstract

Urban environments pose significant challenges for real-time target localization and tracking due to reverberation, occlusions, and frequent GPS degradation. Visual-only systems often suffer from delayed target acquisition when line-of-sight is obstructed, while acoustic sensing provides passive, low-latency directional cues but lacks spatial resolution for persistent tracking. This work presents a real-time, edge-deployable multimodal framework that integrates acoustic sound source localization (SSL) with visual object detection and tracking to close this initialization gap. Acoustic cues are employed to actively steer a servo-mounted camera, reducing the visual search space and accelerating time-to-acquisition. Visual detections are performed using YOLOv8, while temporal consistency is maintained via BoT-SORT with camera motion compensation. An extended Kalman Filter (EKF) fuses acoustic DoA estimates and visual bounding box measurements to enable robust tracking under nonlinear motion and temporary visual occlusions.