Dynamic Fusion of Camera and Multi-UWB Radar Signals Measurements for Robust Noncontact Heartbeat Estimation

Abstract

Noncontact vital sign measurement is crucial for improving health monitoring instruments, demonstrating high accuracy while ensuring user convenience and comfort in clinical, home, and remote environments. While camera-based remote photoplethysmography (rPPG) and ultrawideband (UWB) radar micromotion sensing offer promising approaches, each faces limitations from environmental factors—cameras from illumination and skin variability, and radars from motion artifacts and multipath effects. In this article, we propose a dynamic deep fusion approach for integrating synchronized RGB and RF signals, combining cross-modal complementary features to reconstruct PPG waveforms under realistic conditions. We also introduce a multimodal instrumentation benchmark dataset—the first combining visual and multitype UWB radar measurements (one FMCW and two impulse-radio UWB (IR-UWB) for sensing diversity) with ground-truth PPG—collected from 17 subjects at varying distances (0.5–1.5 m), angles (−45° to +45°), and activities (resting and phone use). The dataset supports future evaluation across varying sensing conditions and subject behaviors. In this study, the proposed fusion framework is evaluated on a subset corresponding to the nominal sensing configuration (0.5-m distance, 0° viewing angle, and resting condition), our fusion system achieves an MAE of 0.1029, an mean squared error (mse) of 0.0152, and a heart-rate (HR) estimation error of $2.651~\pm ~2.430$ BPM, surpassing unimodal and static fusion baselines. These results demonstrate improved signal fidelity and noise robustness, supporting applications in advanced instrumentation, telemetry, and pervasive monitoring systems. The dataset and code are available at: https://github.com/NguyenVanKhai2412/Camera-UWB_Radars-Fusion-VitalEstimation.git