PTT-GGF Network: Robust Voice Activity Detection via Multimodal Gated Fusion in Complex Cockpit Environments

Abstract

Voice activity detection is a fundamental and critical component for automatic speech recognition and understanding applications in air traffic control. However, high-decibel, non-stationary noise in civil aviation cockpits severely degrades the signal-to-noise ratio, causing traditional single-modal voice activity detection algorithms to suffer from high false alarm rates and significant performance degradation. To address this critical issue and manage complex communication intentions, this paper proposes a Push-to-Talk Guided Gated Fusion (PTT-GGF) audio-visual multimodal architecture. First, we construct a temporally aligned Cockpit-AV multimodal dataset using a flight simulator, where a high-fidelity virtual Push-to-Talk logical prior is rigorously generated based on aviation Standard Operating Procedures to handle the highly sparse distribution of speech events during long-haul flights. Second, we design a confidence-aware cross-modal dynamic soft-gating strategy. Rather than simple feature concatenation, this mechanism jointly evaluates acoustic, visual, and Push-to-Talk logical states in real-time, adaptively shifting the decision focus to noise-immune visual features under high acoustic degradation. Experimental results demonstrate that the proposed PTT-GGF effectively mitigates the majority-class bias inherent in highly imbalanced data, maintaining a stable F1-Score of 95.09% under a 1:47 positive-to-negative sample ratio. Furthermore, stress tests reveal its robust performance under extreme conditions, including a -15dB signal-to-noise ratio, dynamic temporal misalignments, and transmission frame losses. With a core fusion module real-time factor of 0.000159, our method significantly outperforms mainstream audio-only and cascaded schemes, providing a highly reliable and low-latency detection paradigm for smart cockpit monitoring systems.