Reliability-Aware Gradient Surgery for Test-Time Adaptation in Audio-Visual Speech Recognition

Abstract

Audio-Visual Speech Recognition (AVSR) has shown robustness in noisy environments, but dynamically varying modality reliability between audio and video causes severe distribution shifts and performance degradation. Existing AVSR methods have mainly relied on data-level restoration or representation-level fusion, but cannot continuously adapt model behavior under evolving test-time conditions. Although Test-Time Adaptation (TTA) provides a natural solution, conventional TTA methods are designed for unimodal settings, leading to noisy modality gradients that dominate the adaptation process. We propose Reliability-Aware Gradient Surgery (RAGS), a gradient-level TTA framework specialized for multimodal environments that estimates modality-wise gradient reliability and employs multiple update strategies. Experimental results demonstrate that RAGS effectively suppresses noisy gradients and consistently improves robustness under severe multimodal corruptions.