Cross-modal bidirectional gating for pose-enhanced video action detection
TL;DR
BPMF is proposed, a mutual gating module that enables pose features to guide RGB attention toward action-critical regions while RGB features reweight pose responses to suppress unreliable structural cues and achieves superior results on JHMDB21.
Abstract
Video action detection requires simultaneous actor localization and action recognition across temporal sequences. Although recent RGB-based methods have achieved strong performance, they often struggle with appearance ambiguity, background clutter, and occlusion. Human pose provides complementary structural cues that are semantically meaningful and less sensitive to irrelevant background information. However, existing RGB-pose fusion strategies are typically oneway, using pose only to guide visual features while ignoring the fact that RGB appearance can also help assess the reliability of pose representations. In this paper, we propose Bidirectional Pose-RGB Modulation Fusion (BPMF), a mutual gating module that enables pose features to guide RGB attention toward action-critical regions while RGB features reweight pose responses to suppress unreliable structural cues. We integrate BPMF into a Mamba-based video action detector with a lightweight pose branch for 2D skeleton modeling. Experiments on JHMDB21 show that our method achieves 76.96% frame-mAP, outperforming both the RGB-only baseline by 4.00 percentage points and the unidirectional fusion variant by 2.84 percentage points. Ablation studies further confirm the effectiveness of the proposed bidirectional design.