PAIR: Prototype-Aware Inertia Rectification for Open-Vocabulary Multi-Label Recognition
Abstract
Open-vocabulary multi-label recognition (OV-MLR) aims to predict multiple labels beyond seen training categories by transferring knowledge from vision-language models. Existing methods typically rely on either raw patch-level matching or a single global token, which makes them vulnerable to seen-category bias under seen-only supervision. In particular, patch features tend to collapse onto a few competitive seen categories, causing predictions to be dominated by seen-category responses, and limiting generalization to unseen labels. To address this issue, we propose PAIR (Prototype-Aware Inertia Rectification), a unified framework that reduces seen-category bias from both local and global perspectives. First, PURE (Prototype Recovery with Uncertainty-aware Region Enhancement) reorganizes patch tokens into teacher-guided latent prototypes, and refines them via transport-based prototype recovery, yielding more transferable local representations. Second, AIR (Alignment Inertia Rectification) builds a prototype-conditioned global representation, and rectifies global alignment through hard-negative shaping and local-to-global consistency regularization. By jointly rectifying local prototype learning and global representation alignment, PAIR provides a more balanced basis for open-vocabulary prediction. Extensive experiments on MS-COCO and NUS-WIDE show that PAIR consistently outperforms the strong OV-MLR baseline, demonstrating its effectiveness across both benchmarks.