Analyzing the Optimization Conflict Between Image- and Text-Based Person Re-Identification

Abstract

Unifying image-based (I2I) and text-based (T2I) person re-identification (ReID) within a single model is complicated by a fundamental conflict between their training objectives. While I2I ReID focuses on identity-level invariance across images of the same person captured by multiple cameras, T2I ReID is driven by instance-specific textual descriptions tied to unique visual traits. When both objectives are optimized on a shared vision encoder, they impose opposing requirements on the same embeddings - I2I contracts intra-identity distances while T2I requires them to persist - so improving one task's representation actively degrades the other's. This paper analyzes this optimization conflict and shows empirically that the two objectives are not jointly satisfiable by a single unconstrained encoder. Motivated by this finding, we investigate the conflict through a controlled two-stage pipeline: a shared vision encoder is first trained for I2I invariance, then frozen while only the text encoder is aligned to the fixed visual space, preventing the T2I objective from unwinding the identity-level structure already learned. We conduct extensive experiments across multiple configurations, varying domain-mixing procedures, learning strategies, and task objectives. We observe that I2I pre-training improves subsequent T2I performance, while textual supervision during this pre-training further improves T2I performance, although the vision encoder is frozen during subsequent cross-modal training. We believe these insights provide a meaningful step toward unified ReID systems that reconcile identity-level and instance-level supervision within a single representation.