Efficient Invariant Representation Learning Based on Cauchy-Schwarz Divergence.

Abstract

Invariant risk minimization (IRM) has recently emerged as a promising method for invariant representation learning. The effectiveness of IRM relies on the assumption of the cross-domain overlap of invariant features. Some mainstream methods satisfy the cross-domain overlap assumption of invariant features by introducing the principle of the information bottleneck (IB) into IRM to compress feature information while ensuring the maximum correlation between the features and labels. However, these methods rely on extra neural networks to implement IRM and the IB, including conditional mutual information (CMI) and mutual information (MI) penalties. This means that extra neural networks will increase parameter complexity and instability in the network training process. Moreover, these methods presuppose data environment partitioning, rendering them challenging to generalize to continuous domain environments. To solve this problem, we propose a novel method for learning invariant representations, termed Cauchy-Schwarz invariant representation learning (CSIR). In CSIR, we implement CMI and MI by leveraging the classic Cauchy-Schwarz (CS) divergence without the need for extra neural networks. Moreover, the new regularization terms favor continuous random variables, which makes them amenable in a continuous domain environment and eliminates the requirement of presupposed domain environment partitions. We conducted experiments on five different datasets and demonstrated that our approach can learn invariant representations more efficiently.