ProReGen: Progressive Residual Generation under Attribute Correlations

Abstract

Attribute correlations in the training data will compromise the ability of a deep generative model (DGM) to synthesize images with under-represented attribute combinations (i.e., minority samples). Existing approaches mitigate this by data re-sampling to remove attribute correlations seen by the DGM, using a classifier to provide pseudo-supervision on generated counterfactual samples, or incorporating inductive bias to explicitly decompose the generation into independent submechanisms. We present ProReGen, a progressive residual generation approach inspired by the classical Robinson’s transformation, to partial out from an image attribute x2 its component mx1 that is predictable by other image attributes x1, and the residual γ=x2-mx1 that is not. This simplifies the problem of learning a DGM gx1,x2 conditioned on correlated inputs, to learning g~x1,γ conditioned on orthogonal inputs. It further allows us to progressively learn g~ by first shifting the burden to abundant majority samples to learn g~x1,γ=0, and then expanding it with additional layers gres to resolve its difference to g~x1,γ using residual attribute γ on limited minority samples. On three benchmark datasets with varying strengths of attribute correlation and one dataset with natural attribute correlation, we demonstrate that ProReGen—with input orthogonalization and progressive residual learning—improved the correctness of minority generations compared to existing strategies.