Adaptive gradient-norm weighting for improved domain adversarial training

Abstract

Unsupervised domain adaptation (UDA) transfers knowledge from a labeled source domain to an unlabeled target domain by jointly optimizing source classification and domain-alignment objectives. A key practical challenge is selecting the trade-off coefficient λ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda $$\end{document}, which controls the strength of domain alignment. Fixed values and manually designed schedules are commonly used, but the appropriate balance can vary across datasets, transfer directions, and training stages. This paper proposes Adaptive Gradient-Norm Weighting (AWD), a lightweight scalarization mechanism that dynamically adjusts λ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda $$\end{document} using the ratio between the gradient norms of the classification and alignment losses. AWD introduces no additional learnable parameters and can be applied to UDA objectives of the form LCE+λLAlign\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {L}_{\textrm{CE}}+\lambda \mathcal {L}_{\textrm{Align}}$$\end{document}. We evaluate AWD within Deep Adaptation Network (DAN) and Domain-Adversarial Neural Network (DANN) on Office-31, Office-Home, DomainNet, and digit adaptation benchmarks using five independent seeds. Across the evaluated benchmarks, AWD consistently improves average performance over fixed and scheduled baselines, with the strongest gains on more difficult transfer settings. On Office-Home with DANN, AWD improves average accuracy over fixed λ=1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda {=}1$$\end{document} by +4.4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+4.4$$\end{document} percentage points, while avoiding manual per-task grid search. The method incurs only modest computational overhead. Overall, the results show that gradient-aware adaptive weighting is a simple, practical, and interpretable mechanism for balancing classification and alignment losses in UDA.