Unified Gradient Projection is proposed, which constrains parameter updates using reference gradients from language-balanced replay in a unified projection space and shows that combining gradient-level projection with data-level replay yields complementary gains in stability and plasticity.
Abstract
Large-scale pretrained ASR models such as Whisper exhibit strong multilingual capabilities. However, fine-tuning on low-resource languages often causes catastrophic forgetting. Although continual learning mitigates this issue, existing methods struggle to regulate cross-task interference in multilingual settings, where dominant languages bias optimization. We propose Unified Gradient Projection (UGP), which constrains parameter updates using reference gradients from language-balanced replay in a unified projection space. By equalizing per-language contributions in the projection, UGP reduces dominant-language bias and improves cross-lingual stability. We further show that combining gradient-level projection with data-level replay yields complementary gains in stability and plasticity. Across diverse low-resource language groups and model scales, UGP enables effective adaptation while substantially mitigating forgetting. On Whisper-large-v3, it achieves near-zero average forgetting.
This work addresses the challenge of building robust foundation models for underrepresented Central Asian languages by presenting GigaAM Multilingual, a Conformer encoder pre-trained on 2M hours of audio using a HuBERT-style objective, and introduces a cluster-level data balancing strategy during pre-training and a domain-aware sampling method during fine-tuning to mitigate head-language dominance.
Andrei Kuzmenko, A. Maximenko, Aleksandr Kutsakov et al.· 0 citations
Centroid Intervention Fusion is proposed, a projection fusion framework that consolidates multiple multilingual intervention projections into a single language-shared operator and outperforms the strongest prior pairwise intervention baseline by up to +3.3% across four model backbones.
The Onramp-Sequence Cross-Distillation (OSCD) is introduced, a post-training algorithm that projects high-resource reasoning trajectories into low-resource vocabulary subspaces during generative training rollouts via an integrated translator agentic loop, ensuring the stable and efficient translation of dynamically generated reference samples for fine-tuning.
Sean Gip Lim, William-Chandra Tjhi, Hai Leong Chieu· 0 citations
Current large language models (LLMs) often exhibit performance imbalances between dominant languages (e.g., English) and non-dominant ones due to the skewed distribution of pretraining data. A common strategy to address this issue is to enhance cross-lingual alignment, thereby facilitating non-dominant language processing. However, existing meth-ods typically rely on additional training objectives or language-specific parameters, which increase training complexity and cost. In this work, we propose a selective bidirectional language projection framework that enables efficient multilingual alignment and language shift using the intrinsic parameters. Specifically, we first identify the layers most sensitive to language projection between non-dominant and dominant languages through neuron activation analysis. We then perform sequential language projection within the selected layers by mapping non-dominant representations into the dominant language space and reverting them before generation. The bidirectional projection benefits the subsequent instruction tuning in non-dominant languages. Experiments on seven benchmarks demonstrate that our method remarkably enhances the performance of non-dominant languages. Further analyses indicate that our method learns better internal representations and exhibits strong generalization capabilities.
Junpeng Liu, Jiuyi Li, Kaiyu Huang et al.· Annual Meeting of the Associ...· 0 citations
Vision-Free Adaptation (VFA) is proposed, a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone.
Yixia Li, Yaqing Shi, Zhiwen Ruan et al.· Annual Meeting of the Associ...· 0 citations
This work proposes At-tribution Scores-based Soft Orthogonality Low-Rank Adaptation (ASO-LoRA), an effective and efficient framework that simultaneously facilitates knowledge transfer while mitigating catastrophic forgetting and achieving a balance between knowledge transfer and preservation.
Yitong Wang, Xue Han, Wenchun Gao et al.· Annual Meeting of the Associ...· 0 citations