Continual Low-Rank Adaptation Via Cumulative Unified Optimization.

Abstract

Low-Rank Adaptation (LoRA) enables foundation models to efficiently adapt to downstream tasks with minimal trainable parameters, yet it lacks the capability of continual learning (CL) from non-stationary data streams. While existing CL methods attempt to address this by storing and retrieving task-specific parameter modules, they suffer from non-trivial challenges of task identification and inconsistency in input-output feature mappings, resulting in suboptimal feature representations. Instead of keeping multiple task-specific parameter modules, we propose a novel continual LoRA mechanism that parameterizes all task-specific adaptations with a single adaptation module, naturally obviating the task identification and mapping inconsistency issues. To this end, we reformulate LoRA-based CL as a consistent feature mapping problem that mimics the behavior of the joint-training upper bound, wherein a unified adaptation parameter matrix is learned to simultaneously capture the input-output relationships established by all task-specific LoRAs. By solving this formulation, our approach facilitates approximate unified optimization over sequential tasks via progressively updating cumulative feature statistics, achieving efficient knowledge retention and renewal. Notably, the proposed arithmetic training mechanism can be applied both to the backbone of the foundation model for parameter-efficient fine-tuning and the classifier optimization, thereby enabling continual learning within an integrated architecture that closely approximates the joint learning paradigm. Extensive experiments across conventional and long-sequence class-incremental benchmarks demonstrate that our method achieves new state-of-the-art performance.