Skip to content

Bilinear Optimization Divergence: Diagnosing Factor-Constrained LoRA Continual Learning

Sep 2026 · 1 citation · 33 references
Computer Science

TL;DR

The analysis and evidence provide an architecture-conditioned account of which constraint to enforce, how to enforce it, and how to interpret its empirical value, and show that stricter feasibility need not improve final task performance.

Abstract

Orthogonality in a LoRA factor does not by itself specify what the composed update protects: the answer depends on the task-start state, the parameterization, and the realized optimizer displacement. We formalize this question through Bilinear Optimization Divergence (BOD), an anchor-relative diagnostic of effective-update response on selected historical features. The finite-step analysis distinguishes two cases. In a shared adapter, protecting the routing displacement leaves a learned-anchor residual through the changing companion factor. In a fresh zero-output block, a feasible routing state can protect the composed update while both current factors remain trainable. These conditions yield Semi-Frozen Orthogonal Routing (SFOR) for shared adapters and current-block hard protection for cumulative O-LoRA; Weight Residual Projection (WRP) enforces the required displacement after the optimizer step. Controlled two-task traces verify the predicted residual paths, reducing normalized historical response from 19.12% to 0.005% in the shared family and from 7.72% to 0.002% in the cumulative family. Four-task experiments on Qwen3-8B characterize the resulting trade-offs: SFOR improves backward transfer (BWT) from -2.47 to -0.86 with nearly unchanged average accuracy (AA), while O-LoRA hard protection improves three-order mean AA from 80.27% to 81.30% and forgetting measure (FM) from 2.20 to 0.43. Component controls also show that stricter feasibility need not improve final task performance. Together, the analysis and evidence provide an architecture-conditioned account of which constraint to enforce, how to enforce it, and how to interpret its empirical value.

View source

Similar papers

Jacobian Rank Collapse in Decision-Focused Learning

Findings distinguish local rank restrictions, coordinate-dependent optimization and predictive accuracy, which helps explain available learning directions, while held-out decision quality remains the test of practical benefit.

Ao-Jie Yuan, Hai-Yu Zhang, Zi-Jian Su · 0 citations
#machine learning Preprint Sep 2026

Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation

FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates, shows that a carefully selected fixed span can recover most of the benefit of mov...

Wen-Song Ye, Zhan-Ming Shen, Zhiqing Xiao et al. · 1 citation · ⚡1
#machine learning Preprint Sep 2026

Reducing the Adaptation Gap Through Reachable Fisher Geometry

Parameter-efficient fine-tuning (PEFT) determines not only how many parameters are trained, but also which local directions a model can move in, so similar adapters can affect subgroup losses differently. Since curvature matrices are infeasible to form at adapter scale, scalar summaries such as the Fisher trace are oft...

Wasif Jalal, Sachin Deb, Asif Salekin · 0 citations
#machine learning Preprint Oct 2026

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

A historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training, and an orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly to the induced weight...

Jun-Kang Liu · 1 citation
Open access Sep 2026

DWAT: Density-Weighted Adversarial Training for Robustness Beyond the Training Perturbation Budget

Density-Weighted Adversarial Training is presented, a plug-in framework that spreads training over a set of sampled ϵ-balls and reweights each candidate adversarial example by a Gaussian density of its distance from the benign sample, and it is proved that the population objective upper-bounds the worst-case adversaria...

Jie-Ying Huang, Rui-Ming Zhu, Jia Xu et al. · 0 citations
Preprint Aug 2026

CG4AI: A Column Generation Framework for Training AI Models Under Constraints

This work proposes CG4AI, a framework that builds a convex combination of AI models while enforcing linear constraints on the combined output, and applies it to digit classification on MNIST and the multi-commodity flow problem, where link capacity constraints are enforced on neural-network routing predictors.

Youcef Magnouche, Abderrahmane Driouch, Sébastien Martin et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.