Flow-matching diffusion models have recently emerged as a strong paradigm for high-fidelity visual generation. However, their prohibitively high fine-tuning cost limits scalability to downstream tasks. While Low-Rank Adaptation (LoRA) combined with spectral initialization has demonstrated accelerated convergence and improved performance in autoregressive language models by better aligning gradient directions, we find that it fails to deliver similar gains in diffusion fine-tuning, often yielding marginal or even negative improvements over vanilla LoRA.We attribute this discrepancy to a fundamental mismatch between LoRA's low-rank parameterization and the intrinsically high-rank gradients induced by the flow-matching objective. In particular, stochastic timestep sampling introduces directionally heterogeneous gradient signals across training steps, leading to misaligned updates under low-rank constraints.To address this issue, we propose Prism-LoRA,a Principal-timestep Restricted Init via Sparse Matrix-decomposition framework that improves gradient alignment during fine-tuning. Our method consists of two key components: (i) principal timestep selection, which restricts initialization gradients to a subset of dominant timesteps to suppress effective gradient rank, and (ii) principal channel filtering, which removes task-irrelevant channels, enabling the one-step spectral initialization gradient to better align with the long-horizon optimization trajectory. Extensive experiments demonstrate that our method consistently improves both convergence speed and final performance across multiple diffusion fine-tuning benchmarks, including subject-driven generation, controllable generation, and deblurring, achieving not only performance improvement but also earlier stages of convergence over baseline LoRA and other spectral-init methods.
It is shown how pretrained information reshapes the implicit bias and training dynamics, and can thereby reduce the sample complexity of recovering the target parameters and support, by gradient-based fine-tuning.
A.-E. Decleves, Etienne Boursier, Nicolas Flammarion· arXiv.org· 0 citations
Experiments show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT, and incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations.
Yi Ouyang, Shi-Wei Li, Hao-Zhao Wang et al.· arXiv.org· 0 citations
The Spectral-Domain Anchor Learning (SDAL) framework is proposed, which transitions from localized spatial gradient descent to explicit spectral-domain parameterization, providing a scalable foundation for large-scale computer vision and continuous spatial learning.
Xing-Ran Guo, Tiao-Jie Xiao, Jie Liu et al.· International Journal of Com...· 0 citations
Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA vari...
Yang-Ming Zhang, Ling Yi, Da-Tong Li et al.· 0 citations
Residual-Gradient Adaptation (ReGRA) is proposed, a two-stage method for making more effective use of a fixed rank budget and achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank.
Fan-Yao Li, Yu-Xi Liu, Xiao-Hao Liu et al.· 0 citations
Parameter-Efficient Fine-Tuning (PEFT) has become a widely used paradigm for adapting large vision models to downstream tasks. However, many existing PEFT methods rely on low-rank or linear transformations, which may restrict the flexibility of feature adaptation when downstream visual distributions differ from those s...
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026