Skip to content

Principal-timestep Restricted Init via Sparse Matrix-decomposition in Flow-matching

Sep 2026 · 0 citations · 52 references
Computer Science

Abstract

Flow-matching diffusion models have recently emerged as a strong paradigm for high-fidelity visual generation. However, their prohibitively high fine-tuning cost limits scalability to downstream tasks. While Low-Rank Adaptation (LoRA) combined with spectral initialization has demonstrated accelerated convergence and improved performance in autoregressive language models by better aligning gradient directions, we find that it fails to deliver similar gains in diffusion fine-tuning, often yielding marginal or even negative improvements over vanilla LoRA.We attribute this discrepancy to a fundamental mismatch between LoRA's low-rank parameterization and the intrinsically high-rank gradients induced by the flow-matching objective. In particular, stochastic timestep sampling introduces directionally heterogeneous gradient signals across training steps, leading to misaligned updates under low-rank constraints.To address this issue, we propose Prism-LoRA,a Principal-timestep Restricted Init via Sparse Matrix-decomposition framework that improves gradient alignment during fine-tuning. Our method consists of two key components: (i) principal timestep selection, which restricts initialization gradients to a subset of dominant timesteps to suppress effective gradient rank, and (ii) principal channel filtering, which removes task-irrelevant channels, enabling the one-step spectral initialization gradient to better align with the long-horizon optimization trajectory. Extensive experiments demonstrate that our method consistently improves both convergence speed and final performance across multiple diffusion fine-tuning benchmarks, including subject-driven generation, controllable generation, and deblurring, achieving not only performance improvement but also earlier stages of convergence over baseline LoRA and other spectral-init methods.

View source

Similar papers

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Experiments show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT, and incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations.

Yi Ouyang, Shi-Wei Li, Hao-Zhao Wang et al. · 0 citations
Sep 2026

Spectral-Domain Anchor Learning for Scale-Robust Continuous Visual Fields

The Spectral-Domain Anchor Learning (SDAL) framework is proposed, which transitions from localized spatial gradient descent to explicit spectral-domain parameterization, providing a scalable foundation for large-scale computer vision and continuous spatial learning.

Xing-Ran Guo, Tiao-Jie Xiao, Jie Liu et al. · 0 citations
#machine learning Preprint Oct 2026

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA vari...

Yang-Ming Zhang, Ling Yi, Da-Tong Li et al. · 0 citations
#machine learning Preprint Oct 2026

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

Residual-Gradient Adaptation (ReGRA) is proposed, a two-stage method for making more effective use of a fixed rank budget and achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank.

Fan-Yao Li, Yu-Xi Liu, Xiao-Hao Liu et al. · 0 citations
Open access 2026

MoPDO: Mixture of PDE-Derived Operators for Parameter-Efficient Vision Fine-Tuning

Parameter-Efficient Fine-Tuning (PEFT) has become a widely used paradigm for adapting large vision models to downstream tasks. However, many existing PEFT methods rely on low-rank or linear transformations, which may restrict the flexibility of feature adaptation when downstream visual distributions differ from those s...

Zhao-Zhi Wang, Tong Zhang, Qi-Xiang Ye · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.