Skip to content
Book Open access

When Modalities Compete: Modality-Aware Optimization on Low-Rank Manifolds

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 45 references

Abstract

Parameter-efficient fine-tuning like low-rank adapters is widely used to adapt multimodal large language models to downstream tasks, yet we identify an unaddressed failure mode called Non-Textual Norm Collapse (NTNC). NTNC causes non-textual modality representations to shrink to near-zero after low-rank projection, leading to lost feature discriminability, vanishing non-textual gradients, and text-dominated optimization that impairs cross-modal adaptation. To resolve this, we propose a geometry-aware framework optimizing adapter updates directly on the fixed-rank matrix manifold, with two key mechanisms: Riemannian Gradient Projection strictly preserves the rank of modality-specific adapters to avoid non-textual feature collapse, and Modality-Aware Momentum Balancing maintains independent Riemannian momentum buffers for modalities and adaptively rebalances their contributions to break text-dominated optimization. Extensive experiments on visual-text, audio-visual-text and speech-text benchmarks show our method consistently outperforms state-of-the-art multimodal PEFT and Riemannian optimization approaches.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.