Skip to content
Open access

Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters

Nov 2026 · IEEE Transactions on Pattern Analysis and Machine Intelligence · 0 citations · 84 references
Computer Science

TL;DR

D2MoE is proposed, a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.

Abstract

Multi-task dense prediction seeks a single, coherent representation that supports simultaneously discriminative semantic recognition and regularized geometric/structural inference. For vision foundation models, pretrained semantic invariances conflict with the frequency and smoothness biases of dense objectives; joint fine-tuning and single-route adapters therefore compress competing gradients into one update channel, causing negative transfer and unstable cross-task agreement. We frame this as two-level adaptation: task-generic alignment that preserves priors, and task-aware specialization that resolves late-stage conflicts. Based on this principle, we propose D2MoE , a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone. D2MoE introduces Task-Generic Low-Rank Adaptation (TG-LoRA) in attention projections throughout the network to provide a stable, shared adaptation channel, and a Task-Adaptive Mixture of Experts (TA-MoE) injected only at the final attention block of each stage to enact sparse, task-conditioned perturbations of token mixing in a single forward pass. A Semantic Prior Injector further couples multi-level task features with frozen foundation priors to reinforce semantic coherence while preserving boundary and structural fidelity. Experiments across diverse benchmarks substantiate that this structured factorization yields robust multi-task improvements with a small trainable footprint, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.

Read PDF

Similar papers

Book Open access Nov 2026

Multi-Task Dense Prediction with Mixture of Unit-Rank Experts

Multi-task dense prediction has recently achieved significant performance gains but remains challenging in learning fine-grained and discriminative task-specific representations. In this paper, we propose a novel decoder-focused Mixture of Unit-Rank Experts (MURE) for multi-task dense prediction. Inspired by the subspa...

Hao Xu, Zhen-Chun Wei, Zheng Wang et al. · 0 citations
Book Open access Nov 2026

Unmixing the Latent: Factor-Aware Sparse Adaptation for Vision-Language Models

Large pre-trained Vision-Language Models (VLMs) excel in zero-shot generalization but face challenges in efficient few-shot adaptation. Existing parameter-efficient methods apply uniform updates across all tokens, overlooking the inherent sparsity of cross-modal interactions and exacerbating overfitting. We propose SFA...

Ze-Xiang Li, Wen-Xu Wang, Jia-Wen He et al. · 0 citations
Sep 2026

DeCo: Efficient Decouple-to-Couple Learning for Multi-Task Visual Grounding

Multi-task visual grounding requires models to jointly understand linguistic semantics and perform accurate visual localization and segmentation. Despite the success of multimodal large language models, effectively adapting them to multiple grounding objectives remains challenging. Existing methods commonly enforce tas...

Xiaoqiang Lu, Li-Cheng Jiao, Long Sun et al. · 1 citation

Architecture Alignment With Sparse Priors in Tabular Foundation Models

Together, these results support architecture-prior alignment: preserving an addressable feature axis provides an inductive bias for task-adaptive relevance inference for task-adaptive relevance inference.

Tian-Qi Zhao, Tian-Yi Zhuang, Shuo Duan et al. · 0 citations
#machine learning Preprint Oct 2026

PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging

Large pretrained models have achieved remarkable success across diverse domains, yet adapting them to multiple downstream tasks remains challenging: full fine-tuning is costly, while maintaining separate task-specific models is storage-inefficient and fails to produce a unified multi-task model. Low-Rank Adaptation (Lo...

Mu-Qing Liu, Chong-Jie Si, Zhuo-Ya Liu et al. · 0 citations
Book Open access Nov 2026

Unleashing the Power of CLIP as a General Teacher for Multi-Task Learning

Multi-task learning (MTL) improves efficiency and performance by jointly learning multiple related tasks through shared representations. However, inter-task interference can impair the generalization of these representations, leading to negative knowledge transfer. Knowledge distillation (KD) has emerged as a promising...

Ya-Yu Zhang, Jin-Long Zuo, Yu-Hua Qian et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.