Unleashing the Power of CLIP as a General Teacher for Multi-Task Learning
Abstract
Multi-task learning (MTL) improves efficiency and performance by jointly learning multiple related tasks through shared representations. However, inter-task interference can impair the generalization of these representations, leading to negative knowledge transfer. Knowledge distillation (KD) has emerged as a promising approach to address this issue. Existing MTL-KD methods, however, typically rely on task-specific teacher models trained on downstream datasets, incurring high computational costs and exhibiting limited generalization. To overcome these limitations, we propose a CLIP-based multi-task distillation framework, CMKD, which leverages the pre-trained CLIP model as a general teacher to enhance MTL model generalization without additional task-specific teacher training. CMKD achieves this through two complementary distillation paths. The shared representation path employs three parallel projection operators to align feature dimensions and semantic representations between CLIP and the shared MTL encoder, facilitating more effective knowledge transfer. The task-specific semantic path aligns CLIP's image-text matching logits with the student's task outputs to transfer cross-modal prior knowledge. Experimental results demonstrate that CMKD consistently improves MTL performance across diverse datasets.