Preprint

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

Minseok Kang Hyunwoo J. Kim Chanyoung Kim Minwoo Kim Jaekoo Lee Dahuin Jung

Aug 2026 · 0 citations · 47 references

Computer Science

TL;DR

A retraining-free VLM pruning framework called PORTA is introduced that derives a task- and modality-agnostic importance formulation based on activation variation, estimated from generic calibration data, which reliably captures feature-level representation utility across modalities.

Abstract

Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments. Existing pruning strategies often depend on task-specific criteria or LLM-oriented importance measures, making them unsuitable for task-agnostic pruning, where no task-specific samples are available at pruning time and the pruned model remains broadly applicable. We introduce a retraining-free VLM pruning framework called PORTA that derives a task- and modality-agnostic importance formulation based on activation variation, estimated from generic calibration data, which reliably captures feature-level representation utility across modalities. PORTA further incorporates an adaptive sparsity allocation mechanism that assigns layer-wise pruning ratios based on output feature variability, avoiding the limitations of uniform sparsity and reducing performance degradation at high compression levels. Extensive experiments across VLM architectures, such as CLIP, BLIP, and Qwen2-VL, demonstrate that PORTA achieves competitive downstream performance under high sparsity without requiring any retraining, supporting efficient VLM compression. Code is available at https://github.com/cau-hai-lab/PORTA.git.

arXiv Semantic Scholar

View source

Similar papers

Preprint Jul 2026

SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

It is observed that attention scores from both vision and text tokens peak at modality separator tokens, suggesting that these separators bridge the two modalities and proposes SepPrune, an efficient, training-free, plug-and-play pruning method that uses the separator token as a unified query to rank and select informative vision tokens.

Yucheng Wang, Qihui Zhu, Yang Liu et al. · 0 citations

Aug 2026

DR-EFT: Exploring and reloading domain-representative experts for the memory-constrained fine-tuning of MoE large models.

An algorithm framework named DR-EFT (Domain-Representative Experts for Fine-Tuning), which explores and loads the domain-representative experts for subsequent retraining and reincorporation and demonstrates robustness through validations on popular MoE LLMs, including Qwen, DeepSeek, and Ernie.

Zhaomeng Cheng, Zhong Ji, Yan Zhang et al. · Neural Networks · 0 citations

Preprint Jul 2026

Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models

An attention-free and lightweight token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse tokens to produce a compact visual representation, and achieves a favorable accuracy-efficiency trade-off.

Xuanyi Hao, Zuoyuan Zhang, Zhibo Wang et al. · 0 citations

Preprint Jul 2026

SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs

SAMPLe (Sharpness-Aware Minimization Prompt Learning), a plug-in sharpness-aware optimizer that enhances prompt generalizability by accounting for loss landscape sharpness, is introduced, establishing itself as a robust, model-agnostic solution for prompt learning.

Hossein Rajoli, Fatemeh Lotfi, Niloufar Alipour Talemi et al. · 0 citations

Conference Open access 2026

LaCo: Layer-wise Compensation for Pruned Large Language Models

Pruning is essential for the efficient deployment of Large Language Models (LLMs); however, it causes severe performance degradation due to the structural distortion induced by sparsity. Existing recovery strategies, such as LoRA, predominantly employ global fine-tuning, often overlooking the mechanistic root of this degradation: the layer-wise accumulation and amplification of local errors. To address this limitation, we propose LaCo ( La yer-wise Co mpensation), a framework that reori-ents the recovery paradigm from global adaptation to hierarchical representation alignment. By sequentially optimizing each layer to re-construct the model’s hidden states, LaCo effectively intercepts the error propagation chain at its source. Extensive experiments demonstrate that LaCo surpasses parameter-efficient baselines in both perplexity reduction and zero-shot reasoning. Notably, it reduces recovery-time memory usage to approximately 1 / 7 of the baseline and requires only 2,048 unlabeled samples to match a LoRA model trained on 50k examples—achieving a ∼ 25 × improvement in data efficiency.

Yingen Liu, Fan Wu, Xuyan Pan et al. · Annual Meeting of the Associ... · 0 citations

Preprint Aug 2026

SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

This work introduces an adaptive visual token selection strategy for VLMs that leverages average text-to-visual attention scores to assess the importance of visual tokens, removing redundant ones during pruning based on a set threshold, thereby optimizing the importance calculation.

Yaozhi Wen, Jialong Guo, Zhenliang Ni et al. · 0 citations