AWa (Activation-Weight distribution-aware), a novel binarization framework that analyzes the distributions of activations and weights to perform binarization in a systematic, stage-wise manner, is proposed and achieves superior zero-shot average accuracy.
Abstract
Large Language Models (LLMs) demonstrate outstanding performance across diverse tasks but face significant deployment challenges due to their massive computational and memory requirements. Binarization, representing weights with 1-bit precision, offers a promising solution by dramatically reducing memory footprint and computational costs. However, existing methods often lead to severe performance degradation or require computationally expensive iterative optimization. We propose AWa (Activation-Weight distribution-aware), a novel binarization framework that analyzes the distributions of activations and weights to perform binarization in a systematic, stage-wise manner. First, we perform a joint saliency estimation that determines the saliency of weight by leveraging activation distributions and the Hessian sensitivity of weights. Subsequently, we apply both column and row scaling to the remaining non-salient weights to mitigate quantization errors. By adopting a distribution-based approach that eliminates repetitive operations, our method improves performance while incurring only a marginal increase in binarization time. Extensive experiments on various models and datasets demonstrate that AWa achieves superior zero-shot average accuracy. It surpasses the fastest existing method by a significant margin of 27.3 percentage points on LLaMA-3 70B, while maintaining negligible binarization overhead. Code will be released upon acceptance.
This paper introduces Gradient Enhancement Task Aware Post-training Quantization, i.e., GTAQ, to address the generalization issue of Large Language Models, and extensively evaluates the LLaMA family of language models on WikiText, C4, and MMLU.
Yihua Shao, Yangyang Gu, Minxi Yan et al.· 0 citations
Machine unlearning aims to remove the influence of specific training data while preserving model utility. Many state-of-the-art approaches pursue this goal by restricting the forgetting update to a subset of parameters selected through gradient-based saliency. Although such methods are widely adopted, the actual contribution of saliency-based weight selection to representation-level forgetting remains unclear. In this work, we perform the first controlled ablation of the saliency masking mechanism used by SalUn. Using a matched-compute experimental design on CIFAR-10 and CIFAR-100 with ResNet-18, we compare saliency-based masking against random masks of equal sparsity and unconstrained updates, while keeping the unlearning objective, optimization schedule, and computational budget fixed. Across multiple representation-level evaluations, including linear probing, prototype recovery, and layer-wise CKA, the three configurations exhibit statistically equivalent representation-level recoverability. We find that forget gradients are strongly concentrated in the final network layers (approximately 92% of the squared gradient energy on CIFAR-10) before any mask is applied, causing all masking strategies to operate within the same representational subspace. Furthermore, saliency masks show limited class specificity (specificity index 0.09-0.11), selecting highly overlapping parameter subsets across different forget classes. Our findings suggest that, in the studied setting, representation-level forgetting is primarily governed by gradient concentration and representation geometry rather than by the specific identity of saliency-selected weights. More broadly, the results support a growing body of evidence indicating that effective representation-level unlearning requires objectives that act directly on latent representations rather than on increasingly sophisticated weight-selection strategies.
Billel Habbati, A. Merlo, Luca Verderame et al.· 0 citations
Dataset distillation aims to synthesize compact datasets that can approximate the performance of full-data training while significantly reducing computational and storage costs. However, diffusion-based distillation methods often struggle to preserve structural coherence and generalization, especially in visually complex domains. This issue often stems from latent prototypes that are weakly aligned with class-discriminative regions and contaminated by irrelevant background, thereby degrading generation quality and generalization. To address this limitation, we propose a saliency-driven distillation framework that constructs class-discriminative latent prototypes to enhance representativeness and generalization. The framework proceeds in two stages: (1) ensemble Grad-CAM++ saliency is used to construct prototypes emphasizing class-discriminative regions, and (2) hard-prototype refinement is then applied to construct challenging yet class-consistent prototypes, thereby enhancing discriminability and diversity. Importantly, the diffusion backbones (e.g., LDM and DiT) remain frozen; only lightweight classifiers used for saliency extraction are trained. Extensive experiments across multiple benchmarks demonstrate consistent performance improvements over strong baselines. Code will be released.
Yawen Zou, Wenqi Cai, Guang Li et al.· 0 citations
Window-based attention mechanisms have been introduced to alleviate the excessive computational cost inherent in global attention mechanisms. In this paper, we propose a novel architecture named FwNet-ECA, which integrates the Fourier transform with learnable weight matrices to enhance spectral features of images. By performing filter enhancement after window-based attention, our method establishes a global receptive field, thereby overcoming the limited receptive field typically associated with windowed attention. Furthermore, we incorporate the existing Efficient Channel Attention module to improve interchannel information exchange. Unlike approaches that rely on physical window shifting, our method leverages frequency-domain enhancement to implicitly connect spatial regions. We evaluate our model on the iCartoonFace dataset and demonstrate competitive performance on fine-grained classification benchmarks. Experimental results show that, compared to shift-based window methods, our model achieves comparable accuracy with fewer parameters and lower computational overhead. Moreover, visualization analyses clearly indicate that the filter enhancement technique is particularly effective in the shallow layers of the network, where feature maps are relatively large. This work presents an effective solution to the limited receptive field problem in window-based attention mechanisms. The code is publicly available at
https://github.com/qingxiaoli/FwNet-ECA
.
Xiaoqing Li, Ya Wang, Nannan Gu et al.· The European Journal on Arti...· 0 citations
With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especially in resource-constrained environments. Traditional pruning methods typically depend on full gradient-based importance estimation, and they necessitate prior finetuning of the model to achieve satisfactory performance. This process often results in intolerable resource consumption. This paper proposes REP-LIE, a new approach to enable resource-efficient pruning during the process of finetuning. REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation. To address the inherent randomness in importance estimation, a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters. The pruned model is further finetuned through lightweight updates, eliminating the need for full-parameter optimization in the process of finetuning. Extensive experiments on both medium-scale encoder models and large-scale generative models (LLaMA-7B and Mistral-7B) demonstrate that REP-LIE still achieves competitive performance compared to existing approaches.
Peng Liu, Hui-Bing Zeng, Yi-Qun Zhang et al.· 0 citations
An attention-free and lightweight token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse tokens to produce a compact visual representation, and achieves a favorable accuracy-efficiency trade-off.
Xuanyi Hao, Zuoyuan Zhang, Zhibo Wang et al.· 0 citations
