Jul 2026· Annual International Computer Software and Applications Conference· pp. 1453-1462· 0 citations· 27 references
Abstract
Traditional model reduction techniques for neural network compression usually leverage existing matrix/tensor decomposition methods to explore the redundancy among model parameters, which may overlook the local structures of the models. In this work, we propose a novel low-rank approximation algorithm, called Dimension-Augmented Tensor-Train (DATT) decomposition, which integrates dimension slicing with the Tensor-Train decomposition. DATT allows different partitions to be represented using distinct, non-shared spanning sets. This enables the decomposition to adapt more flexibly to localized structure, potentially achieving better compression. With the localized low-rank structures, DATT provides a more compact representation of neural network weights. We evaluated the performance of DATT with the other state-of-the-art low-rank approximation-based model compression algorithms, and demonstrated that DATT outperforms them in terms of parameter reduction and model accuracy. In particular, for compressing ResNet-20 on CIFAR-10, DATT achieves a 77.00% reduction in parameters with only a 0.37% drop in top-1 accuracy.
This work presents a method to accelerate the optimization of learning high dimensional functions using deep neural network (DNN), and studies the effect of adding features which distill pretrained DNN into TNs using a discretize and decompose strategy.
High-dimensional token embeddings underpin generative language models, as they can capture subtle semantic information and significantly enhance the modelling of complex language patterns. However, this high dimensionality also introduces considerable model parameters and prohibitively high model storage and memory requirements, which are particularly unaffordable for low-end devices. Targeting no extra training data and insufficient computation cases, we propose a training-free model compression approach based on the tensor-train decomposition (TTD), whereby each pre-trained token embedding is converted into a lower-dimensional matrix product state (MPS). We investigate what language capabilities are preserved under training-free compression at different compression ratios, providing insights into the distinct redundancy structures captured by tensor-based versus pruning-based compression methods. We then comprehensively investigate the low-rank structures extracted by this approach, in terms of the compression ratio, the language task performance, and latency on a typical low-end device (i.e., Raspberry Pi). Our approach trades increased inference latency (no more than 0.5 ms/token reconstruction overhead on Raspberry Pi) for substantial memory and storage reduction, making it suitable for deployment scenarios where memory and storage are the primary bottleneck. Taking GPT family, OPT models, Qwen (2.5–0.5 B and 3–0.6 B) and MiniCPM4-0.5B as case studies, our approach for the embedding layer compression consistently achieves a compression factor 0.5× – 2.0×. The extension of our approach for dense layers (feed-forward layers and attention layers) compression, can improve the model language task performance in zero-shot reasoning tasks. Our performance analysis of different tasks reveals that tensor decomposition preserves higher-level logical reasoning capabilities (e.g., BoolQ, ARC-Challenge), while pruning-based methods like SliceGPT maintain advantages for tasks requiring broad lower-level lexical feature coverage (e.g., HellaSwag, WinoGrande), demonstrating that different compression approaches preserve complementary linguistic capabilities.
Mingxue Xu, Y. Xu, Danilo P. Mandic· Machine Intelligence Researc...· 0 citations
Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.
This paper proposes a novel Tensor Train (TT)-based tensor-on-tensor regression optimization framework for variable selection based on mode-1 hyperslice sparsity. The formulation incorporates an l2,0-regularized term on the first TT-core while imposing Stiefel manifold constraints on the remaining M−1 TT-cores. Leveraging the property that the group sparsity of the first core is equivalent to the hyperslice sparsity of the global structure, we establish theoretical guarantees for the uniform variable-selection consistency of the proposed model. To efficiently solve the proposed model, we design an alternating iterative algorithm equipped with a preconditioned metric and prove its convergence to a critical point. Extensive numerical experiments on both synthetic and real-world datasets demonstrate that the numerical solutions generated by our algorithm exhibit exact support recovery in practice, tightly aligning with our theoretical analysis.
This work presents the tree tensor network Nyström (TTNN), an algorithm that extends recent research on streamable tensor approximation to the more general tree tensor network format, enabling a unified treatment of various existing methods.
Alberto Bucci, Gianfranco Verzella· Numerical Linear Algebra wit...· 3 citations
This work introduces a novel and efficient solver for the underlying HJB equation based on the functional tensor train (FTT) format, which leverages latent low-rank structures to efficiently approximate high-dimensional functions, enabling both model compression and rapid computation.
Robert Gruhlke, Julius Berner, David Sommer et al.· 0 citations
