This work proposes Hyper-Spherical Quantization (HSQ), which decouples semantic content from feature magnitude via angular routing, preventing code assignment from being dominated by scale rather than meaning.
Abstract
In this work, we aim to discretize the high-dimensional visual representations to bridge the gap with language models - a non-trivial challenge, as existing quantization methods suffer from codebook collapse, failing to scale while preserving semantic coherence. We identify the root cause as metric mismatch: standard Euclidean codebook objectives are fundamentally misaligned with the anisotropic geometry of representation space, leading to codebook embeddings with high-variance magnitude scales and uneven angular distributions that hinder scalability. To address this, we propose Hyper-Spherical Quantization (HSQ), which decouples semantic content from feature magnitude via angular routing, preventing code assignment from being dominated by scale rather than meaning. The resulting discrete Representation Autoencoder (dRAE) achieves high-fidelity reconstruction while preserving semantic integrity and supporting scalable codebook budget. Extensive experiments demonstrate consistent performance gains as the vocabulary size scales to 131{,}072, along with 100\% codebook utilization, simplified training pipeline, and strong performance across understanding and generation tasks.
Hadamard-Transform-domain Quantization (HaTQ), which uses orthogonal Hadamard reparameterization before quantization to redistribute weight and activation responses in the original domain across channels, and is compatible with integer-only execution.
Junqi Shi, Chongzhi Wang, Yiwen He et al.· 0 citations
Vector-Quantized Variational Autoencoders (VQVAEs) have enabled strong performance in generative modeling by mapping continuous data to learnable codes. In this work, we identify a surprising yet consistent phenomenon that we term dimensional collapse : despite using high-dimensional embeddings, VQVAEs tend to compress their representations into a much smaller subspace, typically only 4 to 10 dimensions. We provide an in-depth analysis of this phenomenon and reveal its relation to model performance and learning dynamics. Interestingly, VQVAEs naturally gravitate toward this low-dimensional regime, and enforcing higher-dimensional usage (e.g., via rank regularization) could lead to degraded performance. To overcome this low-dimensionality limitation, we propose Divide-and-Conquer VQ (DCVQ) , which partitions the latent space into multiple low-dimensional subspaces, each quantized independently. By design, each subspace respects the model’s preference for low dimensionality, while their combination expands the overall capacity. Our results show that DCVQ overcomes the inherent dimensional bottleneck and achieves improved reconstruction quality across image datasets.
Jiayou Zhang, Yifan Shen, Guan-Hong Chen et al.· Neural Information Processin...· 3 citations
KronQ, a PTQ framework that challenges the assumption that all output channels contribute equally to the layer-wise reconstruction objective by introducing the gradient covariance into the quantization pipeline, and introduces bidirectional incoherence processing.
Donghyun Lee, Yuhang Li, Ruokai Yin et al.· 0 citations
This work proposes Normalizing Autoencoder (NAE), which employs a novel conditional loss that aligns the surrogate loss gradient with that of reconstruction loss, directly improving upon the current standard.
Scalable product quantization has recently attracted considerable attention in large-scale image retrieval, as it avoids the need to train multiple models for generating quantization codes of varying lengths. However, most existing approaches primarily concentrate on approximating the ground-truth similarity between image pairs, while overlooking the correlations among sub-codebooks and among codewords. In addition, limited work has addressed the challenge that increasing the number of subspaces or codewords substantially raises memory consumption. To address these limitations, we propose a novel scalable product quantization framework within an end-to-end network, termed Tensor-based Codebook Product Quantization (TCPQ). This work represents an innovative attempt to integrate tensor theory with product quantization methods. The framework leverages tensor-based methods to capture spatial correlations among sub-codebooks and among codewords, and adopts lightweight codebooks for efficiency. For optimization, a subspace-wise unbiased supervised contrastive loss is proposed to bring embeddings of the same class closer together, push embeddings of different classes farther apart within quantization subspaces, and precisely regulate the minimum distance between positive and negative samples. In addition, the ArcFace loss is incorporated to enhance the discriminative power of the learned features, and an orthogonality constraint is imposed on the factor matrix along the codebook dimension to avoid overfitting. Extensive experiments on three large-scale real-world benchmarks demonstrate that TCPQ achieves state-of-the-art retrieval performance.
Bin Luo, Laurence T. Yang, Debin Liu et al.· IEEE Transactions on Image P...· 0 citations
Most competitive 4-bit LLM research pipelines now open the same way: apply a linear, function-preserving transform (rotation, scaling, permutation, non-orthogonal affine) so the outlier mass sits more favorably against the group scales, and only then round. Yet we are aware of no survey dedicated to this transform stage, and its literature is quietly re-deriving an older theory. We identify and formalize the principle that organizes it, the Great Inversion: allocation-flexible coding rewards energy concentration, whereas the grouped shared-scale quantization a deployed matrix instruction performs rewards within-group flattening. Classical transform coding (1963: decorrelate, allocate bits, quantize) spends different bits per coordinate at a fixed total rate; for a Gaussian source at high rate the Karhunen-Loeve transform's concentration minimizes distortion. A deployed operand tile instead carries one absolute-maximum scale per group and equal bits everywhere, with no allocation; on a uniform grid that objective rewards flattening, approached by Hadamard incoherence. We prove that opposition under within-group majorization: the prescriptions point in opposite directions, each backed by a proof against its own objective, and for a generic spectrum no optimality guarantee transfers. A second axis is the number format: the non-uniform FP4 grid makes flattening buy less, MXFP4's power-of-two block scale still rewards a rotation confined to that block, and NVFP4's mantissa-carrying scale largely removes that pull, so the target pole depends jointly on allocation regime and format. We survey 200 works to a June 2026 cutoff; classify 43 transform methods by structure, data-awareness, searched-versus-constructed, and runtime cost; record, where reported, how they compose with GPTQ rounding; distill a first-choice guide by deployment regime; and close with the open problems it exposes.