Aug 2026· Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design· pp. 1-7· 0 citations· 24 references
TL;DR
TERINT-GEMV is designed, a multiplier-free bit-serial accelerator that ingests the 6-bit indices directly through an on-chip lookup table, computing ternary × INT8 dot products with dynamic zero-skipping.
Abstract
Emerging 1.58-bit large language models such as BitNet restrict weights to {-1, 0, +1}, dramatically reducing model size. Yet commodity hardware fails to fully exploit this structure: each ternary value carries only log2 (3) ≈ 1.58 bits of information but must occupy a 2-bit container, wasting 21% of memory bandwidth. Furthermore the lack of native mixed-precision support forces costly runtime dequantization. We present a hardware-software co-design that addresses both inefficiencies. On the software side, we compress ternary weights to 1.5 bits per weight via a 64-entry codebook with 6-bit indices, sacrificing only 5.4% of representational capacity; Vector Quantization aware fine-tuning retains 92.6% of baseline accuracy across six benchmarks. On the hardware side, we design TERINT-GEMV, a multiplier-free bit-serial accelerator that ingests the 6-bit indices directly through an on-chip lookup table, computing ternary × INT8 dot products with dynamic zero-skipping. Synthesized using a 28 nm commercial PDK, TERINT-GEMV achieves 99.44 TOPS/W, outperforming recently published BitNet accelerators by up to 7.2× in energy efficiency, and delivers 0.065 J/token with a 2.1× throughput gain over the leading CPU-based ternary runtime at one fifth the power.
On-device large language model (LLM) serving drives low-precision computing to address memory and compute limits. This paper presents L2Mersit, a scaling-free, range-adjustable exponent-encoded data format tailored for sub-8-bit LLM quantization. Building upon the Mersit framework, L2Mersit employs dual mode operation, comprising range-expanded and precision-enhanced modes that dynamically adapt to activation distributions with minimal control overhead. The proposed design eliminates on-the-fly scaling and auxiliary computations while effectively preserving range and precision, thereby achieving both superior perplexity and hardware efficiency. Experimental results demonstrate that L2Mersit achieves the highest accuracy among all 6-bit exponent-encoded formats while reducing the hardware complexity of auxiliary units for low-precision computing, resulting in a 62.7% area reduction.
M. Kim, Hyeonseong Kim, Ik-Joon Chang et al.· Proceedings of the ACM/IEEE...· 0 citations
Microscaling (MX) is now the standard for low-bit large language model (LLM) inference. Its 4-bit form MXFP4 still loses substantial accuracy, because existing MX formats fix either the element format or the precision-recovery scheme across blocks, and thus capture only limited quantization heterogeneity. Quantization heterogeneity appears at two levels: 1) across blocks, the preferred element format and precision-recovery scheme vary; 2) across operands, weights and activations require different encoding. We introduce AdaMX (Adaptive Microscaling), a heterogeneity-aware format and accelerator. It selects the precision-recovery scheme per block and the representation per operand, at no increase in equivalent bit width (EBW). One design covers two block sizes, giving a higher-accuracy operating point and a lower-EBW operating point that saves storage. We implement a 22nm FD-SOI AI accelerator prototype with the proposed decoder, computing unit, and quantization logic. Against an otherwise identical MXFP4 accelerator with FP4-only multipliers, AdaMX adds about 1% system energy. At the lower-EBW point, AdaMX stays more accurate than the baseline while lowering both memory footprint and energy. Across LLMs from 3B to 70B, AdaMX removes 83% of the MXFP4 accuracy loss on commonsense and 82% on MMLU, and 43% and 27% of the NVFP4 loss. AdaMX also generalizes to multimodal models. On Gemma-4 12B, it leads MXFP4 on all four vision-language benchmarks and keeps up to 96% of FP16 accuracy.
Junyi Luo, Xin Jiang, Tai-Hao Wen et al.· 0 citations
Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which pose significant challenges to low-bit quantization. In this work, we investigate sparse and low-bit activation strategies tailored for 1-bit LLMs. We first introduce BitNet v2 a4.8, a hybrid quantization-sparsification architecture designed to mitigate quantization errors arising from outlier activation channels. BitNet v2 a4.8 utilizes 4-bit quantization for inputs to attention and feed-forward layers, while sparsifying intermediate activations and subsequently quantizing them to 8 bits. To further leverage the 4-bit computation capabilities of next-generation GPUs, we present BitNet v2, a novel framework that enables fully 4-bit activations for 1-bit LLMs. To address the outlier activation channels, we propose $/mathcal{H}$, a lightweight module that applies an online Hadamard transformation prior to quantization, effectively regularizing activation distributions into more Gaussian-like forms conducive to low-bit representation. Extensive experiments demonstrate that BitNet v2 a4.8 achieves performance comparable to BitNet b1.58 with similar training costs. Furthermore, BitNet v2, when trained from scratch with 8-bit activations, matches the performance of BitNet b1.58. Remarkably, training BitNet v2 directly with native 4-bit activations leads to only marginal performance degradation, while significantly reducing computational overhead during batched inference.
Hongyu Wang, Shuming Ma, F. Wei· IEEE Transactions on Pattern...· 0 citations
Large language models (LLMs) are increasingly constrained by memory capacity, weight bandwidth, and checkpoint storage during deployment. Existing low-bit compression methods mainly follow two directions. Scalar or group-wise quantization is simple and compatible with efficient low-precision kernels, but its representation capacity becomes limited when the target budget approaches 2 bits per weight. Vector-quantized weight compression provides a richer block-level representation, but usually introduces explicit codebooks, index lookup, and additional storage accounting. This paper presents BiSCo-LLM, a codebook-free binary spherical coding framework for extreme low-bit LLM weight compression. The core pipeline is built on three components. First, local weight chunks are mapped onto a unit hypersphere and binarized into compact spherical codes, so that the main payload is a bit-packed sign stream rather than explicit VQ centroids. Second, a residual BSQ stage encodes the reconstruction error left by the base spherical codec, providing an explicit rate-distortion path without stored codebooks. Third, category-wise recovery distillation is performed after replacing each Transformer module category, reducing the mismatch between local weight reconstruction and assembled model behavior. A small 8-bit protected-channel path is used as an auxiliary stabilization mechanism for sensitive channels and is counted separately from the BSQ payload. The reported storage budget includes binary codes, neural decoders, protected-channel payloads, LoRA adapters, and metadata.
Yuantian Shao, Peisong Wang, Zhilei Liu et al.· 0 citations
On-device LLM inference is increasingly important for latency- and privacy-sensitive applications, yet it remains challenging due to the high compute and storage demands. Ternary-weight LLMs are a promising direction because they dramatically reduce model size and simplify arithmetic. In practice, deploying pretrained models on edge devices typically relies on post-training quantization (PTQ), but ternary PTQ often needs fine-grained scaling to preserve accuracy, which amplifies scale-metadata traffic and sub-byte decoding overhead that fits poorly with conventional NPU datapaths. This paper presents T-ACE, a Ternary Accuracy-aware Compute Engine that enables efficient ternary LLM inference under PTQ by jointly designing the data representation and execution pipeline. T-ACE co-packs 64 ternary weights and power-of-two scale metadata into a naturally aligned 16-byte block, eliminating separate scale fetches and preserving aligned memory access. To decode compact ternary packing efficiently, T-ACE proposes a compact two-stage 5-trit unpacker and integrates on-the-fly decoding and scaling directly into the ternary GEMM pipeline. The evaluation on an FPGA prototype shows that decoding and scaling are fully overlapped with GEMM execution, incurring no additional cycles over baseline. Moreover, the comparison against A100/H100 baselines in a normalized setting shows that T-ACE improves accuracy-adjusted compute density (ACD) by 66.8% and accuracy-adjusted energy efficiency (AEE) by 17.6% over the best GPU baseline.
Wonseok Jung, Junseok Kang, Sangwon Shin et al.· International Conference on...· 0 citations
Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, approaching the limit of $\log_2 3 \approx 1.585$ bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted"A"on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.