In-Browser Split Inference for Autoregressive Large Language Models With Activation Quantization

Abstract

On-device inference of large language models (LLMs) preserves user privacy and eliminates per-query server costs. However, weak client devices cannot run full models locally, and complete offloading negates the benefits of on-device inference. This study introduces SplitLLM, a novel zero-installation system that partitions autoregressive LLM inference between a WebGPU browser client and remote server, maintaining a key–value cache on both sides of the partition. The influence of the client device, model, split point, and network on the throughput is systematically evaluated. The results show that the optimal split point depends on the client device, whereas the per-token round trip time is invariant to the split point. This explains why adapting the split point to the network conditions yields no communication benefit, providing support for a lightweight placement rule that requires no trial offloading. The transmitted hidden state is compressed to 8 bits using static, offline-calibrated per-channel scales shared between the client and server, reducing the payload by half while keeping perplexity within 0–7% of full precision. In contrast, per-tensor quantization inflates perplexity by several orders of magnitude at intermediate split points. Under bandwidth-constrained conditions, the proposed compression speeds up decode and prefill by 1.7– $2.6\times $ and 2.4– $2.9\times $ , respectively. Overall, this work offers a promising platform for enhancing the accessibility of LLMs, especially for resource-constrained client devices.