Quantization Trade-offs for LLM Inference on Apple Silicon: A Practical Speed, Memory, and Accuracy Benchmark

Abstract

Deploying large language models (LLMs) on consumer hardware typically requires quantization to reduce memory footprint and increase inference speed, but the practical trade-offs of this compression are not well characterized on Apple Silicon, where most existing benchmarks focus on NVIDIA GPU deployments. We present a systematic evaluation of three inference frameworks — MLX, llama.cpp, and their respective 4-bit and 8-bit quantization schemes — across three open-weight language models (Llama-3.2-3B, Qwen2.5-3B, and Phi-3.5-mini) on an Apple M4 MacBook Air. We find that MLX consistently outperforms llama.cpp in generation speed, that 8-bit quantization achieves perfect accuracy while 4-bit shows a measurable accuracy cost concentrated in code and arithmetic tasks, and that peak memory usage is driven primarily by each model's default context-window allocation rather than by quantization level.