SL²OT: Scaling LLM Locality on Tile with Distributed 3D-DRAM as Unified Memory
Abstract
Large language models (LLMs) are increasingly limited by memory bandwidth and latency rather than compute throughput. Our modeled 3D-DRAM system offers 4× the raw bandwidth of the HBM reference. However, high remote-access latency (~3× local latency) and redundant NoC traffic reduce its effective bandwidth below that of HBM. We present SL2OT, a hardware--software memory system that harnesses the raw bandwidth of distributed 3D-DRAM to improve LLM performance, guided by four principles---Scalability, Locality, Orchestration, and Temporality. Unified Virtual Memory (UVM) localizes remote accesses through asynchronous prefetching while preserving a unified address space. Asynchronous Request Coalescing (ARC) merges asynchronously arriving requests to the same address across tiles, reducing redundant transfers. Kernel Memory Migration (KMM) prefetches data for upcoming kernels, overlapping host-to-device transfers with computation. Accel-Sim evaluations using A100-80GB SASS traces show 1.22--3.03× speedup over HBM across operators and transformer blocks, and 3.51× speedup over naive distributed 3D-DRAM on GEMM. A matched FlashAttention study reduces mean global-read latency by 73.4% relative to Mesh. SL^2 OT reaches 72.2% effective bandwidth utilization, versus 15.1% without SL2OT.