Skip to content
Book Open access

SL²OT: Scaling LLM Locality on Tile with Distributed 3D-DRAM as Unified Memory

2027 · Proceedings of the 32nd ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 · 0 citations · 26 references

Abstract

Large language models (LLMs) are increasingly limited by memory bandwidth and latency rather than compute throughput. Our modeled 3D-DRAM system offers 4× the raw bandwidth of the HBM reference. However, high remote-access latency (~3× local latency) and redundant NoC traffic reduce its effective bandwidth below that of HBM. We present SL2OT, a hardware--software memory system that harnesses the raw bandwidth of distributed 3D-DRAM to improve LLM performance, guided by four principles---Scalability, Locality, Orchestration, and Temporality. Unified Virtual Memory (UVM) localizes remote accesses through asynchronous prefetching while preserving a unified address space. Asynchronous Request Coalescing (ARC) merges asynchronously arriving requests to the same address across tiles, reducing redundant transfers. Kernel Memory Migration (KMM) prefetches data for upcoming kernels, overlapping host-to-device transfers with computation. Accel-Sim evaluations using A100-80GB SASS traces show 1.22--3.03× speedup over HBM across operators and transformer blocks, and 3.51× speedup over naive distributed 3D-DRAM on GEMM. A matched FlashAttention study reduces mean global-read latency by 73.4% relative to Mesh. SL^2 OT reaches 72.2% effective bandwidth utilization, versus 15.1% without SL2OT.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.