Skip to content
Log in
Preprint

Reproducible Evaluation of MoE Expert Caching: Replay Semantics, Workload Contamination, and Operating Regimes

Yu Zhang
Aug 2026 · 0 citations · 24 references
Computer Science

TL;DR

With a trace-driven, event-atomic simulator over three MoE models, a large offline-optimal gap substantially overstates the gains recovered by representative lightweight causal mechanisms.

Abstract

Mixture-of-Experts (MoE) models have outgrown accelerator memory, and offloading expert weights to host memory is now standard. This makes expert cache management an attractive lever: a policy that raised the hit rate would cut expert traffic per token. Evaluating that is a measurement problem, and we find the measurement fragile. With a trace-driven, event-atomic simulator over three MoE models (40, 64, 128 experts), we isolate three evaluation axes that change conclusions, not just numbers. Replay semantics: under a fused-event traffic contract, an inconsistent per-access replay inflates recency-based policies by 27-29% while leaving frequency-based and static ones within 4%, inverting the policy ranking. Workload contamination: probe sets using one instruction template per category produce verbatim-identical generation prefixes; a matched-pair rendering intervention moves the measured early-window effect by 19.4-31.9 points and reverses which workloads look most cache-friendly. Operating regimes: normalized miss fractions do not transfer across models, so the per-step expert union relative to per-layer capacity must be reported -- yet permuting only the temporal order of an identical event stream moves the offline-optimal gap from 44.9% to 30.8%, so it is not sufficient. Corrected, a stable gap to the offline optimum remains (44.2-45.9% over 13 frozen workload compositions). A forced-admission oracle attributes 84.3-96.6% of it to knowing which resident expert is used furthest in the future. A causal next-use predictor, used as an eviction rule, recovers -11.4% of the gap; it picks an optimal victim 3.4% of the time, against 2.4% for a random resident block and 20.6-22.1% for LRU and LFRU. Our position is narrow: in our evaluated settings a large offline-optimal gap substantially overstates the gains recovered by representative lightweight causal mechanisms.

arXiv Semantic Scholar
View source

Similar papers

#small language model Book Open access Aug 2026

Balancing and Beyond: Communication-Centric Optimizations in Expert Parallelism

EPIC mitigates imbalance via performance-aware expert migration and runtime expert activation, and then improves communication with topology-adaptive transport kernels and fine-grained computation-communication overlap.

Jiamin Cao, Qingxu Li, Yaozhong Liu et al. · 0 citations
Preprint Aug 2026

Why Do Prefetchers Fail? Let Agents Answer

To the authors' knowledge, this is the first empirical demonstration that an agent-driven hardware-design process can produce an RTL-practical prefetcher that outperforms state-of-the-art human designs on unseen workloads.

Xiangfeng Sun, Ceyu Xu, Ningzhi Ai et al. · 0 citations
#artificial intelligence Preprint Aug 2026

TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Serving

Energy-aware LLM serving requires comparing configurations under realistic request shapes, yet exhaustive target-GPU profiling is costly and a cheap predictor can be dangerously confident outside its measured scope. We present TokenPowerSandbox, an evidence-gated workflow that combines an interpretable CPU-resident projector, short target-GPU probes, full-workload verification, and tamper-evident freeze-before-measurement provenance. On one NVIDIA H100 80GB serving Qwen2.5-7B-Instruct with vLLM, three anchor repeats and six development workloads calibrate workload transfer. The same frozen model is evaluated on a blind holdout and a separately predeclared no-refit confirmation totaling 51 post-freeze runs. Energy MAPE is 6.23% and 7.35%, with Spearman rank correlations of 0.976 and 0.933. However, a predeclared TTFT gate passes at concurrency four (9.27% MAPE) and triggers abstention below four (64.80%), showing why energy accuracy cannot certify latency.

Chenxu Niu · 0 citations
Preprint Aug 2026

Aneto: Predicting System Performance by Exploiting Cross-Workload Regularity

Predicting how a workload responds to a change in memory technology requires estimating how much of each cache miss actually stalls the processor. Obtaining this stall fraction accurately has traditionally demanded detailed simulation, repeated measurements, or heavy profiling. One-shot alternatives exist but sacrifice accuracy. We observe that hardware counters from a single native run suffice to infer the stall fraction without simulation. Across more than 100 diverse workloads spanning integer, floating-point, graph, and AI benchmarks, the relationship between CPI and the maximum memory stall per instruction follows a predictable pattern on each microarchitecture. Aneto is a mechanistic-empirical regression model that exploits this observation. Once fitted on a machine across a small set of reference workloads, the model estimates the performance-latency sensitivity of any new workload from a single run, enabling first-order CPI prediction under any memory configuration. Across six machines and two simulators, Aneto reaches 2x lower CPI error than the best prior one-shot predictor. We validate the predictions directly against hardware measurements on an ARM server, from local DDR to HBM and up to ~3x the baseline memory penalty, where the median CPI error is 12.7% and the 90th percentile 35.9%. At an 8x memory-latency extrapolation beyond the reach of direct measurement, Aneto agrees with a reference model on Zen 5 to within 14.6% at the median and 41% at the 90th percentile. Additionally, Aneto provides qualitative insights into workloads and architectures.

Raúl Taranco, Rene Mueller, Michael Giardino · 0 citations