PINCH: Predictive Importance-Sampling-Informed Cache for I/O-Bound DNN Training

Abstract

DNN training on large datasets is often bottlenecked by data I/O rather than GPU computation. Shuffled SGD destroys temporal locality, rendering conventional caching and prefetching ineffective. Two classes of system address this: clairvoyant caches that exploit the deterministic access trace of seeded shuffling to plan storage proactively, and importance-aware caches that direct resources toward high-value samples. No system combines both because they impose contradictory requirements: clairvoyant planning needs a fixed training set for trace predictability, while importance sampling needs a dynamic subset that adapts to evolving sample scores. We propose PINCH, a cache middleware that decouples the two at the granularity of a planning horizon. At each horizon boundary PINCH re-selects the training subset using importance and stability signals; within the horizon the subset is fixed and the trace is deterministic, enabling planned cache retention and bounded prefetch. Evaluation shows that PINCH achieves up to 12.2 × training speedup over a no-cache baseline, outperforming both clairvoyant-only and importance-only baselines at every operating point.