Milli: Continuous Spatiotemporal Manifold Engine for Token-Efficient Visual Intelligence
Abstract
Frontier Multimodal Large Language Models (MLLMs) process dynamic visual feeds by discretizing time into frame arrays and space into rigid 14x14 pixel patches. To prevent quadratic self-attention O(N^2) from exhausting context windows, production pipelines downsample incoming video to 1–2 frames per second (FPS), discarding up to 96.7% of physical motion and introducing a 500–1000 ms temporal blind gap. In this paper, we present Milli, a continuous coordinate perception architecture that models visual dynamics by decomposing visual scenes into a time-invariant static base manifold (S), a regularized patch-based optical transport velocity field (v), and a sparse spatiotemporal residual error field (delta). Rather than ingesting redundant pixel matrices, downstream models receive continuous motion splines and sparse event trigger packets dispatched only when residual energy surges. We evaluate Milli on a real, locked, single-video validation sequence: a 73.50-second 720x1280 (portrait, 30.0 FPS, 2,205 frames) high-dynamic physics simulation (Video by theballsimulator [DecsCniIhKj].mp4, SHA-256: 8fd2189bb4c7e9838756248bc99e9dba1e9826e585ac8403028a6efc51d0d08c). On this sequence, standard 14x14 patch tokenization generates 10,233,405 visual tokens, compressed 30 FPS Perceiver architectures consume 568,890 tokens, and decimated 1-FPS production APIs ingest 19,092 tokens. Milli reduces visual context consumption to 1,627 tokens in Pure Tracking Mode (a 91.48% reduction vs. 1-FPS; 99.98% vs. raw ViT) and 5,517 tokens in Full Resynchronization Mode across 5 major structural rupture events (a 71.10% reduction vs. 1-FPS; 99.95% vs. raw ViT). Scored against a pre-annotated visual ground truth list of 11 discrete physical collision and barrier-shattering events in the initial benchmark segment, Milli’s residual shock detector achieves 54.55% recall (6/11) and 50.00% precision (6/12), while standard 1-FPS temporal decimation exhibits total blind failure (0.0% recall, 0/11) because all collision events fall entirely within inter-frame blind intervals. Given the small sample size (n = 11), this 54.55% point estimate carries an exact 95% binomial confidence interval of 23.38% to 83.25% (Clopper-Pearson; Wilson score: 28.01%–78.73%). Analytical edge boundary localization via directional second derivatives achieves an empirical median precision of 0.086 px and an RMS precision of 0.203 px on high-resolution interface benchmarks (1200x1500). Reference CPU implementation of the regularized flow solver executes in 68.9 ms per frame.
