TTMC: Brain-Inspired Test-Time Memory Calibration with Orthogonal Projection for Online Continual Learning

Abstract

Existing Online Continual Learning (OCL) methods, particularly those based on Parameter-Efficient Fine-Tuning (PEFT), predominantly operate under a Static Inference Assumption, which means freezing model parameters immediately after training. This paradigm ignores the inevitable covariate shift in non-stationary test streams, resulting in suboptimal decision boundaries that fail to adapt to the immediate testing context. Furthermore, despite the computational efficiency of current analytic learning approaches, their reliance on Gaussian Random Projection introduces severe geometric distortion and feature aliasing in finite dimensions. Drawing inspiration from the Learning to Memorize at Test Time mechanism and the Pattern Separation process in the hippocampus, we propose Test-Time Memory Calibration (TTMC), a novel gradient-free analytic framework. Unlike traditional approaches that treat memory as a static statistic, TTMC views it as a dynamic associative pool. Specifically, we introduce a transductive calibration mechanism that seamlessly fuses the second-order statistics of the unlabelled test stream into the accumulated long-term memory via a closed-form solution, allowing for real-time alignment with the test distribution. To ensure the geometric stability of this dynamic test-time calibration, we theoretically prove the necessity of substituting random projections with Isometric Orthogonal Projections, which guarantee energy conservation and minimize feature interference. Our empirical evaluation demonstrates that TTMC consistently establishes a new state-of-the-art across diverse benchmarks. Notably, these performance gains are realized with zero backward passes and negligible computational overhead, underscoring the superior efficiency and structural robustness of our dynamic analytic framework. Code is available at https://github.com/hanyuyang99/TTMC.git.