Incremental Vision–Language Object Detection via Sparse Frequency Transform

Abstract

Vision-language object detectors (VLODs) pretrained on large image-text corpora exhibit strong zero-shot detection abilities, yet their performance often degrades on specialized downstream tasks that arrive sequentially. Hence, we study Incremental Vision-Language Object Detection (IVLOD), which requires continually adapting a VLOD across tasks with domain and/or class shifts while mitigating catastrophic forgetting of prior tasks and preserving its zero-shot generalization. To address this problem, we propose a novel method, named Sparse Frequency Transform (SFT), that minimizes inter-task interference via a frequency-domain design. Building on a theoretical link between forgetting and Frobenius inner product (FIP), SFT enforces sparsity-independence on spectral supports, yielding zero inter-task FIP by construction and achieving reduced forgetting with low impact on the pretrained model. Spectral supports are selected via an online gradient-magnitude scoring rule with masking of previously used supports, then converted via inverse discrete cosine transform (IDCT) to dense time-domain updates that preserve FIP constraints and improve optimization efficiency. Extensive experiments under both full-shot and few-shot IVLOD settings demonstrate that our SFT is capable of consistently learning new tasks while preserving the zero-shot generalization capabilities of the pretrained model.