Conference Open access

Open-Set Test-Time Adaptation in VLMs via Noise-Immune Self-Purification

Ming-Xu Feng Feng-Qiang Wan Yang Yang

Sep 2026 · Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence · pp. 4286-4294 · 0 citations · 51 references

TL;DR

Noise-Immune Self-Purification (NISP) constructs a noise-immune fine-grained adaptation where the adapter enforces consensus-discrepancy constraints to refine coarse pseudo-labels and suppress noise propagation, and devise the Jaccard Consistency Score for OOD discrimination.

Abstract

Unlike test-time adaptation (TTA), open-set TTA (OSTTA) aims to robustly adapt to in-distribution (ID) domain shifts while suppressing the detrimental impact of out-of-distribution (OOD) samples encountered at test time. To this end, we propose a novel OSTTA approach named Noise-Immune Self-Purification (NISP). By exploiting the zero-shot priors of pre-trained vision-language models (VLMs), NISP advances from coarse pseudo-labeling to fine-grained, noise-resilient adaptation. Technically, we first introduce a dual-cluster Bayesian Gaussian mixture model to fit VLMs-derived scores, achieving coarse pseudo-ID/OOD separation via posterior-risk thresholding. Subsequently, NISP constructs a noise-immune fine-grained adaptation where the adapter enforces consensus-discrepancy constraints to refine coarse pseudo-labels and suppress noise propagation. We then devise the Jaccard Consistency Score for OOD discrimination. Overall the coarse-to-fine pipeline enables rigorous self-purification and robust online adaptation. Theoretically we show that consensus-discrepancy losses mitigate the deleterious effects of noise. Empirically, NISP achieves state-of-the-art results across multiple OSTTA benchmarks, validating its efficacy. The code is available at https://github.com/njustkmg/IJCAI26-NISP.

DOI Semantic Scholar

Read PDF

Similar papers

Preprint Aug 2026

Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Models

PuRF is introduced, a novel PuRiFication-driven cache-based method for multi-label test-time adaptation of vision-language models that consistently outperforms state-of-the-art methods on ViT-B/32 across five datasets.

Yiwen Liang, Hui Chen, Yizhe Xiong et al. · 0 citations

#artificial intelligence Preprint Sep 2026

Noisy Test-Time Reinforcement Learning for Code LLMs

Large language models (LLMs) have demonstrated remarkable performance across various code-related tasks. However, unlike carefully curated datasets that are typically high-quality and error-free, real-world user instructions are often vague and error-prone, posing significant challenges to the robustness of code LLMs....

Xi-Kai Yang, Hieu Trung Nguyen, Dun-Yuan Xu et al. · 0 citations

Preprint Aug 2026

Reliable Neural Collapse Approximation for Open-World Test-Time Adaptation

A new method named Reliable Neural Collapse approximation (ReNC) for Open-World Test-Time Adaptation (OWTTA) is introduced, leveraging neural collapse as a structural prior for reliable target-domain adaptation.

Jia-Qi Lin, Yuangang Pan, Chang-Ran Wang et al. · 0 citations

#artificial intelligence Preprint Aug 2026

From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model

This work proposes a principled VLM TTA method called \algname, and theoretically reveals that the InfoNCE loss can be neatly reformulated as a Wasserstein OT formulation, thereby unifying the objectives of the inference and adaptation of VLMs to achieve their mutual benefits.

Qi Yu, Zhichen Zeng, Katherine Tieu et al. · 0 citations

#machine learning Preprint Sep 2026

Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation

FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates, shows that a carefully selected fixed span can recover most of the benefit of mov...

Wen-Song Ye, Zhan-Ming Shen, Zhiqing Xiao et al. · 1 citation · ⚡1

Conference Open access Sep 2026

On-Device Realistic Test-Time Adaptation via Bias-Resistant Statistical Alignment

Test-Time Adaptation (TTA) aims to adapt pretrained models to unseen test data, which is crucial for resource-constrained edge devices that must handle distribution shifts on the fly without human supervision. However, conventional TTA methods often fail in realistic scenarios characterized by continuous shifts and sev...

Hao-Jie Bai, Aiguo Chen, Rui-Ting Dai et al. · Proceedings of the Thirty-Fi... · 0 citations