Skip to content

Representation Editing for Multimodal Test-Time Adaptation

Sep 2026 · 0 citations
Computer Science

TL;DR

This work proposes FourIer Representation Editor (FIRE), a novel multimodal TTA approach that directly edits semantically rich intermediate representations, and demonstrates the superiority of FIRE over existing multimodal TTA methods.

Abstract

Multimodal test-time adaptation (TTA) aims to adapt a pretrained multimodal model online to distribution shift across modalities using unlabeled test data, showing broad potential in real-world applications. However, existing methods primarily focus on adjusting fused features to bridge the source-target gap, lacking explicit control over intermediate representation misalignment, which is a key driver of performance drop under distribution shift. In this work, we tackle this challenge from the perspective of representation engineering. Unlike previous TTA methods that update fusion weights in place, we propose FourIer Representation Editor (FIRE), a novel multimodal TTA approach that directly edits semantically rich intermediate representations. Specifically, we first adopt representation editors into each intermediate layer of the unimodal encoders, enabling layer-wise calibration of unimodal representations. To further enhance the diversity and stability of the low-rank editing subspaces, each representation editor performs frequency domain mixing via the fast Fourier transform to construct structured bases. Moreover, we introduce multi-level adaptation objectives to optimize these editors, jointly promoting cross-modal semantic alignment, source-target statistical alignment, and asymmetric prediction consistency. In this way, FIRE yields aligned unimodal representations for fusion and further improves prediction reliability. Extensive experiments on two widely used multimodal benchmarks under various corruption types demonstrate the superiority of FIRE over existing multimodal TTA methods.

View source

Similar papers

Oct 2026

Exploring Multimodal Adapters with Multi-Task Decoders for Video Action Recognition.

Large-scale vision-language pretrained models like CLIP, paired with parameter-efficient fine-tuning techniques, have emerged as promising solutions for image-to-video transfer in video action recognition. However, existing methods often prioritize strong supervised performance at the expense of transferability and gen...

Meng-Meng Wang, Ze-Yi Huang, Bo-Yuan Jiang et al. · 0 citations
Open access Aug 2026

TCFNet: an end-to-end framework for multimodal action quality assessment via temporal enhancement and contrastive fusion

TCFNet comprehensively integrates complementary information from multiple modalities, including RGB, optical flow, and audio, and incorporates specialized modules to capture long-range temporal dependencies and enhance rhythmic consistency.

Zhenxian Lin, Ming-Hui Zhang, Cheng-Mao Wu et al. · 0 citations
#artificial intelligence Preprint Sep 2026

PACE: Progressive Angular-to-Norm Contrastive Embedding

This paper proposes PACE, a two-stage framework that progressively expands both the representation and trainable parameter spaces, and introduces Focal Embedding Loss, a confidence-adaptive objective that downweights queries with high positive retrieval confidence while emphasizing ambiguous queries with competitive ne...

Yan-Ping Li, Wei Zhou, Ya-Wen Liu et al. · 1 citation
Preprint Aug 2026

Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Models

PuRF is introduced, a novel PuRiFication-driven cache-based method for multi-label test-time adaptation of vision-language models that consistently outperforms state-of-the-art methods on ViT-B/32 across five datasets.

Yiwen Liang, Hui Chen, Yizhe Xiong et al. · 0 citations
Preprint Aug 2026

Exploring the Design Space of Representation Learning for Audio Transformations

This framework produces both a transformation embedding and a processed-audio embedding, and it finds that the two play complementary roles: distance-based tasks favor the former, while probe-based tasks favor the latter.

Sungho Lee, Marco A. Martínez-Ramírez, Junghyun Koo et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.