Skip to content

Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex

Sep 2026 · arXiv.org · Vol abs/2609.36366 · 0 citations · 71 references
Biology Computer Science

TL;DR

This work uses per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, and finds that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas associated with dynamic motion perception.

Abstract

Understanding how the brain parses actions and events from time-varying natural inputs is a central challenge in neuroscience. Recent work has used deep neural network (DNN) models to build stimulus-computable fMRI encoding models that predict single-voxel responses to complex natural videos. However, the majority of video-computable encoding models predict responses using simple linear mappings from model tokens, overlooking the spatiotemporal structure shared by video representations and neural responses. Recent cross-attention encoding models address this limitation for static images, enabling flexible stimulus-dependent weighting of image content across space. Here, we extend this framework to naturalistic video, using per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, fitting this model to fMRI responses to short video clips. We compare joint spatiotemporal attention with factorized and selectively constrained alternatives, and find that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas associated with dynamic motion perception. Moreover, our method provides interpretable, stimulus-specific attention maps that dynamically follow moving objects, revealing which locations and temporal moments contribute to each neural response. We further show that attention maps from parcels in different category-selective networks (face-, body-, scene-selective) differentially weight content in accordance with expected semantic selectivity. Together, this work provides a new computational framework for understanding how visual information is adaptively weighted by cortical populations during dynamic visual perception.

View source

Similar papers

Open access Sep 2026

Preferred visual experiences provide intuitive descriptions of the functional properties of the cortical navigation network

This work used a variational autoencoder to generate video clips predicted to correspond to maximal activity in each brain region, and identified a network of 11 cortical regions that support active navigation and are tuned for complex feature combinations that capture dynamic scenes that incorporate both world events...

Tian-Jiao Zhang, Cheol Jun Cho, Jack L. Gallant · 0 citations
Open access Sep 2026

Img2EEG: A Scalable and Interpretable Encoding Framework for Simulating Human EEG Responses to Visual Inputs

Understanding how visual information processing unfolds over time requires models that not only predict neural responses but also expose the representations that support them and generalize beyond sampled stimulus spaces. Here we introduce Img2EEG, a participant-specific image-to-EEG encoding framework that integrates...

Zi-Tong Lu, Julie D. Golomb · 0 citations
Open access Aug 2026

Visual-Semantic Tuning across the Cortex Shifts between Tasks

Attention is a powerful mechanism that dynamically optimizes brain representations to prioritize behaviorally relevant information. While previous studies focusing on single tasks have demonstrated that attention shifts tuning toward attended targets, real-world behavior often requires humans to switch between tasks wi...

Tian-Jiao Zhang, J. Gallant · 0 citations
Open access Sep 2026

Task-optimized neural networks reveal distinct contributions of specialized and broader visual learning to neural representations of face familiarity

How neural activity across the ventral visual hierarchy supports face recognition is an open question. A long-standing debate asks whether face processing, particularly in fusiform cortex, relies on face-specific computations or representations shared with broader visual recognition. Here we combine source-resolved mag...

Hamza Abdelhedi, Shahab Bakhtiari, Karim Jerbi · 0 citations
Open access Sep 2026

Selective Visuo-Spatial Attention Revealed by Fine-grained Deep Visual Representations during Naturalistic Movie Watching.

Selective visuo-spatial attention is a complex and dynamic process that prioritizes relevant visual information while suppressing irrelevant distractors. Recent studies have demonstrated that incorporating computational visual attention models with functional magnetic resonance imaging (fMRI) in naturalistic paradigms...

Liting Wang, Xin Zhang, Peng-Cheng Xie et al. · 0 citations
Open access Oct 2026

Decomposing deep neural network-brain representational similarity reveals distinct sources across the human visual hierarchy

Deep neural networks predict neural responses across the visual hierarchy, yet published alignment scores do not reveal whether this correspondence reflects learned representations, architectural inductive biases, low-level image statistics, or categorical structure. We decompose DNN-brain alignment into four sources...

Xiao-Nai Li, Fang-Yao Zhang, Yuxuan Zhang · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.