Skip to content

VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

Sep 2026 · arXiv.org · Vol abs/2609.33325 · 0 citations · 63 references
Computer Science

TL;DR

This work introduces VisionHOPE, the first generic visual backbone formulated as a self-modifying learning system, in which what the model remembers and how it learns co-evolve within an image, and derives a stability-matched step-size control scheme that combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition.

Abstract

Visual backbones have evolved from Convolutional Neural Networks (CNNs) with local aggregation to Vision Transformers (ViTs) with global interactions, State-Space Models (SSMs) with input-dependent state transitions, and Test-Time Training (TTT) layers that adapt an inner learner while processing an image. Across this progression, visual computation has become increasingly adaptive to each input, yet the rules governing that adaptation remain largely prescribed by the trained backbone. We introduce VisionHOPE, the first generic visual backbone formulated as a self-modifying learning system, in which what the model remembers and how it learns co-evolve within an image. Building on the self-referential construction of Nested Learning (NL), VisionHOPE realizes this co-evolution through five coupled memories that store content, generate key and value representations, and govern learning rate and retention. These memories evolve jointly as visual context accumulates along each scan. However, directly applying the unconstrained self-referential update to a visual backbone leads to instability. We therefore derive a stability-matched step-size control scheme that combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition, and prove that the resulting memory dynamics are non-expansive along each scan. For two-dimensional feature maps, we adapt NL's chunk formulation by aligning chunks with image rows and columns across four directional scans. The proposed VisionHOPE achieves competitive results on ImageNet-1K, COCO, and ADE20K, establishing self-modifying learning systems as a practical foundation for general-purpose visual backbones. The code is available at https://github.com/PSRben/VisionHOPE.

View source

Similar papers

Review Open access Nov 2026

Modern computer vision from CNNs to foundation models: a unified framework for representation learning

Modern computer vision is increasingly driven by the search for representations that generalize across tasks, modalities, and learning settings, rather than excelling at a single benchmark. This has produced a broad collection of architectures and paradigms, including convolutional neural networks (CNNs), Vision Tran...

A. Moujahid, F. Dornaika, Salah-Eddine Mechkouri et al. · 0 citations
#artificial intelligence Preprint Oct 2026

VICO: Visual Environments Co-Evolving for Vision-Language Model Reasoning

Reinforcement learning with verifiable rewards (RLVR) has become a standard recipe for post-training vision-language models (VLMs), but it typically assumes a static training environment. As the actor improves, fixed tasks drift out of its learning frontier: many become trivial, others remain unsolvable; and the learni...

Meng Lu, Li-Geng Zhu, Olivia C. Xiao et al. · 2 citations
Preprint Oct 2026

AdaLoop: Thinking with the Vision Encoder

AdaLoop is presented, a framework for question-conditioned visual recurrence in VLMs that adapts visual computation to the question and reuses the encoder's transformations across multiple iterations, with the pretrained vision weights frozen.

Xinran Nicole Han, Yi-Yang Lu, Katelyn Gan et al. · 0 citations
Sep 2026

LiAuto-MindViT: A Hybrid Vision Backbone with Adaptive Bidirectional Mamba

While Mamba-based models have shown strong potential for long sequence modeling, adapting them to vision is challenging due to the requirement of local neighborhood correlations and multi-directional spatial contexts for visual understanding. In this paper, we present LiAuto-MindViT, a novel hybrid vision backbone that...

Li Mu, Shuai Chen, Wen Zheng et al. · 0 citations
Preprint Oct 2026

ALPS: Visual State Encoders for World Models

World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typ...

Yan-Qing Liu, Le-Tian Zhang, Cheng-Zhi Liu et al. · 0 citations
Preprint Nov 2026

Unbundling Multimodal Vision Encoding

How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretrai...

Ellis Brown, Chris Hoang, Shengyi Qian et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.