Skip to content
Log in
Sponsored Artificial Intelligence: A Modern Approach, Global Edition Artificial Intelligence: A Modern Approach, Global Edition Amazon

Replay-free Sequential Fine-tuning of Medical VLMs

Hefei Li Yuhui Zhang Xiaohan Wang S. Yeung-Levy Josh Achiam Steven Adler Lama Sandhini Agarwal Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt S. Altman Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr Yana Hasson Arthur Lenc Katherine Mensch Malcolm Millican Huanran Chen Yinpeng Dong Zeming Wei Yao Yichi Huang Han Zhang Su Jun Zhu Adrito Das D. Khan Dimitrios Psychogyios Haotian Liu Chunyuan Li Qingyang Wu
· 0 citations · 15 references

TL;DR

The streamlined, replay-free approach proves highly effective on a new continual learning benchmark and a practical path toward building prehensive, continually-learning medical VLMs and advancing the development of medical AI.

Semantic Scholar
View source

Similar papers

Open access Jul 2026

Biological Continued Pretraining Reshapes the Capability Profile of a Foundation Model Without Catastrophic Forgetting

It is widely assumed that continued pretraining (CPT) on a narrow, out-of-distribution corpus such as raw biological sequence must trade away a general-purpose model’s broad competence — the “alignment tax” or catastrophic-forgetting intuition. We test this directly, without any new training, by re-analyzing three checkpoints from a single lineage of a 26B-parameter Mixture-of-Experts model (Gemma-4-26B-A4B): the instruction-tuned base, the same model after biological CPT (8.7B tokens of DNA, protein, and biomedical text), and after subsequent supervised fine-tuning (SFT). Across three independent capability axes — general knowledge/reasoning (MMLU, ARC, HellaSwag), code generation (MBPP), and biomedical knowledge (BixBench) — we find that biological CPT does not degrade the model; it lifts it: MMLU +13 points, MBPP pass@1 nearly doubles (0.33 →0.63), and BixBench discrimination rises sharply (MCC 0.23 → 0.92). The single measured regression is truthfulness (TruthfulQA 8.8 points), a small and interpretable domain drift. A clean vocabulary-expansion ablation (< 0.4 pt on every general metric) confirms the gains are attributable to CPT, not tokenizer changes. Crucially, subsequent SFT narrows the model back: all three axes fall to near-base levels, revealing a consistent division of labor — CPT re-organizes and lifts the shared capability substrate; SFT cashes it out onto target tasks. We argue this reframes biological sequence not as a competitor for a foundation model’s capacity but as a form of structured scientific data that reshapes its capability profile, and that CPT and SFT should be budgeted as complementary rather than substitutable stages. All checkpoints, evaluation code, and per-example outputs are public. Highlights A training-free re-analysis of one 26B MoE lineage isolates the effect of biological continued pretraining (CPT) from tokenizer changes and from fine-tuning. Biological CPT does not cause catastrophic forgetting; it raises general knowledge (MMLU +13 pts) and code generation (MBPP pass@1 0.33 → 0.63). CPT also makes chain-of-thought reasoning 41% shorter and near-backtrack-free while pre-serving accuracy — an effect invisible to accuracy metrics. A consistent CPT-lifts / SFT-narrows division of labor recurs across four axes, reframing biological sequence as structured scientific data that reshapes a model’s capability profile. The Bigger Picture Adapting a general-purpose AI model to a specialized domain — here, the language of DNA and proteins — is usually assumed to come at a cost: teach it biology and it forgets how to reason about everything else. This “no free lunch” intuition shapes how practitioners budget compute and whether they attempt domain adaptation at all. We test the assumption directly, and without running any new training, by comparing three snapshots of the same model taken before and after biological training. The result overturns the intuition: feeding the model raw biological sequence made it better at general knowledge, at writing code, and even changed how it reasons — producing shorter, more decisive chains of thought without losing accuracy. The gains appear during the sequence-pretraining stage and are partly given back during task-specific fine-tuning, revealing that the two stages play complementary rather than interchangeable roles. This suggests a broader principle for data-centric AI: structured scientific data — biological sequence today, and by extension code, mathematics, and chemistry — is not merely knowledge to be absorbed but a lever that reshapes what a foundation model can do.

Liang Wang · 0 citations
Preprint Jul 2026

An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answering

Deploying medical visual question answering (MedVQA) systems in real-world clinical settings requires models that adapt to new clinical tasks without forgetting previously acquired knowledge. Continual learning (CL) provides a practical framework for this setting. Despite rapid progress in medical vision-language models, the behavior of CL methods when training these models across heterogeneous MedVQA tasks remains underexplored. This work presents a systematic evaluation of CL for MedVQA across diverse clinical objectives, including classification, multi-label classification, detection, cell counting, and report generation. Specifically, we explore (1) the ability of existing CL methods to mitigate catastrophic forgetting; (2) their sensitivity to task ordering, analyzing how different task sequences influence performance retention and forgetting; and (3) the evolution of low-rank adaptation parameters as new tasks are learned, revealing patterns of weight drift under different CL methods. Our findings suggest that existing CL methods struggle to maintain stability-plasticity balance when tasks with different objectives and supervision formats are interleaved. Code and full experimental setup will be publicly available.

Mai A. Shaaban, Tausifa Jan Saleem, Alaa Mohamed et al. · 0 citations
Preprint Jul 2026

The Art of Not Forgetting A Local Learning Architecture for Continual Learning

The results suggest that the combination of sparse representations, local learning, and persistent memory is a promising direction for continual learning, while motivating further investigation into the respective roles of learning rules, representations, and architectural design in mitigating catastrophic forgetting.

Ashmith Atmuri, Yashaswini Rao Bhogarajula · 0 citations
Preprint Jul 2026

TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models

Medical Vision-Language Models (VLMs) exhibit strong zero-shot performance, yet their effectiveness still declines on out-of-distribution (OOD) data due to domain shifts and class bias inherited from large-scale pretraining. Existing few-shot adaptation methods typically introduce additional trainable components, which can be unstable in extremely low-data regimes (e.g., 1-shot), and lack robustness on different medical data. We present TCLA, a purely training-free few-shot adaptation method for Medical VLMs, which is fast and model-agnostic. TCLA corrects inference logits based on a small set of support samples, boosting pretrained VLMs performance by improving inter-class deconfusion and reducing domain shift. Extensive experiments on nine datasets across multiple medical imaging modalities including X-ray, Ultrasound, MRI, CT, Histopathology, demonstrate that TCLA consistently improves OOD performance of Medical VLMs and, in most of cases, outperforms existing training-based adaptation methods.

Tianyou Jiang, Ziyu Zhou · 0 citations
Preprint Jul 2026

The Art of Not Forgetting

We introduce CMP (Cognitive Memory Primitive), an architecture that represents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns entirely through local, gradient-free updates, with no backpropagation anywhere in the network. We use this architecture to test a specific hypothesis: that catastrophic forgetting, usually treated as a training-time defect to be patched with replay or regularization, is instead a structural consequence of how backpropagation assigns credit and that a learning rule that is local and sparse by construction should resist it without a patch. On a controlled domain-incremental protocol across 15 text domains, three-seed replicated, CMP's backward transfer is 15-19x better than a matched-size Transformer trained with online EWC, and the result survives a domain-order control (reported as a range, +0.24 to +0.44, rather than a single figure). We report this alongside a real, substantial accuracy gap versus the Transformer baseline, a null result on a recognized vision benchmark, and a diagnosed, unresolved failure attempting to combine this architecture with a separate mechanism that improves raw accuracy, disclosed because an honest negative result is more useful than an omitted one. The central claim is narrow and falsifiable: local, sparse, non-backpropagation learning measurably resists catastrophic forgetting better than backpropagation with its standard fix, under conditions we state precisely.

Ashmith Atmuri, Akshay Kumar, Yashaswini Rao Bhogarajula · 0 citations