Skip to content

LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling

Sep 2026 · arXiv.org · Vol abs/2609.37675 · 0 citations · 67 references
Computer Science Biology

TL;DR

This work introduces ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments that demonstrates that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning.

Abstract

Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of tokenization remains underexploited. Unlike human language, proteins preserve structure despite extensive sequence variation, a property standard tokenization strategies fundamentally fail to capture. We introduce ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments. ZEST allows embedding domain-level biological priors directly at the tokenization stage rather than learning them implicitly through scale. ZEST natively compresses sequences to an average token length of 3.3 residues, enabling our model to process $\geq3380$ residues within a standard 1024-token context window. Building on this, we present LEMON (Layered Extraction of Molecular Ordering from Nature), a compact 200M-parameter sequence-based model for detection of remote homology between protein sequences trained on a single H100 GPU for one week. Despite its modest size, LEMON outperforms models of 600M to 3B parameters and is on-par with TM-Vec. Our results demonstrate that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning. All code and model weights are publicly available (at \href{https://github.com/LDWLab/ZEST}{github.com/LDWLab/ZEST}) under the MIT license.

View source

Similar papers

Oct 2026

Fast and accurate construction of multiple sequence alignments from protein language embeddings with ARIES.

Multiple sequence alignment (MSA) is a foundational task in computational biology, underpinning protein structure prediction, evolutionary analysis, and domain annotation. Traditional MSA algorithms rely on pairwise amino acid substitution matrices derived from conserved protein families. While effective for aligning c...

Minh Hoang, Isabel Armour-Garb, Mona Singh · 0 citations
#artificial intelligence Preprint Sep 2026

ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing

ProtLingo, an efficient PLM framework that augments a pretrained single-sequence backbone with conditional local memory and sparse expert routing, achieves competitive performance with a 150M-scale backbone, including strong parameter efficiency on mutation-effect prediction and preserved long-range structural represen...

Ming-Rui Li, Si-Xian Shen, Min-Zhang Li et al. · 0 citations
#protein folding Open access Sep 2026

AtlasFold: Protein structure prediction with metagenomic-scale language models

The Atlas model family is presented, an open and trainable system spanning protein language modeling, monomer folding, and protein-complex prediction, and the training code and data, stage checkpoints, and model weights under the MIT License are released to provide a foundation for advancing PLM-based protein structure...

Seonghwan Seo, Hyeongwoo Kim, Seokhyun Moon et al. · 0 citations
#machine learning Preprint Sep 2026

Analysis of Quantized and Efficiently Adapted Protein Language Models

QLoRA and 4-bit quantization reduce PLM computational requirements, particularly GPU memory usage, and support QLoRA as a first-pass strategy for memory limited adaptation, reserving full fine-tuning for challenging tasks, unstable architectures or low validation recovery.

I. Zeisler, Sebastian Clancy, P. Bayat et al. · 0 citations
#machine learning Preprint Oct 2026

RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

The genome holds the blueprint that governs the biological properties of the cell. Consequently, advancing our knowledge of genomic function is crucial both for a broader understanding of biology and for continued biomedical advances. The success of large language models on natural language and protein sequences has mo...

F. I. Marin, Panagiotis Antoniadis, Dionysia Danai Brilli et al. · 0 citations
Open access Sep 2026

Predicting genome-wide functional constraints with GPN-Star

GPN-Star substantially outperforms previous methods in prioritizing pathogenic and fine-mapped genome-wide association study variants, yields strong enrichments of complex trait heritability and improves power in rare variant association testing, which positions GPN-Star as a scalable, powerful and flexible tool for ge...

Cheng-Zhong Ye, Gonzalo Benegas, Carlos Albors et al. · 10 citations · ⚡1

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.