Skip to content

PFArena: Benchmarking Language Models for Protein Modification

Sep 2026 · 0 citations · 40 references
Computer Science Biology

TL;DR

The evaluation reveals that model performance shifts systematically with the availability of target-specific experimental evidence: PLMs demonstrate proficiency in open-ended single-mutant generation by leveraging protein-specific priors, whereas LLMs and agents perform strongly in multi-mutant ranking, particularly when target-specific fitness data is available.

Abstract

Protein modification requires navigating an immense sequence space, yet wet-lab validation remains low-throughput and costly. Although computational paradigms including protein language models (PLMs), large language models (LLMs), and LLM-based agents have shown promise in protein modification, their relative efficacy across realistic experimental decision-making settings remains unclear. To bridge this gap, we introduce PFArena, a benchmark comprising four controlled task interfaces that cover single-mutant generation and multi-mutant ranking. By providing varying levels of mutation fitness data, PFArena reflects four representative research scenarios characterized by differing degrees of prior experimental context. We assess six PLMs, six LLMs, and five LLM-based agents using complementary metrics to measure both peak and overall protein modification performance. Our evaluation reveals that model performance shifts systematically with the availability of target-specific experimental evidence: PLMs demonstrate proficiency in open-ended single-mutant generation by leveraging protein-specific priors, whereas LLMs and agents perform strongly in multi-mutant ranking, particularly when target-specific fitness data are available. Nevertheless, all model families face fundamental challenges with increasing search-space size and mutation depth. We release our code and benchmark suite to facilitate reproducible research in model-assisted protein modification.

View source

Similar papers

Open access Sep 2026

NeoToxPred: a fine-tuned protein language model with orthologous and length-stratified negative controls for robust toxicity classification

Although protein toxins represent valuable pharmacological templates, predicting toxicity directly from primary sequences is inherently challenging because of their evolutionary dynamics. Active toxins and their benign homologues frequently share identical structural scaffolds and differ by only a few key residue subst...

Seongmin Kim, Min-Seok Kim, Chungoo Park · 0 citations
Open access Sep 2026

Comprehensive evaluation of AlphaFold/OpenFold prediction of experimentally unresolved proteins through novel metrics

A comparative framework leveraging Bland–Altman agreement analysis is introduced, to evaluate per-residue Cα-confidence differences and Per-Residue profiles (PRPs), complemented by Uniform Manifold Approximation and Projection (UMAP).

Florencia R. Díaz, Daniela Orschanski, Juan I. Folco et al. · 0 citations
Open access Sep 2026

Protein Language Models: Learning From Evolution, Designing Beyond It

Protein language models (PLMs) have transformed our ability to learn from evolutionary sequence space, but protein engineering ultimately asks a different question: not what evolution selected, but what we should build next. Zero-shot likelihoods therefore provide useful, but not universal, measures of fitness and can...

Maurice Brenner, Julius Schlensok, A. Plaikner et al. · 0 citations
Open access Oct 2026

Generalizing Protein Fitness Prediction Across Distribution Shifts via Structural and Co-evolutionary Priors

Protein fitness prediction is essential for protein engineering. Recent fitness-landscape benchmarks such as FLIP2 have shown that methods performing well under conventional train–test splits can fail when the test set is shifted out of distribution (OOD), including mutations at unseen positions and variants from unsee...

Nopsinth Vithayapalert, Maytus Piriyajitakonkij, Francesca Grisoni · 0 citations
#protein folding Open access Sep 2026

Simplifying in silico protein evolution with minimal screening by unZipro.

UnZipro is presented, an efficient, scalable, and generalizable framework for zero-shot, in silico protein evolution, that integrates a compact, pre-trained inverse folding model with meta-learning to derive family-specific fitness landscapes.

Zhao-Hui Qin, Sanzeng Zhao, Zhao-Long Deng et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.