It is shown that enhancing an LLMs capabilities for robust prediction of small molecule geometries still retains nearly all of its pre-trained language abilities by randomly mixing in small quantities of natural language prompt-response pairs into the fine-tuning.
Abstract
The power of Large Language Models (LLMs) has led us to investigate how they might be fine-tuned for learning the"language of molecular geometry". The fine-tuning of the LLMs using Cartesian coordinates or Z-matrices provides an extremely simple method for accurately predicting equilibrium structures and diverse sets of conformers of small organic and drug-like molecules with excellent accuracy and outperforming specialized deep learning models. While the most common representation of molecular geometries are Cartesian coordinates performs adequately, we find that the inherent invariances and relational nature of geometries represented as Z-matrices provides a better grammar for LLM adaptation. Finally, we show that enhancing an LLMs capabilities for robust prediction of small molecule geometries still retains nearly all of its pre-trained language abilities by randomly mixing in small quantities of natural language prompt-response pairs into the fine-tuning.
Pretrained molecular language models are increasingly used as molecular encoders for learning structure-property relationships. However, their practical suitability for molecular discovery within and beyond their pretraining domain remains unclear. Herein, we systematically benchmark four molecular language models across six virtual molecular libraries spanning drug discovery, organic materials, and catalysis. Native molecular language model embeddings show substantial variation in discovery performance across libraries, whereas molecular fingerprints provide a consistently strong and robust baseline. Consistent with a potential domain-representation mismatch, we show that explicit domain adaptation substantially improves representation performance. Fine-tuning molecular language model encoders on structures from the target virtual library consistently improves sample efficiency, with several adapted encoders emerging as the top-performing representations across the benchmark tasks. These results show that molecular representation quality depends strongly on the target domain and that explicit adaptation can improve the practical utility of molecular foundation models. More broadly, our findings establish domain-adapted molecular representations as a promising strategy for sample-efficient adaptive decision making in virtual screening and self-driving laboratories.
Henrik Wille, Luis-Finley Schütz, Felix Strieth-Kalthoff· 0 citations
A high-resolution layer-by-layer interpretability analysis of 8 models from the ESM2 and AMPLIFY families on 22 concepts from human proteome annotations found that these models encode concepts of increasing levels of complexity along their depth: basic physicochemical properties and linear motifs are best captured by early-layer embeddings, secondary structure from subsequent layers, and domain-level semantics from middle layers.
Shawn T. Whitfield, Tom Marty, Robert M. Vernon et al.· bioRxiv· 0 citations
A clear pattern is revealed in LLM spatial capabilities: while they still lag behind state-of-the-art approaches, they are promising and can handle multiple spatial constraints simultaneously, enabling scaling to heterogeneous setups.
Thomas MacDougall, Maksim Kuznetsov, Roman Schutski et al.· 1 citation
MolLingua, a token-centric, dual-modal framework designed for native molecular understanding, uses a dual-branch Residual Vector Quantization engine to discretize heterogeneous, high-dimensional spatial 2D and 3D features into compact code sequences rather than relying solely on continuous projections.
Haoyang Liu, Xikang Feng, Fei Guo et al.· IEEE journal of biomedical a...· 0 citations
A modular Context-Augmented Prompting framework that enables agentic tool use at inference time: a trained GNN expert model provides a predictive hint with confidence, and a GNN extracts an instance-specific explanatory subgraph via a necessity-based edge-drop intervention.
K. Bougiatiotis, Dimitrios Kelesis, Georgios Paliouras· 1 citation
Accurately identifying protein binding sites for small molecules and ions is crucial for understanding biological processes and advancing drug discovery. Pretrained protein language models (pLMs) have emerged as powerful tools for this purpose, but existing prediction models often face a trade-off when using pLMs: freezing pLMs limits their adaptability, while fully fine-tuning them requires high computational costs. To address this trade-off, we attempted to introduce Parameter-Efficient Fine-Tuning (PEFT) as a promising solution to balance efficiency and performance. Specifically, we applied five PEFT strategies (i.e., LoRA, QLoRA, DoRA, AdaLoRA, and IA3) to fine-tune four pLMs (i.e., ProtT5, ProtBERT, ESM2–150M, and ESM2–650M) and assessed their predictive performance across protein binding site tasks for 11 representative small molecules and ions. The results clearly indicate that the LoRA-enhanced ESM2–650M consistently outperforms all other combinations. Despite this robust baseline, training independent models for specific small molecules remains challenging due to the scarcity of high-quality binding data. To bridge this gap, we implemented a Grouped Multi-Task Learning (GMTL) strategy, allowing the model to capture shared binding patterns among ligands with similar biological significance. Experimental results demonstrate that this strategy significantly enhances predictive performance. Building upon these insights, we present Symphony-Bind. It is a GMTL framework that leverages LoRA-enhanced ESM2–650M to extract embeddings, which are subsequently refined by a shared ConvBERT module and then processed by ligand-specific MLPs for precise binding site prediction. Performance evaluation on 11 representative ligand tasks shows that Symphony-Bind achieves average MCC values of 0.561, 0.629, and 0.324 for the nucleotide, cofactor, and inorganic ion groups, surpassing evaluated sequence-based state-of-the-art methods while remaining competitive with structure-based models.
