Fine-Tuning Small Language Models for Reliable VASP INCAR Generation
TL;DR
It is found that model size matters less than expected: once fine-tuning and post-processing are applied, performance saturates at a few billion parameters, and Qwen3-4B outperforms larger models in the same family.
Abstract
Language models can prepare VASP INCAR files from natural-language requests, but so far only large proprietary cloud models come close to handling the tightly coupled, physics-sensitive settings reliably, a dependence that fits poorly with local, high-throughput materials workflows where privacy, cost, and offline deployment matter. We show that a small language model (SLM) can close this gap. The SLM is fine-tuned on reference VASP calculations and paired with VASPGuard, a deterministic post-processor that checks syntax, workflow, and material-dependent constraints; we call the combined model INCAR-SLM. On INCARBench, a benchmark for VASP INCAR generation, INCAR-SLM built on Qwen3-4B outperforms every general-purpose LLM evaluated, exceeding GPT-5.4 by 15.55 points on the 100-point INCAR Score. Most of this gain comes from fine-tuning, with VASPGuard correcting the errors that remain. We further find that model size matters less than expected: once fine-tuning and post-processing are applied, performance saturates at a few billion parameters, and Qwen3-4B outperforms larger models in the same family.
