Rethinking large language model evaluation in traditional Chinese medicine: toward quality-grounded standards and fine-grained assessment

Abstract

Abstract Large language models (LLMs) are increasingly applied to clinical decision support for traditional Chinese medicine (TCM). However, current evaluation practices remain inadequate. Uncurated knowledge sources, such as un-selected ancient texts and web-crawled data, may contain outdated or prohibited recommendations. Specific clinical cases, even a single case, used as the benchmark may mislead the evaluation by individual experience instead of evidence-based consensus. Coarse-grained scoring strategies, such as true/false selections or holistic Likert ratings, obscure component-level errors and underestimate clinical complexity. TCM clinical reasoning is multidimensional, encompassing multiple coexisting syndromes and detailed procedural parameters. We propose a path forward built on two pillars: (1) structuring authoritative knowledge, including national clinical guidelines, industry standards, standard textbooks, the Chinese Pharmacopoeia, and consensus derived from large-scale high-quality cases; and (2) fine-grained, component-wise evaluation, in which complex TCM reasoning is decomposed into minimal assessable units across diagnosis and treatment, scored with component-level precision/recall and error grading, and judged primarily by the identification of critical errors that could cause medical harm, supplemented by expert-assessed global acceptability. Such quality-grounded, component-wise frameworks would enhance scientific credibility, support regulatory scrutiny and clinical adoption, and enable targeted model improvement aligned with the WHO Traditional Medicine Strategy 2025-2034.