Evaluating Large Language Models for Query Refinement in Systematic Review Retrieval of Traditional Chinese Medicine

Abstract

Efficient and comprehensive literature retrieval is a critical prerequisite for high-quality systematic reviews and meta-analyses in traditional Chinese medicine (TCM). Conventional manual query construction is time-consuming and highly dependent on researcher expertise, particularly when searching across multilingual databases. With the rapid development of large language models (LLMs), new opportunities have emerged for optimizing literature retrieval processes. However, the effectiveness, robustness, and methodological limitations of LLM-assisted query optimization in TCM systematic reviews remain insufficiently evaluated. Four LLMs, namely GPT-4o, GPT-5, DeepSeek-V3, and DeepSeek-R1, were evaluated using six prompt strategies with different languages and levels of complexity. Literature retrieval was conducted in CNKI, Wanfang Data, PubMed, and Embase for three representative disease domains: psoriasis, COVID-19, and type 2 diabetes mellitus. Manually constructed search queries served as the reference comparator. Retrieval performance was evaluated using recall, miss rate (1 - recall), precision, and F1 score. Robustness, variability, and leave-one-out analyses were conducted to assess consistency across disease domains, prompt strategies, models, and the composition of the reference dataset. The gold-standard dataset comprised 182 unique records after deduplication (psoriasis, 50; COVID-19, 74; type 2 diabetes mellitus, 58) extracted from studies included in previously published meta-analyses rather than from an exhaustive set of all potentially eligible studies meeting predefined PICO criteria. Results: Compared with manual queries, LLM assisted retrieval achieved higher precision and F1 scores but generally showed slightly lower recall. Simple prompt designs consistently produced better retrieval performance than complex prompts. Among the evaluated models, GPT-4o demonstrated the highest precision, whereas Deepseek-V3 achieved the highest recall. Sensitivity analyses revealed substantial heterogeneity in retrieval performance across disease domains, with psoriasis showing greater variability than COVID-19 and type 2 diabetes mellitus. Leave one out analyses indicated that overall results were stable, although a small number of records had a noticeable influence on refined retrieval outcomes. Compared with manual queries, LLM-assisted retrieval showed numerically higher precision and F1 scores in many evaluated settings, whereas recall was generally slightly lower. These metrics were calculated against a reference dataset derived from previously published systematic reviews and meta-analyses rather than an exhaustive relevance standard. Across the full modelprompt evaluation matrix, recall ranged from 0.081 to 0.310, precision from 0.0016 to 0.036, and F1 scores from 0.003 to 0.062. Simpler prompt strategies tended to have higher observed metrics than more complex prompt designs. Retrieval performance varied across disease domains and LLM configurations. Within the evaluated settings, GPT-4o achieved higher observed precision in the psoriasis dataset, whereas DeepSeek-V3 achieved higher recall in some settings. Because different models were evaluated in different disease domains and no formal inferential comparisons were performed, these findings are descriptive rather than evidence of model superiority. The findings suggest that LLM-based query optimization may improve certain aspects of literature retrieval efficiency under the current evaluation framework, particularly in terms of observed precision relative to the study reference set. However, retrieval performance remained sensitive to prompt design, disease context, and model selection. In addition, because studies retrieved by the LLMs but not included in prior meta-analyses were treated as false positives, the reported evaluation metrics should be interpreted cautiously. LLMs and prompt engineering may function as complementary tools for literature retrieval in TCM systematic reviews but cannot currently replace manual query construction. Their performance remains context-dependent and variable. Further methodological improvements, including optimized prompting, cross-lingual standardization, broader validation datasets, and integration of domain-specific knowledge-enhanced models, are necessary to achieve more accurate, stable, and reproducible AI-assisted literature retrieval in traditional medicine research.