Evaluating LLM-Based Retrieval-Augmented Generation for Soil Science Question Answering

Abstract

Retrieval-augmented generation (RAG) systems for scientific literature require evidence-based choices of document segmentation, representation, retrieval, and generation components, particularly when the source collection varies in topical specificity and document structure. This study addresses the lack of an end-to-end, component-level comparison of these choices for soil science question answering. A three-stage evaluation was conducted across general, domain-specific, and geospatial soil science corpora. The corpus combines foundational soil science books, peer-reviewed research articles, European soil monitoring material, and geospatial mapping publications, thereby covering both broad disciplinary concepts and specialized scientific evidence. The study compares four chunking strategies, three embedding models, five retrieval methods, and five large language models. In Experiment 1, semantic chunking with text-embedding-3-large achieved the highest aggregate retrieval scores (recall@1 = 0.824; MRR = 0.819), whereas text-embedding-3-small delivered practically comparable performance at lower cost. In Experiment 2, hybrid reciprocal rank fusion achieved recall@5 values of 0.957, 0.960, and 0.647 for the general, domain-specific, and geospatial corpora, respectively; the cross-encoder reranker showed weaker rank quality on scientific content. In Experiment 3, model responses attained BERTScore values of 0.909–0.927 and faithfulness of at least 0.993; these automated measures indicate low contradiction with retrieved context but do not establish answer completeness or human-perceived correctness. The study provides a reproducible component-level evaluation design, characterizes the effect of corpus specificity on RAG retrieval, and identifies a practical configuration for soil science literature retrieval. Among the models retained for direct aggregate comparison, Llama 3.1 8B offered the most favorable observed balance of answer quality, latency, cost, and model openness.