Efficacy Evaluation and Optimization of RAG Knowledge Bases in the Oil and Gas Industry Using an LLM-as-a-Judge Mechanism

Abstract

The increasing adoption of Retrieval-Augmented Generation (RAG) in the oil and gas industry has created a growing need for systematic evaluation of domain-specific knowledge bases, particularly with respect to retrieval failures, numerical and entity inconsistencies, knowledge timeliness, and unsupported generation. This study presents a domain-adapted evaluation and optimization framework for industrial RAG knowledge bases based on an LLM-as-a-Judge paradigm. The framework organizes evaluation into four dimensions—Data, Retrieval, Generation, and Utility (DAAE)—and combines deterministic metrics with LLM-based semantic assessment. A two-tier evaluation procedure combines retrieval-based screening with fine-grained LLM judging while retaining retrieval failures in end-to-end evaluation statistics. Rather than introducing new retrieval or generation algorithms, the framework integrates established RAG techniques with evaluation criteria motivated by oil-and-gas knowledge characteristics, including domain-entity and numerical consistency, temporal validity, chunk-level semantic integrity, and controlled abstention. Evaluation results are mapped to corresponding optimization actions across the data, retrieval, and generation layers, including semantic-aware chunking, metadata augmentation, hybrid sparse–dense retrieval, Cross-Encoder reranking, and structured evidence-grounded prompting. The framework was evaluated using the Intelligent Knowledge Base for Natural Gas Economic Research and an expert-annotated benchmark comprising 150 domain questions. In the industrial before–after comparison, Retrieval Hit@5 increased from 58.0% (87/150) to 89.3% (134/150), Context Precision increased from 0.64 to 0.88, and Faithfulness increased from 0.65 to 0.94. These values are reported as system-level point estimates rather than as component-wise causal effects. The results demonstrate the practical value of evaluation-guided optimization for improving the reliability of domain-specific RAG systems in natural-gas economic research and provide an industrial case for systematic RAG assessment and iterative optimization in the energy sector.