This work introduces a metamorphic testing framework that evaluates the consistency of RAG systems under corpus evolution, formalising a fault taxonomy and 11 mutation operators that systematically perturb the system at both the pre-chunk (retrieval index) and post-chunk (retrieved context) levels.
Abstract
Retrieval-Augmented Generation (RAG)-based LLM systems rely on external document corpora that can evolve and change over time. However, current evaluation methodologies (e.g., RAGAS) assess correctness against static snapshots, failing to detect faults when routine updates, factual changes, or noise alter the underlying data. We introduce a metamorphic testing framework that evaluates the consistency of RAG systems under corpus evolution. We formalise a fault taxonomy and 11 mutation operators that systematically perturb the system at both the pre-chunk (retrieval index) and post-chunk (retrieved context) levels. An empirical evaluation across five datasets and over 28k mutants reveals metamorphic violation rates of 4.9-10.2%. In a meta-evaluation against ground truth, our metamorphic oracle achieves F1 scores of 0.927-1.000, while the best RAGAS metric reaches only 0.570. Finally, we provide actionable insights into mitigating these faults through retrieval re-configuration, generator upgrades, and LLM-based reranking.
Evaluating the temporal consistency of RetrievalAugmented Generation (RAG) systems is challenging because conventional evaluation methods rely on ground-truth answers, which are difficult to define under continuously evolving knowledge. This paper presents a temporal metamorphic testing framework for RAG-based large language models (LLMs) that evaluates relational consistency across multiple executions over time rather than comparing outputs against static reference answers. To support temporal metamorphic testing, we construct controlled temporal revision families from seed facts extracted from real-world documents. Each fact is systematically cloned and transformed across multiple timestamps to simulate evolving knowledge states. Based on this design, temporal metamorphic relations are instantiated to systematically test RAG systems under different knowledge snapshots. Experiments on 2344 metamorphic test instances show that 48.04% of them violate the expected temporal relations, indicating that temporal inconsistencies are widespread in the evaluated RAG system. These results demonstrate that temporal metamorphic testing provides an effective way to assess temporal reliability in RAG systems without requiring explicit ground-truth answers.
Hongyu Wang, Dave Towey, Ge Zhang et al.· Annual International Compute...· 0 citations
Chunk Coverage (CC), an oracle-independent test adequacy criterion for testing the retrieval component of RAG systems, is introduced and results show that CC captures retrieval diversity relevant to effective testing without requiring test oracles.
Jinhan Kim, Samuele Pasini, Paolo Tonella· 1 citation
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatible configurations, and compare it with a baseline test-input generator. Across 72,000 test executions, RagTester detected 21,633 failures, 6.6% more than the baseline, and outperformed it in 20 of the 24 configurations. The detected failures include inaccurate retrieval, unsupported answers, incomplete use of retrieved context, and difficulties interpreting complex passages. These results show that coverage-oriented test generation can effectively expose failures caused by the interaction between retrieval and generation components and support the assessment of RAG configurations before deployment.
Ange Maiztegi, J. Ayerdi, Miren Illarramendi et al.· 0 citations
Software testing often depends on a test oracle to determine whether or not an output is correct. However, many real-world systems lack reliable or affordable oracles. Metamorphic testing (MT) alleviates this problem by using metamorphic relations (MRs), which examine the correctness of relations between outputs produced from related inputs across multiple executions. While the effectiveness of MT depends heavily on the quality of MRs, existing MR evaluation is typically guided by informal or study-specific criteria, limiting standardization and repeatability, especially for complex and AI-integrated systems. This paper proposes a structured MR-evaluation rubric that operationalizes key criteria to support consistent and repeatable assessment. We conducted an empirical study on nine systems under test (SUTs) spanning three levels of complexity, using fresh sets of LLM-generated MRs for each SUT. We then compared evaluations from human experts against two widely used large language models (LLMs), namely ChatGPT and Gemini, by configuring these models as MR evaluators using the same rubric. The results show that human and LLM evaluators are more consistent when assessing simpler SUTs, but their judgments increasingly diverge as system complexity grows and evaluation criteria become more varied. The study further reveals clear differences in LLM-evaluation behaviors, indicating that evaluator choice can affect MR assessment outcomes in complex settings. Overall, the proposed rubric provides a standard framework for MR evaluation, and supports a systematic comparison between human and LLM evaluators, offering empirical evidence on when LLM-based MR evaluation is more reliable, and where additional care is needed as SUT complexity increases.
Yifan Zhang, Dave Towey, Matthew Pike et al.· Annual International Compute...· 0 citations
This work introduces a search-based approach that identifies and evolves a set of natural language transformation rules with strong downstream effects on coding performance, and proposes DUALFIX, a staged repair pipeline that combines the evolved transformation rules with execution-feedback repair, addressing both specification-level and implementation-level failures.
Amal Akli, Melissa Akli, Cedric Richter et al.· 0 citations
Decompose--Enhance--Correct (Decompose--Enhance--Correct), a visual-consistency-guided agentic framework that improves frozen table parsers without retraining is proposed, which derives a 1,977-table Consensus-Hard Set from 4,556 candidates through offline metrics and cross-model consensus.
Jutao Xiao, Yuan Qu, Dongsheng Ma et al.· 0 citations