A systematic analysis of existing evaluation frameworks and metrics for RAG-based Question Answering (QA) systems reveals urgent needs for hybrid evaluation frameworks, reasoning path traceability, combined pipeline assessment, and harmlessness evaluation, laying groundwork for the next generation of evaluation methodologies.
A novel knowledge graph-based evaluation framework is proposed introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors.
Subavarshana Arumugam, Mamta Nallaretnam, K. Wickramasinghe et al.· 0 citations
This survey provides a comprehensive review of developments in NQG, spanning traditional neural approaches to the latest paradigms driven by large language models (LLMs) and multimodal large language models (MLLMs).
S YN T AB QA defines a detailed question typology, enabling fine-grained evaluation and facilitating the creation of diverse QA datasets, and not only provides an automated test bed for any tabular dataset but can be used in few-shot settings to supply LLMs with tailored examples, improving their focus and accuracy.
Jorge Osés Grijalba, Eugenio Martínez-Cámara, Luis Alfonso Ureña López et al.· International Conference on...· 0 citations
A semantic correctness taxonomy is introduced that assigns open-ended answers to eight ordered classes, separating verbose-but-correct answers from those contaminated by hallucinated content and CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements using bidirectional NLI.
Elitsa Yotkova, Violeta Kastreva, Petar Velkov et al.· 0 citations
A systematic review of 137 shortlisted research articles reveals that the performance of a cross-lingual QA system is structurally influenced by the interaction between linguistic typology, model design, and not exclusively by model scale or architecture.