An Evidence-Based Knowledge Graph Framework with Large Language Models and Human-in-the-Loop for Ovarian Cancer Resistance

Abstract

The reciprocal relationship between large language models (LLMs) and knowledge graphs (KGs) opens new opportunities for knowledge discovery from scientific literature; however, most approaches address only one direction of this interaction. In complex biomedical domains, ensuring reliability, interpretability, and traceability remains challenging, making expert supervision essential. In this work, we propose a framework that integrates LLMs, KGs, and a Human-in-the-Loop scheme for constructing, evaluating, and reasoning over Evidence-based Knowledge Graphs (E-KGs), explicitly linking each extracted triplet to supporting textual evidence, and demonstrate its application to study the phenomenon of resistance in ovarian cancer. The E-KG was constructed using MedGemma-27B, with experts participating throughout the E-KG lifecycle, supported by our interactive visualization tools. We evaluated the E-KG through an expert-driven protocol assessing coherence and biological plausibility with inter-annotator agreement (Gwet’s AC1 coefficient), complemented by GraphRAG-based tasks, including link prediction (Hits@k, MRR) and a question-answering case study (faithfulness and answer relevance by LLM-as-a-judge and expert audit). The framework produced a coherent, plausible, and auditable E-KG, highlighting the potential of evidence-grounded knowledge to support LLM-based reasoning. To our knowledge, we introduce the first E-KG specialized in ovarian cancer resistance, providing an expert-validated resource that advances trustworthy and explainable AI for biomedical knowledge discovery.