A novel, interpretable metric for intrinsic KG quality assessment that measures how closely an automatically extracted graph approximates an"ideal"graph capturing the key noun phrases, predicate relations, and basic linguistic phenomena such as negation expressed in the source text is proposed.
Abstract
Knowledge Graphs (KGs) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance. Existing evaluation practices rely heavily on task-specific metrics or small-scale manual verification, offering limited insight into the structural and semantic fidelity of extracted graphs. We propose a novel, interpretable metric for intrinsic KG quality assessment that measures how closely an automatically extracted graph approximates an"ideal"graph capturing the key noun phrases, predicate relations, and basic linguistic phenomena such as negation expressed in the source text. Our framework integrates two complementary components: (1) an entity-level assessment that evaluates completeness, resolution quality, and connectivity, and (2) a relation-level assessment that judges predicate preservation and multiplicity using lexical similarity, dependency-parse alignment, and light-weight negation handling to ensure semantic faithfulness. We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets, including WebNLG, TinyButMighty, and BenchIE, demonstrating that it reliably identifies omissions, redundancy, and structural deviations that existing metrics overlook. Our work offers a scalable, model-agnostic, and interpretable framework for comparing automated KG construction methods and provides a foundation for standardised evaluation. We further validate the metric through an ablation study isolating noun and verb components, and a downstream evaluation showing that KGCQual scores correlate significantly with link prediction performance on the same extracted KGs. The code repository is available at https://github.com/kracr/kg-quality-metric.
This work targets a KG for Sophocles’ Antigone that supports two coupled uses: structured retrieval, through integrity and competency questions expressed in SPARQL over dramatic structure and interpretive annotations; and interactive exploration, through a lightweight read client that navigates lines across languages, shows scene context, and reports corpus statistics.
A production extraction layer that converts a live document stream into a validated knowledge graph aligned to a formal ontology, and improved search recall from roughly 70 to 95 percent with no false merges, and corrected seven classes of silent quality defect.
Vaibhav Dangaich, Kevin Lewis, Kundeshwar Pundalik· 0 citations
As experienced workers retire across industrialized economies, organizations risk losing procedural expertise that often remains tacit, undocumented or scattered across unstructured documents. Translating this knowledge into structured, machine-readable representations is difficult to scale, labor-intensive, and prone to inconsistency when done manually. This paper addresses the automated construction of knowledge graphs from natural language procedural descriptions, developing a generic approach for transforming unstructured expert knowledge into structured knowledge graphs that support downstream retrieval and question-answering applications. Three text-to-knowledge-graph approaches were designed, implemented, and systematically evaluated. The first employed a large general-purpose language model (Qwen3-32B) with a single-stage zero-shot extraction prompt, the second applied the same strategy using a smaller base model (Llama2-13B), and the third combined supervised fine-tuning of the smaller model on synthetic extraction data with a decomposed extraction architecture targeting one to two ontology elements per phase. All approaches were evaluated across six procedural descriptions spanning multiple technical domains, with chunk size, model temperature, and ontology detail as configuration parameters. Results were assessed for intrinsic quality and extrinsic fitness for use, measured by question-answering accuracy in a Graph-RAG application. The results demonstrate that extraction strategy is a more decisive factor than model capacity. The fine-tuned model achieved a Question Answering (QA) pass rate of 55.3%, compared to 46.8% for the large model and 33% for the small base model, outperforming the general-purpose model on both intrinsic quality metrics and downstream performance. Average node degree, duplication rate, and ontology simplicity emerge as the strongest predictors of retrieval performance. Shorter ontologies consistently outperformed richer ones, suggesting that ontology design should be driven by the intended downstream application rather than semantic completeness. Over-extraction of procedural steps did not degrade performance but instead improved retrieval robustness by providing additional entry points for graph traversal. The findings offer practical guidance for designing scalable, locally deployable knowledge graph construction pipelines for procedural texts under computational and confidentiality constraints.
Erik Sörqvist, Kenneth Obinna, Clara Bersch et al.· European Conference on Knowl...· 0 citations
This study explores whether human-written descriptions in Reactome can be used to infer the experts'defined global hierarchical structure and indicates that the global hierarchical structure of pathways can be inferred by experts textual metadata.
Susanna Bravi, R. De-Luca, R. Sicilia et al.· 0 citations
Large Language Models (LLMs) show considerable potential for materials-science question answering. However, LLM responses may still be affected by unsupported parametric associations, while dense Retrieval-Augmented Generation (RAG) can fragment relational evidence across text chunks. Moreover, general graph-based retrieval does not necessarily preserve the hierarchical relations and factual attributes required to resolve implicit material constraints. To address these limitations, we propose MCTD-KG, a multi-source heterogeneous knowledge graph integrated with a Knowledge-Enhanced RAG framework for complex material question answering. MCTD-KG adopts a Classification–Term–Data ontology to connect disciplinary taxonomies, domain concepts, semantic relations, and empirical records from toolbooks and the scientific literature. Through LLM-assisted knowledge extraction, entity normalization, and multi-source integration, the resulting graph contains more than 530,000 entities across three layers, including 61,768 text-extracted Term-layer entities. During inference, Dual-Channel Retrieval jointly retrieves query-relevant relational paths and associated material attributes, while an explicit semantic filtering stage screens candidate evidence against the query constraints. Evaluation on an expert-validated benchmark of 1577 questions shows that the proposed framework achieves an overall accuracy of 68.48%, compared with 17.40% for the zero-shot Pure LLM, 24.79% for the best Vanilla RAG setting, and 44.96% for GraphRAG. It also achieves 45.22% accuracy on four-hop questions, compared with 39.49% for GraphRAG. These results indicate that integrating multi-source domain knowledge with relation-preserved retrieval and attribute-supported filtering provides more focused and inspectable evidence, thereby supporting more accurate complex material question answering.
Peize Li, Xi Guo, Nan Yin et al.· Electronics· 0 citations