Structural Metric Evaluation of LLM Generated Ontologies
Abstract
Ontologies support reasoning, information retrieval, data integration, and FAIR data practices by representing domain knowledge as structured entities and relationships. They are a fundamental component of neuro-symbolic AI systems, providing risk mitigation for AI systems through explicitly-encoded knowledge. Large language models offer a promising way to generate ontology-like structures from unstructured text, but evaluating these generated ontologies remains difficult. Semantic validation is essential, but it is often costly, domain-specific, and difficult to scale across large numbers of generated artifacts. This paper investigates whether structural metrics can support evaluation for LLM-generated ontologies. We compute a suite of metrics across OntoPortal ontologies, LLMs4OL gold-standard ontologies, ontologies generated from the LLMs4OL dataset, and LLM-generated ontologies from Wikipedia articles. We examine whether generated ontologies are structurally distinguishable from curated ontology populations. Our results suggest that structural metrics can identify systematic differences between generated and curated ontologies, particularly in hierarchy, connectivity, branching, and relation usage. These metrics do not replace semantic validation, but they can support scalable triage, comparison, and diagnosis of generated ontologies, providing a first step toward understanding how ontology-generation methods can be improved.