AI-Driven Knowledge Externalisation: From Unstructured Documents to Structured Data Models

From Unstructured Procedural Text to Knowledge Graphs: Comparing LLM-Based Extraction Strategies

As experienced workers retire across industrialized economies, organizations risk losing procedural expertise that often remains tacit, undocumented or scattered across unstructured documents. Translating this knowledge into structured, machine-readable representations is difficult to scale, labor-intensive, and prone to inconsistency when done manually. This paper addresses the automated construction of knowledge graphs from natural language procedural descriptions, developing a generic approach for transforming unstructured expert knowledge into structured knowledge graphs that support downstream retrieval and question-answering applications. Three text-to-knowledge-graph approaches were designed, implemented, and systematically evaluated. The first employed a large general-purpose language model (Qwen3-32B) with a single-stage zero-shot extraction prompt, the second applied the same strategy using a smaller base model (Llama2-13B), and the third combined supervised fine-tuning of the smaller model on synthetic extraction data with a decomposed extraction architecture targeting one to two ontology elements per phase. All approaches were evaluated across six procedural descriptions spanning multiple technical domains, with chunk size, model temperature, and ontology detail as configuration parameters. Results were assessed for intrinsic quality and extrinsic fitness for use, measured by question-answering accuracy in a Graph-RAG application. The results demonstrate that extraction strategy is a more decisive factor than model capacity. The fine-tuned model achieved a Question Answering (QA) pass rate of 55.3%, compared to 46.8% for the large model and 33% for the small base model, outperforming the general-purpose model on both intrinsic quality metrics and downstream performance. Average node degree, duplication rate, and ontology simplicity emerge as the strongest predictors of retrieval performance. Shorter ontologies consistently outperformed richer ones, suggesting that ontology design should be driven by the intended downstream application rather than semantic completeness. Over-extraction of procedural steps did not degrade performance but instead improved retrieval robustness by providing additional entry points for graph traversal. The findings offer practical guidance for designing scalable, locally deployable knowledge graph construction pipelines for procedural texts under computational and confidentiality constraints.

Structure then Query: Enabling Precise Analytical Queries over Unstructured Documents

Experiments on three real-world datasets demonstrate that AnnoIndex consistently outperforms state-of-the-art baselines, achieving the highest average F1 score while maintaining robust performance on complex multi-hop join and progressive reasoning queries.

From Tacit Knowledge to Structured Documents: A Framework for Knowledge Elicitation

Recent advances in Large Language Model (LLM) “agent” systems have moved language models beyond single-turn generation toward goal-directed workflows that can plan, ask clarifying questions, and iteratively refine outputs. In parallel, retrieval-augmented generation (RAG) has become a practical way to ground these agents in enterprise knowledge—enabling models to leverage internal documentation and policies without expensive and recurring retraining. However, RAG is only as strong as the underlying corpus: when key information is missing, outdated, or fragmented, retrieval cannot fill the gaps. In many organizations, assembling high-quality, up-to-date documents remains a persistent bottleneck, limiting both the reliability of downstream applications and the speed at which knowledge can be operationalized. To address this, we introduce the Assistant-Scribe-Knowledge Checker (ASK) framework for generating structured specification documents through guided interviews. ASK decomposes the end-to-end workflow into three specialized LLM agents: an Assistant that asks questions to the user and steers the interview through adaptive follow-ups; a Scribe that continuously summarizes what has been said and records it into a schema-constrained specification document with explicit content requirements; and a Knowledge Checker that evaluates the evolving document against those specifications, detects gaps or weakly supported entries, and advises the Assistant on the next best questions to ask to reach the desired completeness and quality. This separation supports seamless document creation while improving quality control over content, structure, and coverage. By distributing responsibilities across three lightweight, role-specific agents, ASK reduces reliance on a single large model and enables modular scaling across teams and domains. We evaluated the ASK framework in a consulting-firm setting where consultants are required to produce project “return-of-experience” documents (successes, failures, challenges, and solutions) to capture reusable knowledge. Compared to documents authored manually, ASK-guided interviews consistently produced more complete, better-structured, and higher-quality specifications with less variability across authors. Beyond measurable quality gains, consultants also reported a clear preference for the interview-based workflow, citing lower effort and a more natural way to articulate tacit project knowledge.

From Knowledge Retrieval to Execution: Designing Executable Knowledge Systems

Generative AI is changing the role of knowledge in organisations. Traditional knowledge management (KM) systems have primarily supported storage, access and retrieval, assuming that knowledge is interpreted and applied by human users. In AI-enabled environments, however, organisational knowledge increasingly becomes a direct input into execution, shaping generated proposals, analyses, summaries, recommendations and other workflow outputs. This shift exposes a limitation of retrieval-oriented KM: fragmented, outdated or weakly governed knowledge can be amplified through AI-generated outputs, reducing consistency, reliability and trust. This paper introduces executable knowledge systems as a conceptual model for structuring organisational knowledge to support reliable human and AI-assisted execution. The term executable is used in a socio-technical sense. Knowledge does not necessarily become code, but is curated, validated and embedded into workflows so that it can guide outputs, decisions and actions. The paper distinguishes this concept from prior work on executable knowledge graphs and executable knowledge bases, which primarily focus on deterministic execution through rules, scripts or formalised representations. The paper further develops a framework of decay and compounding loops to explain how AI-enabled knowledge systems evolve over time. In decay loops, AI-generated outputs re-enter the knowledge environment without sufficient validation, allowing inconsistency and low-quality knowledge to accumulate. In compounding loops, curated knowledge assets are refined through governed feedback, domain ownership and controlled reuse, enabling improvements in reliability over time. The framework is informed by an exploratory case study within a global professional services organisation, where a curated knowledge environment was introduced to support AI-assisted workflows in the Retail, Consumer Products, Travel and Transportation domain. The evaluation compared outputs generated from a controlled, subject matter expert (SME)-validated knowledge dataset with outputs generated from an unconstrained organisational knowledge base. Findings indicate improved retrieval relevance and output quality when AI systems operate on validated knowledge assets. The paper contributes to KM research by reframing KM as a system design challenge for AI-enabled execution and by positioning governance, validation and feedback control as central mechanisms for reliable organisational knowledge use.

AI-Guided Metadata Construction for Meaning-Driven Digital Knowledge Systems: A Framework for Automated Metadata Generation and Semantic Discovery

Integrating Large Language Models (LLMs) with Oracle 26AI for Advanced Enterprise Analytics and Knowledge Management

Results indicate that LLM-augmented analytics on a converged Oracle 26AI platform can reduce average analytical query resolution time by approximately 60 percent relative to traditional BI report cycles, achieve semantic retrieval precision above 90 percent for enterprise knowledge corpora, and reduce generative output hallucination rates by more than half when grounding is enforced through in-database retrieval.