Provenance-Native Audit Infrastructure for LLM-Maintained Wiki Knowledge Systems
TL;DR
An automated gap analysis shows the architecture demonstrates alignment with five of eight requirements derived from the EU AI Act and FDA AI/ML guidance, and module-level evaluations of claim extraction, evidence binding, and silent-edit detection are reported together with a measured baseline comparison on retraction propagation.
Abstract
LLM-maintained wiki systems accumulate structured knowledge by having a language model incrementally build and revise a persistent wiki layer over raw evidence sources. When such systems support decision-making in regulated domains, each output must be traceable to grounded evidence through an auditable chain. This paper presents a provenance-native architecture in which a dedicated provenance layer is embedded in the wiki’s operational loop, recording evidence bindings, claim versions, rule compilations, and decision traces as side effects of normal system operations. The architecture is organized around four technical components: (1) a typed provenance graph that binds claims to evidence passages with confidence-weighted edges; (2) a dual-layer diff mechanism combining textual and semantic comparison to detect silent LLM edits; (3) three confidence propagation strategies over multi-hop evidence–claim–rule–decision chains, compared on their suitability for regulatory communication; and (4) a retraction propagation engine that identifies all downstream dependents when upstream evidence is invalidated. We evaluate the system on a 112-document corpus producing 3174 graph nodes and 4398 edges. In controlled retraction experiments, the engine achieves 100% downstream recall with zero false propagation in under 600 ms per event. A sensitivity analysis over the human-review threshold parameter γ reveals a sharp phase transition in the fraction of flagged decisions, providing a concrete basis for negotiating review policies with regulators. An automated gap analysis shows the architecture demonstrates alignment with five of eight requirements derived from the EU AI Act and FDA AI/ML guidance, and we report module-level evaluations of claim extraction, evidence binding, and silent-edit detection together with a measured baseline comparison on retraction propagation. Code and experimental protocols will be released publicly upon publication.