: High-stakes clinical decision support (CDS) demands a property that aggregate accuracy cannot capture: a trace that a clinician who was not in the room can inspect layer by layer when the system is wrong. We argue that the way to obtain this property is to refuse to entangle the large language model (LLM) with the rest of the pipeline. We propose KRD (Knowledge–Rule–Decision) , a four-component architecture that separates fact extraction, a compile-time clinical knowledge layer in the spirit of the LLM Wiki pattern of Karpathy, a rule layer of hand-written contraindications and heuristics, and a decision interface whose compose method short-circuits to a rule-cited blocking response whenever any hard violation fires. We evaluate KRD against a pure language model, a retrieval-augmented language model, a rule-only system, and a light hybrid on a benchmark of 32 type-1 diabetes scenarios. A strict version of the unsafe-suggestion rate stratifies the five systems monotonically into four distinct tiers from 0.867 down to zero, with S4 and S5 tied at the floor; the full KRD stack and the light hybrid reach the hard-safety ceiling together; KRD leads the light hybrid on evidence trace completeness by 25% relative and on reviewer correction burden by 12% relative, both directionally clear and borderline significant under bootstrap intervals; and KRD issues 17 language model calls per benchmark pass against the light hybrid’s 32, a 47% reduction that is a direct consequence of the architectural choice to evaluate the rule layer before invoking the model. We also report honestly that the evidence gate is inert on this benchmark because every compiled concept is graded A or B, and we trace five fact-extraction failures to a single field and a single linguistic pattern. The contribution is not that KRD is universally optimal but that layer-wise auditability is a design discipline whose cost in this setting was lower than its critics would have predicted.
Bailing Zhang, Genlang Chen· Journal of Intelligent Medic...· 0 citations
. Large Language Models (LLMs) are increasingly used to maintain persistent, domain-specific knowledge bases—a paradigm in which assertions must remain accurate as the field evolves. Existing staleness detection treats each assertion independently, missing a structural reality: when a foundational concept becomes outdated, every dependent concept inherits some degree of that staleness through ontology relationships. We present WikiMonitor-Onto, a lightweight propagation layer built on WikiMonitor that models staleness as a signal flowing through a domain ontology graph. We extract a concept graph of 642 nodes and 487 edges from 61 AI lecture documents, define three typed propagation relations ( is-a , depends-on , related-to ), and propagate staleness via weighted Breadth-First Search (BFS) with exponential hop decay. On a human-annotated gold standard of 62 concepts (25 indirect-stale, 37 fresh), the independent baseline detects zero indirect-stale concepts by construction, while WikiMonitor-Onto achieves precision 0.824 and recall 0.560 at the optimally tuned configuration ( λ = 0.30). Grid search reveals that is-a and depends-on carry equal optimal propagation weight (both 0.90). A sensitivity analysis confirms that propagation is robust to the choice of seed-value distribution, with F1 varying by only 0.10 (0.571–0.667) across four tested distributions. Propagation saturates at hop depth 1 under conservative thresholds, suggesting that one-hop propagation suffices for high-precision deployment.
Bailing Zhang· Journal of Advances in Artif...· 0 citations
Knowledge rules in AI engineering carry an awkward double duty. A clinician or a compliance officer has to be able to read and revise them, and a downstream system has to be able to execute and check them. Getting from the natural-language version of such a rule to a symbolic program is rarely the hard part; getting to one that is not merely runnable but logically sound is. We call this the compilation gap and attack it with NeSyWikiCompiler, a four-stage neural-symbolic pipeline. A language-model frontend reads each specification into NeSy-IR, an intermediate representation that holds onto exactly the details downstream code generation needs and that language models routinely drop: predicate types, the direction of numeric comparisons, the modality of each constraint, and a pointer back to the source text. From a single IR, deterministic compilers emit both a Prolog program and a Z3 program. The Z3 side is then checked with Clark’s completion, which surfaces a failure that, in our experiments, plain Prolog execution cannot see at all: rules whose violation condition can never be satisfied, so that the checker silently never fires. A repair loop that is CEGIS-informed but ultimately deterministic for structural errors handles the two kinds of failure differently: an LLM revision step targets semantic slips such as lost numeric thresholds and reversed arguments, while a deterministic reconstruction step—reached once the LLM rounds have failed—is the path taken for the structural contradictions, which the LLM step does not fix. In a cross-domain evaluation covering clinical decision rules, AI course knowledge, and legal compliance specifications—three representative domains rather than an exhaustive sample—all compiled programs achieve syntax validity across both backends. Formal verification exposes a class of structural contradictions that, in this pilot-scale benchmark, is concentrated in the legal specifications, where normative hedging constructs appear to induce rule-constraint conflicts. Deterministic repair recovers most structural failures while LLM-only repair consistently reproduces the same broken rule patterns. These findings characterise a previously unrecognised failure mode in LLM-to-logic compilation and demonstrate a practical engineering toolchain for producing verifiable knowledge-base programs from natural-language specifications. We present the domain-level rates as failure-mode discovery on a small corpus rather than as estimates that generalise without further study.